部署 harbor 私有仓库
高枫_henu 人气:2安装下载依赖包
安装docker-compose
从 docker compose 发布页面下载最新的 docker-compose 二进制文件,本文以1.25.4为例
cd /opt/k8s/work/harbor/ wget https://github.comhttps://img.qb5200.com/download-x/docker/compose/releaseshttps://img.qb5200.com/download-x/download/1.25.4https://img.qb5200.com/download-x/docker-compose-Linux-x86_64 mv docker-compose-Linux-x86_64 /opt/k8s/binhttps://img.qb5200.com/download-x/docker-compose chmod a+x /opt/k8s/binhttps://img.qb5200.com/download-x/docker-compose export PATH=/opt/k8s/bin:$PATH下载harbor二进制文件
从 harbor 发布页面下载最新的 harbor 离线安装包，本文以v1.9.4为例
cd /opt/k8s/work/harbor/ wget https://github.com/goharbor/harbor/releaseshttps://img.qb5200.com/download-x/download/v1.9.4/harbor-offline-installer-v1.9.4.tgz tar -xzvf harbor-offline-installer-v1.5.1.tgz导入 docker images
导入离线安装包中 harbor 相关的 docker imagescd /opt/k8s/work/harbor/harbor docker load -i harbor.v1.9.4.tar.gz
以http形式启动
修改harbor.yml文件
修改 hostname、data_volume属性值cd /opt/k8s/work/harbor/harbor cp harbor.yml harbor.yml.bak vim harbor.yml修改信息如下
diff harbor.yml harbor.yml.bak 5c5 < hostname: 192.168.0.107 --- > hostname: reg.mydomain.com 40c40 < data_volume: https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data --- > data_volume: https://img.qb5200.com/download-x/data加载和启动 harbor 镜像
cd /opt/k8s/work/harbor/harbor mkdir -p https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data chmod 777 /var/runhttps://img.qb5200.com/download-x/docker.sock https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data ./install.sh启动日志
Note: docker version: 18.09.6 Note: docker-compose version: 1.25.4 # Configuration file of Harbor [Step 1]: loading Harbor images ... Loaded image: goharbor/harbor-core:v1.9.4 Loaded image: goharbor/clair-photon:v2.1.0-v1.9.4 Loaded image: goharbor/harbor-portal:v1.9.4 Loaded image: goharbor/nginx-photon:v1.9.4 Loaded image: goharbor/chartmuseum-photon:v0.9.0-v1.9.4 Loaded image: goharbor/prepare:v1.9.4 Loaded image: goharbor/redis-photon:v1.9.4 Loaded image: goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.4 Loaded image: goharbor/notary-server-photon:v0.6.1-v1.9.4 Loaded image: goharbor/harbor-log:v1.9.4 Loaded image: goharbor/harbor-db:v1.9.4 Loaded image: goharbor/harbor-jobservice:v1.9.4 Loaded image: goharbor/harbor-registryctl:v1.9.4 Loaded image: goharbor/notary-signer-photon:v0.6.1-v1.9.4 Loaded image: goharbor/harbor-migrator:v1.9.4 [Step 2]: preparing environment ... prepare base dir is set to /opt/k8s/work/harbor/harbor Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /confighttps://img.qb5200.com/download-x/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml Generated and saved secret to file: /secret/keys/secretkey Generated certificate, key file: /secret/core/private_key.pem, cert file: /secret/registry/root.crt Generated configuration file: /compose_locationhttps://img.qb5200.com/download-x/docker-compose.yml Clean up the input dir [Step 3]: starting Harbor ... Creating network "harbor_harbor" with the default driver Creating harbor-log ... done Creating harbor-db ... done Creating redis ... done Creating harbor-portal ... done Creating registry ... done Creating registryctl ... done Creating harbor-core ... done Creating harbor-jobservice ... done Creating nginx ... done ✔ ----Harbor has been installed and started successfully.---- Now you should be able to visit the admin portal at http://192.168.0.107. For more details, please visit https://github.com/goharbor/harbor .查看启动状态
root@master:/opt/k8s/work/harbor/harbor# docker-compose ps Name Command State Ports --------------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Up (healthy) harbor-db https://img.qb5200.com/download-x/docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /entrypoint.sh /etc/regist ... Up (healthy) 5000/tcp registryctl /harbor/start.sh Up (healthy)- 如果有不是healthy状态的对象，到/var/log/harbor/目录下查看对应对象的日志
harbor官方提供的启动文件中容器的日志类型是syslog，不支持用docker logs查看
ls /var/log/harbor/ core.log jobservice.log portal.log postgresql.log proxy.log redis.log registryctl.log registry.log
浏览器访问 http://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。
创建一个新的项目
- docker命令拉取和上传镜像
执行login
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. Error response from daemon: Get https://192.168.0.107/v2/: dial tcp 192.168.0.107:443: connect: connection refused因为docker命令默认采用https和API交互，而我们的harbor是http的，所以不能执行，需要在/etchttps://img.qb5200.com/download-x/dockerhttps://img.qb5200.com/download-x/daemon.json中追加insecure-registries配置
{ "registry-mirrors": ["https:/https://img.qb5200.com/download-x/docker.mirrors.ustc.edu.cn","https://hub-mirror.c.163.com"], "insecure-registries": ["192.168.0.107"], "max-concurrent-downloads": 20, "live-restore": true, "max-concurrent-uploads": 10, "data-root": "https://img.qb5200.com/download-x/data/k8shttps://img.qb5200.com/download-x/dockerhttps://img.qb5200.com/download-x/data", "log-opts": { "max-size": "100m", "max-file": "5" } }重新启动docker服务
root@slave:~# systemctl restart docker root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https:/https://img.qb5200.com/download-x/docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded向harbor中刚创建的私有项目里面上传镜像
root@slave:~# docker tag k8s.gcr.io/pause:3.1 192.168.0.107/k8s/pause:3.1 root@slave:~# docker push 192.168.0.107/k8s/pause:3.1 The push refers to repository [192.168.0.107/k8s/pause] e17133b79956: Pushed 3.1: digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d size: 527浏览器查看
从harbor中下载镜像
root@slave:~# docker rmi 192.168.0.107/k8s/pause:3.1 Untagged: 192.168.0.107/k8s/pause:3.1 Untagged: 192.168.0.107/k8s/pause@sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d root@slave:~# docker pull 192.168.0.107/k8s/pause:3.1 3.1: Pulling from k8s/pause Digest: sha256:fcaff905397ba63fd376d0c3019f1f1cb6e7506131389edbcb3d22719f1ae54d Status: Downloaded newer image for 192.168.0.107/k8s/pause:3.1
以https形式启动
下列操作的工作目录/opt/k8s/work/harbor/harbor是解压harbor离线安装文件后 生成的 harbor 目录
停止harbor（如果没有启动过，则跳过该步骤）
cd /opt/k8s/work/harbor/harbor root@master:/opt/k8s/work/harbor/harbor# COMPOSE_HTTP_TIMEOUT=200 docker-compose down -v Stopping nginx ... done Stopping harbor-jobservice ... done Stopping harbor-core ... done Stopping registryctl ... done Stopping redis ... done Stopping harbor-portal ... done Stopping harbor-db ... done Stopping registry ... done Stopping harbor-log ... done Removing harbor-log ... done Removing network harbor_harbor执行docker-compose down -v后，有可能对应的容器还在，可以执行强制停掉容器
docker ps | grep busybox_busybox | awk '{print $1}' | xargs -I {} docker rm -f {}
创建 harbor nginx 服务器使用的 x509 证书，本文档采用cfssl工具生成，也可以利用openssl生成，具体步骤参考harbor-openssl
创建harbor用的CA根证书
创建配置文件
cd /opt/k8s/work/harbor/ cat > harbor-ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "harbor": { "usages": [ "signing", "key encipherment", "server auth", "client auth" ], "expiry": "87600h" } } } } EOF创建证书签名请求文件
cd /opt/k8s/work/harbor/ cat > harbor-ca-csr.json <<EOF { "CN": "harbor-ca", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "NanJing", "L": "NanJing", "O": "k8s", "OU": "system" } ], "ca": { "expiry": "87600h" } } EOF生成自签名的根证书,分发证书
cd /opt/k8s/work/harbor/ cfssl gencert -initca harbor-ca-csr.json | cfssljson -bare harbor-ca ls harbor-ca*pem mkdir -p /etc/harbor/cert mv harbor-ca*pem harbor-ca-config.json /etc/harbor/cert
创建harbor用的证书
生成证书请求文件
cd /opt/k8s/work/harbor/ cat > harbor-server-csr.json <<EOF { "CN": "harbor", "hosts": [ "127.0.0.1", "192.168.0.107" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "ST": "NanJing", "L": "NanJing", "O": "k8s", "OU": "system" } ] } EOF- hosts 字段指定授权使用该证书的当前部署节点 IP
生成证书
cfssl gencert -ca=/etc/harbor/cert/harbor-ca.pem \ -ca-key=/etc/harbor/cert/harbor-ca-key.pem \ -config=/etc/harbor/cert/harbor-ca-config.json \ -profile=harbor harbor-server-csr.json | cfssljson -bare harbor-server ls harbor-server*pem mv harbor-server*pem /etc/harbor/cert
编辑配置文件
修改 hostname、data_volume属性值,去除默认的http方式，追加https的配置
cd /opt/k8s/work/harbor/harbor vim harbor.yml修改内容如下
diff harbor.yml harbor.yml.bak 5c5 < hostname: 192.168.0.107 --- > hostname: reg.mydomain.com 8c8 < #http: --- > http: 10c10 < # port: 80 --- > port: 80 13c13 < https: --- > # https: 15c15 < port: 443 --- > # port: 443 17,18c17,18 < certificate: /etc/harbor/cert/harbor-server.pem < private_key: /etc/harbor/cert/harbor-server-key.pem --- > # certificate: /your/certificate/path > # private_key: /your/private/key/path 40c40 < data_volume: https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data --- > data_volume: https://img.qb5200.com/download-x/data生成配置文件（如果没有启动过，则跳过该步骤）
cd /opt/k8s/work/harbor/harbor ./prepare执行日志
prepare base dir is set to /opt/k8s/work/harbor/harbor Clearing the configuration file: /config/log/rsyslog_docker.conf Clearing the configuration file: /config/log/logrotate.conf Clearing the configuration file: /config/jobservice/config.yml Clearing the configuration file: /config/jobservice/env Clearing the configuration file: /confighttps://img.qb5200.com/download-x/db/env Clearing the configuration file: /config/registryctl/config.yml Clearing the configuration file: /config/registryctl/env Clearing the configuration file: /config/nginx/nginx.conf Clearing the configuration file: /config/registry/config.yml Clearing the configuration file: /config/registry/root.crt Clearing the configuration file: /config/core/app.conf Clearing the configuration file: /config/core/env Generated configuration file: /config/log/logrotate.conf Generated configuration file: /config/log/rsyslog_docker.conf Generated configuration file: /config/nginx/nginx.conf Generated configuration file: /config/core/env Generated configuration file: /config/core/app.conf Generated configuration file: /config/registry/config.yml Generated configuration file: /config/registryctl/env Generated configuration file: /confighttps://img.qb5200.com/download-x/db/env Generated configuration file: /config/jobservice/env Generated configuration file: /config/jobservice/config.yml loaded secret from file: /secret/keys/secretkey Generated configuration file: /compose_locationhttps://img.qb5200.com/download-x/docker-compose.yml Clean up the input dir修改生成文件的访问权限
cd /opt/k8s/work/harbor/harbor chmod -R 777 common启动harbor
如果是首次启动 执行
cd /opt/k8s/work/harbor/harbor mkdir -p https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data chmod 777 /var/runhttps://img.qb5200.com/download-x/docker.sock https://img.qb5200.com/download-x/data/k8s/harborhttps://img.qb5200.com/download-x/data ./install.sh否则执行
docker-compose up -d查看启动状态
root@master:/opt/k8s/work/harbor/harbor# docker-compose ps Name Command State Ports --------------------------------------------------------------------------------------------------------------- harbor-core /harbor/harbor_core Up (healthy) harbor-db https://img.qb5200.com/download-x/docker-entrypoint.sh Up (healthy) 5432/tcp harbor-jobservice /harbor/harbor_jobservice ... Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) 8080/tcp nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp redis redis-server /etc/redis.conf Up (healthy) 6379/tcp registry /entrypoint.sh /etc/regist ... Up (healthy) 5000/tcp registryctl /harbor/start.sh Up (healthy)- nginx 暴露了两个端口http的80和https的443，使用http访问80会自动重定向到https
浏览器访问 https://192.168.0.107,用账号 admin 和 harbor.yml 配置文件中的默认密码 Harbor12345 登陆系统。
- 其中的k8s是在http模式下创建的项目，直接启动https的可参照上面创建
docker命令拉取和上传镜像
如果之前已经执行过docker login，配置修改后需要重新docker login，需要删除 /root/.docker/config.json文件
如果在/etchttps://img.qb5200.com/download-x/dockerhttps://img.qb5200.com/download-x/daemon.json中配置过insecure-registries，需要去掉,并重新启动docker服务执行login
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. Error response from daemon: Get https://192.168.0.107/v2/: x509: certificate signed by unknown authority错误是说我们用的证书是自签名的证书，签发证书机构未经认证
解决方法是将签署 harbor-server 证书的 CA 证书拷贝到 /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107 目录下（需要在所有要访问harbor的节点上都执行此操作）root@slave:~# mkdir -p /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107 root@slave:~# scp root@192.168.0.107:/etc/harbor/cert/harbor-ca.pem /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107/ca.crt root@192.168.0.107's password: harbor-ca.pem 100% 1306 283.7KB/s 00:00 root@slave:~# ls /etchttps://img.qb5200.com/download-x/docker/certs.d/192.168.0.107/ ca.crt重新执行login
root@slave:~# docker login -u admin -p Harbor12345 192.168.0.107 WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https:/https://img.qb5200.com/download-x/docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded
上传镜像
root@slave:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE kubernetesui/metrics-scraper v1.0.3 3327f0dbcb4a 2 weeks ago 40.1MB busybox latest 6d5fcfe5ff17 7 weeks ago 1.22MB coredns/coredns 1.6.6 cc4d8e8c6169 2 months ago 40.8MB k8s.gcr.io/pause 3.1 da86e6ba6ca1 2 years ago 742kB 192.168.0.107/k8s/pause 3.1 da86e6ba6ca1 2 years ago 742kB kubeimage/pause 3.1 da86e6ba6ca1 2 years ago 742kB nginx 1.9.1 94ec7e53edfc 4 years ago 133MB root@slave:~# docker tag busybox:latest 192.168.0.107/k8s/busybox:latest root@slave:~# docker push 192.168.0.107/k8s/busybox:latest The push refers to repository [192.168.0.107/k8s/busybox] 195be5f8be1d: Pushed latest: digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 size: 527通过浏览器查看刚上传的镜像
从harbor下载镜像
root@slave:~# docker rmi 192.168.0.107/k8s/busybox:latest Untagged: 192.168.0.107/k8s/busybox:latest Untagged: 192.168.0.107/k8s/busybox@sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 root@slave:~# docker pull 192.168.0.107/k8s/busybox:latest latest: Pulling from k8s/busybox Digest: sha256:edafc0a0fb057813850d1ba44014914ca02d671ae247107ca70c94db686e7de6 Status: Downloaded newer image for 192.168.0.107/k8s/busybox:latest
kubernetes集群中使用私有的harbor镜像仓库
kubelet配置
kubernetes以pod为管理单元，而不是docker容器，在创建pod时，会先启动一个名为k8s.gcr.io/pause:3.1的镜像。
该镜像存在于google的镜像仓库中，国内不能直接访问，需要FQ下载，如果集群新加节点，或者节点上这个镜像被清理掉，将导致pod不能启动
一般做法是搭建私有镜像仓库，把这个镜像push到我们的私有镜像仓库，再配置kubelet使用我们私有仓库中的pause镜像
具体做法是修改kubelet的配置文件,配置文件的位置通过/etc/systemd/system/kubelet.service中的--config属性指定
例如--config=/etc/kubernetes/kubelet-config.yaml编辑配置文件/etc/kubernetes/kubelet-config.yaml ，追加--pod-infra-container-image属性值
--pod-infra-container-image: 192.168.0.107/k8s/pause:3.1重启kubelet服务
systemctl restart kubelet创建拉取镜像的secret
如果宿主机上执行过docker login，成功后会生成认证文件/root/.docker/config.json，可用如下命令生成secretkubectl create secret generic registrykey \ --from-file=.dockerconfigjson=/root/.docker/config.json \ --type=kubernetes.iohttps://img.qb5200.com/download-x/dockerconfigjson- registrykey是生成secret的名称
也可以直接在 kubectl中传入用户名和密码直接生成secret
kubectl create secret docker-registry registrykey --docker-server=192.168.0.107 --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@harbor.com- registrykey是生成secret的名称
- --docker-server 指定harbor的地址
- --docker-username 登陆harbor的用户名
- --docker-password 登陆harbor的密码
在pod中使用创建的secret
cat > private-busybox.yml << EOF apiVersion: v1 kind: Pod metadata: name: private-busybox spec: containers: - name: private-busybox image: 192.168.0.107/k8s/busybox:latest command: - sleep - "3600" imagePullSecrets: - name: registrykey EOF创建pod
root@master:/opt/k8s/yml# kubectl create -f private-busybox.yml pod/private-busybox created root@master:/opt/k8s/yml# kubectl get pod NAME READY STATUS RESTARTS AGE busybox 1/1 Running 56 2d9h nginx-deployment-56f8998dbc-6b6qm 1/1 Running 0 2d9h private-busybox 1/1 Running 0 5s
加载全部内容