【四】K8s集群—HarBor 私有仓库部署

一、概述

前面我们把 K8s 集群部署好了，但是每次拉取镜像时都通过公有镜像仓库拉取非常慢，效率不高，于是我们需要在本地搭建一个私有镜像仓库来提供 K8s 集群使用，这样我们提交镜像和拉取镜像时就非常方便，速度也快。

Kubernetes 集群部署-kubeadm方式 这是前一篇 K8s 集群部署笔记。

二、准备工作

设置主机名

[root@localhost ~]# hostnamectl  set-hostname harbor

添加 Host 解析

[root@harbor ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.115.11 k8s-master01
192.168.115.12 k8s-node01
192.168.115.13 k8s-node02
192.168.115.14 hub.test.com

k8s 集群每个节点添加解析（注意：K8s 每个节点，不是 Harbor）

[root@k8s-master01 ~]# echo "192.168.115.14 hub.test.com" >> /etc/hosts
[root@k8s-node01 ~]# echo "192.168.115.14 hub.test.com" >> /etc/hosts
[root@k8s-node02 ~]# echo "192.168.115.14 hub.test.com" >> /etc/hosts

安装 Docker

[root@harbor ~]# yum install -y yum-utils device-mapper-persistent-data lvm2
[root@harbor ~]# yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
[root@harbor ~]# yum update -y && yum install -y docker-ce

创建 /etc/docker 目录

[root@harbor ~]# mkdir /etc/docker
# 配置 daemon.json
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
      "max-size": "100m"  
  },
  "insecure-registries": ["https://hub.test.com"]
}
EOF

[root@harbor ~]# mkdir -p /etc/systemd/system/docker.service.d

重启 docker 服务

[root@harbor ~]# systemctl daemon-reload && systemctl restart docker && systemctl enable docker

K8s 集群每个节点都需要添加（注意：K8s 每个节点，不是 Harbor）

[root@k8s-master01 ~]# cat /etc/docker/daemon.json 
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
      "max-size": "100m"  
  },	# 注意这里有个英文逗号！！！
  "insecure-registries": ["https://hub.test.com"]
}

三、创建 https 证书

安装 openssl

[root@harbor]# yum install openssl -y

创建证书目录，并赋予权限

[root@harbor ~]# mkdir -p /cert/harbor
[root@harbor ~]# chmod -R 777 /cert/harbor
[root@harbor ~]# cd /cert/harbor

创建服务器证书密钥文件 harbor.key

[root@harbor harbor]# openssl genrsa -des3 -out harbor.key 2048

输入密码，确认密码，自己随便定义，但是要记住，后面会用到。

创建服务器证书的申请文件 harbor.csr

[root@harbor harbor]# openssl req -new -key harbor.key -out harbor.csr

输入密钥文件的密码, 然后一路回车。

备份一份服务器密钥文件

[root@harbor harbor]# cp harbor.key harbor.key.org

去除文件口令

[root@harbor harbor]# openssl rsa -in harbor.key.org -out harbor.key

输入密钥文件的密码

创建一个自当前日期起为期十年的证书 harbor.crt

[root@harbor harbor]# openssl x509 -req -days 3650 -in harbor.csr -signkey harbor.key -out harbor.crt

四、安装 Harbor

下载 Harbor 包

下载链接：https://github.com/goharbor/harbor/releases

解压并安装 harbor

[root@harbor ~]# tar -zxvf harbor-offline-installer-v2.2.2.tgz
[root@harbor ~]# mv harbor /usr/local/
[root@harbor ~]# cd /usr/local/harbor/

修改配置文件

[root@harbor ~]# cd /usr/local/harbor/
[root@harbor harbor]# ls
common.sh  harbor.v2.2.2.tar.gz  harbor.yml.tmpl  install.sh  LICENSE  prepare
[root@harbor harbor]# cp harbor.yml.tmpl harbor.yml
[root@harbor harbor]# vim harbor.yml

需要修改的地方如下，其他的默认即可

将http端口改成10080，因为默认用的80端口已经被占用，http可以指定任意端口；

开始安装 harbor

[root@harbor harbor]# ./install.sh 
[Step 0]: checking if docker is installed ...

Note: docker version: 20.10.6

[Step 1]: checking docker-compose is installed ...

Note: docker-compose version: 1.23.1

[Step 2]: loading Harbor images ...
Loaded image: goharbor/harbor-jobservice:v2.2.2
Loaded image: goharbor/harbor-exporter:v2.2.2
Loaded image: goharbor/nginx-photon:v2.2.2
Loaded image: goharbor/trivy-adapter-photon:v2.2.2
Loaded image: goharbor/prepare:v2.2.2
Loaded image: goharbor/harbor-db:v2.2.2
Loaded image: goharbor/harbor-registryctl:v2.2.2
Loaded image: goharbor/notary-server-photon:v2.2.2
Loaded image: goharbor/notary-signer-photon:v2.2.2
Loaded image: goharbor/redis-photon:v2.2.2
Loaded image: goharbor/registry-photon:v2.2.2
Loaded image: goharbor/chartmuseum-photon:v2.2.2
Loaded image: goharbor/harbor-portal:v2.2.2
Loaded image: goharbor/harbor-core:v2.2.2
Loaded image: goharbor/harbor-log:v2.2.2
...省略部分输出信息
[Step 5]: starting Harbor ...
Creating network "harbor_harbor" with the default driver
Creating harbor-log ... done
Creating registryctl   ... done
Creating harbor-db     ... done
Creating harbor-portal ... done
Creating redis         ... done
Creating registry      ... done
Creating harbor-core   ... done
Creating nginx             ... done
Creating harbor-jobservice ... done
✔ ----Harbor has been installed and started successfully.----

测试 Harbor，在 k8s 集群节点中下载镜像推送到 Harbor（注意：K8s 某个节点测试）

登录 Harbor

[root@k8s-master01 ~]# docker login https://hub.test.com
Username: admin
Password: Harbor12345 # 默认密码，可通过 harbor.yml 配置文件修改

下载镜像推送到 Harbor

[root@k8s-node01 ~]# docker pull nginx
[root@k8s-node01 ~]# docker tag nginx:latest hub.test.com/library/mynginx:v1
[root@k8s-node01 ~]# docker push hub.test.com/library/mynginx:v1

创建 Pod 测试

[root@k8s-master01 ~]# kubectl run nginx-deployment --image=hub.test.com/library/mynginx:v1 --port=80
[root@k8s-master01 ~]# kubectl get svc
NAME               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
kubernetes         ClusterIP   10.96.0.1      <none>        443/TCP   131m
nginx-deployment   ClusterIP   10.102.181.9   <none>        80/TCP    9m27s
[root@k8s-master01 ~]# kubectl get pod -o wide
NAME               READY   STATUS    RESTARTS   AGE     IP           NODE         NOMINATED NODE   READINESS GATES
nginx-deployment   1/1     Running   0          9m46s   10.244.2.3   k8s-node02   <none>           <none>

访问 Pod 测试

[root@k8s-master01 ~]# curl -i 10.244.2.3
HTTP/1.1 200 OK
Server: nginx/1.19.10
Date: Mon, 24 May 2021 08:55:12 GMT
...

五、Windows 访问 Harbor Web界面

Windows 添加 hosts 解析路径

C:WindowsSystem32driversetchosts

添加信息

192.168.115.14 hub.test.com

浏览器访问测试

https://hub.test.com

用户密码：admin / Harbor12345

可以看到这是刚刚我们在 K8s 集群推送的 nginx 镜像，下载次数为1，刚刚构建 Pod 的时候拉取了这个镜像。