环境
| hostname | ip |
|---|---|
| 192.168.19.101 | etcd-1,mater |
| 192.168.19.102 | etcd-2,node |
| 192.168.19.103 | etcd-3,node |
| 192.168.19.104 | node |
配置tls认证证书及密钥
安装cfssl 工具
$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ chmod +x cfssl_linux-amd64
$ sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x cfssljson_linux-amd64
$ sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
$ chmod +x cfssl-certinfo_linux-amd64
$ sudo mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
创建CA证书
$ mkdir /root/ssl
$ cd /root/ssl
$ cfssl print-defaults csr > ca-csr.json
#对CA证书签名请求修改为下
$ tee ca-csr.json <<-'EOF'
{
"CN": "panjb-k8s",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SiChuan",
"L": "chengdu",
"O": "k8s",
"OU": "System"
}
]
}
EOF
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca #生产CA证书和私钥
$ ls
ca.csr ca-csr.json ca-key.pem ca.pem
“CN”：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；
“O”：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；Group)
创建其他证书申请的CA配置文件
$ cfssl print-defaults config >ca-config.json
tee ca-config.json <<-'EOF'
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"panjb-k8s": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示client可以用该 CA 对server提供的证书进行验证；
client auth：表示server可以用该CA对client提供的证书进行验证；
创建 kubernetes 证书
创建 kubernetes 证书签名请求
tee kubernetes-csr.json <<-'EOF'
{
"CN": "panjb-k8s",
"hosts": [
"127.0.0.1",
"192.168.19.101",
"192.168.19.102",
"192.168.19.103",
"192.168.19.104",
"10.254.0.1",
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SiChuan",
"L": "chengdu",
"O": "k8s",
"OU": "System"
}
]
}
EOF
如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表，由于该证书后续被 etcd 集群和 kubernetes master集群使用，所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP（一般是kue-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.254.0.1。
生成 kubernetes 证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=panjb-k8s kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kubernetes*
kubernetes.csr kubernetes-csr.json kubernetes-key.pem kubernetes.pem
创建 admin 证书
创建 admin 证书签名请求
$ tee admin-csr.json <<-'EOF'
{
"CN": "admin",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SiChuan",
"L": "chengdu",
"O": "system:masters",
"OU": "System"
}
]
}
EOF
后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；
kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Rolecluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；
OU 指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；
生成 admin 证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=panjb-k8s admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr admin-csr.json admin-key.pem admin.pem
创建 kube-proxy 证书
创建 kube-proxy 证书签名请求
$ tee kube-proxy-csr.json <<-'EOF'
{
"CN": "system:kube-proxy",
"hosts": [],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "SiChuan",
"L": "chengdu",
"O": "k8s",
"OU": "System"
}
]
}
EOF
CN 指定该证书的 User 为 system:kube-proxy；
kube-apiserver 预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；
生成 kube-proxy 客户端证书和私钥
$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=panjb-k8s kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr kube-proxy-csr.json kube-proxy-key.pem kube-proxy.pem
校验证书
以 kubernetes 证书为例
使用 opsnssl 命令
[root@etcd-1 ssl]# openssl x509 -noout -text -in kubernetes.pem
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
74:81:e7:d1:0f:8b:2b:f4:ac:72:2a:9f:b4:d7:ec:ce:65:10:c6:cf
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=CN, ST=SiChuan, L=chengdu, O=k8s, OU=System, CN=panjb-k8s
Validity
Not Before: Oct 11 18:07:00 2017 GMT
Not After : Oct 9 18:07:00 2027 GMT
Subject: C=CN, ST=SiChuan, L=chengdu, O=k8s, OU=System, CN=panjb-k8s
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (2048 bit)
Modulus:
00:c0:98:5d:fb:d1:67:ee:af:11:33:02:51:4c:7c:
e2:be:c6:8f:5c:a6:e3:3d:e6:b1:e1:46:eb:3b:84:
7b:1a:fe:a5:49:df:9e:34:67:0f:00:c1:c1:06:d6:
6e:63:7e:9a:9e:14:0c:be:58:ca:90:f3:30:8e:e4:
c0:07:49:66:f0:65:4d:e7:2d:3b:67:60:88:e9:6d:
1c:02:b3:6f:4d:c1:63:8f:85:d3:f0:d1:b5:0e:ac:
5a:94:9c:da:2f:dc:1f:e8:bd:be:49:59:ef:b0:24:
ae:84:da:d8:b6:8e:f1:52:5c:ce:87:b8:ce:77:20:
bf:f5:a2:1b:1f:a5:43:2d:18:43:d1:14:30:06:06:
ed:c8:4c:1f:f0:e0:20:be:87:ac:dd:3e:2c:2c:c6:
63:32:0a:0b:84:a5:1e:8d:cc:a2:59:77:4d:09:8c:
b1:0a:c0:56:50:6e:69:59:cf:e0:fd:33:cf:44:4c:
9c:7e:d0:9c:d8:58:23:9e:ae:41:ae:6e:7d:51:d4:
60:f7:9f:66:00:77:04:45:5e:78:f4:0f:72:bd:da:
f2:76:57:34:6a:c2:33:39:01:51:b3:eb:ed:89:c9:
f8:be:a0:f6:10:18:16:17:ce:f4:be:98:90:30:6e:
fb:05:39:81:17:a3:18:de:36:d6:ac:a9:cf:d7:44:
e2:2b
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
96:FD:84:03:18:8C:D2:D8:99:28:25:08:94:5F:80:F8:64:BC:02:74
X509v3 Authority Key Identifier:
keyid:A1:98:B3:41:BB:16:75:15:AC:CA:BB:39:5A:A2:55:57:F8:31:51:27
X509v3 Subject Alternative Name:
DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.19.101, IP Address:192.168.19.102, IP Address:192.168.19.103, IP Address:192.168.19.104, IP Address:10.254.0.1
Signature Algorithm: sha256WithRSAEncryption
5a:21:75:06:0b:6b:bf:cc:c7:49:3a:c2:1b:7c:d4:e5:8d:80:
c3:af:e1:a0:c3:ae:46:ad:c3:a6:45:af:ba:be:82:e8:ec:3c:
4f:03:f9:89:66:24:ad:f1:c9:cd:01:d8:0b:46:f4:a0:50:00:
36:b5:a5:11:6b:fd:b9:99:3a:b9:cb:be:71:05:b1:0c:09:75:
7c:e9:46:2e:8d:29:61:45:40:23:dd:e6:3d:fa:e7:86:a7:f2:
36:ed:c0:41:48:4c:51:74:c7:47:2b:9d:af:00:08:a1:fd:4d:
d5:e4:57:64:9d:f1:55:1a:78:16:5f:c9:22:d3:26:27:cc:fa:
a7:12:ae:1c:22:a0:e0:d3:8e:03:8d:82:9b:93:7d:c0:c2:71:
fd:8c:6c:c2:54:4c:af:06:4b:70:82:21:a1:d1:5c:48:1c:32:
b5:bc:8c:77:fd:6b:9e:04:a0:34:3f:23:c1:13:6f:ac:f7:12:
7e:3c:6b:ed:99:9e:bc:0c:58:42:bd:f1:7f:ea:8b:1e:93:9c:
e8:b6:e5:03:38:3e:da:a7:1e:19:1c:67:4a:98:6b:e0:e4:45:
bf:91:32:4b:6e:1b:4a:d9:80:ef:72:65:0a:91:ff:af:ed:68:
dc:ea:de:0c:12:61:ff:95:6d:46:14:73:f4:5e:b1:81:51:f2:
96:3b:47:a8
确认 Issuer 字段的内容和 ca-csr.json 一致
确认 Subject 字段的内容和 kubernetes-csr.json 一致；
确认 X509v3 Subject Alternative Name 字段的内容和 kubernetes-csr.json 一致；
确认 X509v3 Key Usage、Extended Key Usage 字段的内容和 ca-config.json 中 kubernetes profile 一致；
使用 cfssl-certinfo 命令
$ cfssl-certinfo -cert kubernetes.pem
{
"subject": {
"common_name": "panjb-k8s",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "chengdu",
"province": "SiChuan",
"names": [
"CN",
"SiChuan",
"chengdu",
"k8s",
"System",
"panjb-k8s"
]
},
"issuer": {
"common_name": "panjb-k8s",
"country": "CN",
"organization": "k8s",
"organizational_unit": "System",
"locality": "chengdu",
"province": "SiChuan",
"names": [
"CN",
"SiChuan",
"chengdu",
"k8s",
"System",
"panjb-k8s"
]
},
"serial_number": "665139919623901799018181161602228019860390069967",
"sans": [
"kubernetes",
"kubernetes.default",
"kubernetes.default.svc",
"kubernetes.default.svc.cluster",
"kubernetes.default.svc.cluster.local",
"127.0.0.1",
"192.168.19.101",
"192.168.19.102",
"192.168.19.103",
"192.168.19.104",
"10.254.0.1"
],
"not_before": "2017-10-11T18:07:00Z",
"not_after": "2027-10-09T18:07:00Z",
"sigalg": "SHA256WithRSA",
"authority_key_id": "A1:98:B3:41:BB:16:75:15:AC:CA:BB:39:5A:A2:55:57:F8:31:51:27",
"subject_key_id": "96:FD:84:3:18:8C:D2:D8:99:28:25:8:94:5F:80:F8:64:BC:2:74",
"pem": "-----BEGIN CERTIFICATE-----
MIIEiTCCA3GgAwIBAgIUdIHn0Q+LK/SsciqftNfszmUQxs8wDQYJKoZIhvcNAQEL
BQAwZDELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB1NpQ2h1YW4xEDAOBgNVBAcTB2No
ZW5nZHUxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRIwEAYDVQQDEwlw
YW5qYi1rOHMwHhcNMTcxMDExMTgwNzAwWhcNMjcxMDA5MTgwNzAwWjBkMQswCQYD
VQQGEwJDTjEQMA4GA1UECBMHU2lDaHVhbjEQMA4GA1UEBxMHY2hlbmdkdTEMMAoG
A1UEChMDazhzMQ8wDQYDVQQLEwZTeXN0ZW0xEjAQBgNVBAMTCXBhbmpiLWs4czCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMCYXfvRZ+6vETMCUUx84r7G
j1ym4z3mseFG6zuEexr+pUnfnjRnDwDBwQbWbmN+mp4UDL5YypDzMI7kwAdJZvBl
TectO2dgiOltHAKzb03BY4+F0/DRtQ6sWpSc2i/cH+i9vklZ77AkroTa2LaO8VJc
zoe4zncgv/WiGx+lQy0YQ9EUMAYG7chMH/DgIL6HrN0+LCzGYzIKC4SlHo3Moll3
TQmMsQrAVlBuaVnP4P0zz0RMnH7QnNhYI56uQa5ufVHUYPefZgB3BEVeePQPcr3a
8nZXNGrCMzkBUbPr7YnJ+L6g9hAYFhfO9L6YkDBu+wU5gRejGN421qypz9dE4isC
AwEAAaOCATEwggEtMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUlv2EAxiM0tiZKCUI
lF+A+GS8AnQwHwYDVR0jBBgwFoAUoZizQbsWdRWsyrs5WqJVV/gxUScwga0GA1Ud
EQSBpTCBooIKa3ViZXJuZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0ghZrdWJlcm5l
dGVzLmRlZmF1bHQuc3Zjgh5rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXKC
JGt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbIcEfwAAAYcEwKgT
ZYcEwKgTZocEwKgTZ4cEwKgTaIcECv4AATANBgkqhkiG9w0BAQsFAAOCAQEAWiF1
Bgtrv8zHSTrCG3zU5Y2Aw6/hoMOuRq3DpkWvur6C6Ow8TwP5iWYkrfHJzQHYC0b0
oFAANrWlEWv9uZk6ucu+cQWxDAl1fOlGLo0pYUVAI93mPfrnhqfyNu3AQUhMUXTH
RyudrwAIof1N1eRXZJ3xVRp4Fl/JItMmJ8z6pxKuHCKg4NOOA42Cm5N9wMJx/Yxs
wlRMrwZLcIIhodFcSBwytbyMd/1rngSgND8jwRNvrPcSfjxr7ZmevAxYQr3xf+qL
HpOc6LblAzg+2qceGRxnSphr4ORFv5EyS24bStmA73JlCpH/r+1o3OreDBJh/5Vt
RhRz9F6xgVHyljtHqA==
-----END CERTIFICATE-----
"
}
将证书拷贝到全部服务器
创建目录
$ mkdir -p /etc/kubernetes/ssl
$ cp ./*pem /etc/kubernetes/ssl