kubernetes 1.6 集群实践 （一）

环境

hostname	ip
192.168.19.101	etcd-1,mater
192.168.19.102	etcd-2,node
192.168.19.103	etcd-3,node
192.168.19.104	node

配置tls认证证书及密钥

安装cfssl 工具

$ wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
$ chmod +x cfssl_linux-amd64
$ sudo mv cfssl_linux-amd64 /usr/local/bin/cfssl
$ wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
$ chmod +x cfssljson_linux-amd64
$ sudo mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
$ wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
$ chmod +x cfssl-certinfo_linux-amd64
$ sudo mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo

创建CA证书

$ mkdir /root/ssl
$ cd /root/ssl
$ cfssl print-defaults csr > ca-csr.json
#对CA证书签名请求修改为下
$ tee  ca-csr.json  <<-'EOF'               
{
"CN": "panjb-k8s",
"key": {
    "algo": "rsa",
    "size": 2048
    },
"names": [
    {
    "C": "CN",
    "ST": "SiChuan",
    "L": "chengdu",
    "O": "k8s",
    "OU": "System"
    }
        ]
}
EOF
$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca   #生产CA证书和私钥
$ ls
ca.csr  ca-csr.json  ca-key.pem  ca.pem

“CN”：Common Name，kube-apiserver 从证书中提取该字段作为请求的用户名 (User Name)；浏览器使用该字段验证网站是否合法；
“O”：Organization，kube-apiserver 从证书中提取该字段作为请求用户所属的组 (Group)；Group)

创建其他证书申请的CA配置文件

$ cfssl print-defaults config >ca-config.json
tee ca-config.json <<-'EOF'
{
  "signing": {
    "default": {
      "expiry": "87600h"
    },
    "profiles": {
      "panjb-k8s": {
        "usages": [
            "signing",
            "key encipherment",
            "server auth",
            "client auth"
        ],
        "expiry": "87600h"
      }
    }
  }
}
EOF

ca-config.json：可以定义多个 profiles，分别指定不同的过期时间、使用场景等参数；后续在签名证书时使用某个 profile；
signing：表示该证书可用于签名其它证书；生成的 ca.pem 证书中 CA=TRUE；
server auth：表示client可以用该 CA 对server提供的证书进行验证；
client auth：表示server可以用该CA对client提供的证书进行验证；

创建 kubernetes 证书

创建 kubernetes 证书签名请求

tee kubernetes-csr.json <<-'EOF'
{
    "CN": "panjb-k8s",
    "hosts": [
      "127.0.0.1",
      "192.168.19.101",
      "192.168.19.102",
      "192.168.19.103",
      "192.168.19.104",
      "10.254.0.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"
    ],
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
    {
    "C": "CN",
    "ST": "SiChuan",
    "L": "chengdu",
    "O": "k8s",
    "OU": "System"
    }
    ]
}
EOF

如果 hosts 字段不为空则需要指定授权使用该证书的 IP 或域名列表，由于该证书后续被 etcd 集群和 kubernetes master集群使用，所以上面分别指定了 etcd 集群、kubernetes master 集群的主机 IP 和 kubernetes 服务的服务 IP（一般是kue-apiserver 指定的 service-cluster-ip-range 网段的第一个IP，如 10.254.0.1。

生成 kubernetes 证书和私钥

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=panjb-k8s kubernetes-csr.json | cfssljson -bare kubernetes
$ ls kubernetes*
kubernetes.csr  kubernetes-csr.json  kubernetes-key.pem  kubernetes.pem

创建 admin 证书

创建 admin 证书签名请求

$ tee admin-csr.json <<-'EOF'
{
  "CN": "admin",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
    "C": "CN",
    "ST": "SiChuan",
    "L": "chengdu",
    "O": "system:masters",
    "OU": "System"
    }
  ]
}
EOF

后续 kube-apiserver 使用 RBAC 对客户端(如 kubelet、kube-proxy、Pod)请求进行授权；
kube-apiserver 预定义了一些 RBAC 使用的 RoleBindings，如 cluster-admin 将 Group system:masters 与 Rolecluster-admin 绑定，该 Role 授予了调用kube-apiserver 的所有 API的权限；
OU 指定该证书的 Group 为 system:masters，kubelet 使用该证书访问 kube-apiserver 时 ，由于证书被 CA 签名，所以认证通过，同时由于证书用户组为经过预授权的 system:masters，所以被授予访问所有 API 的权限；

生成 admin 证书和私钥

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=panjb-k8s admin-csr.json | cfssljson -bare admin
$ ls admin*
admin.csr  admin-csr.json  admin-key.pem  admin.pem

创建 kube-proxy 证书

创建 kube-proxy 证书签名请求

$ tee kube-proxy-csr.json <<-'EOF'
{
  "CN": "system:kube-proxy",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
    "C": "CN",
    "ST": "SiChuan",
    "L": "chengdu",
    "O": "k8s",
    "OU": "System"
    }
  ]
}
EOF

CN 指定该证书的 User 为 system:kube-proxy；
kube-apiserver 预定义的 RoleBinding cluster-admin 将User system:kube-proxy 与 Role system:node-proxier 绑定，该 Role 授予了调用 kube-apiserver Proxy 相关 API 的权限；

生成 kube-proxy 客户端证书和私钥

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=panjb-k8s  kube-proxy-csr.json | cfssljson -bare kube-proxy
$ ls kube-proxy*
kube-proxy.csr  kube-proxy-csr.json  kube-proxy-key.pem  kube-proxy.pem

校验证书

以 kubernetes 证书为例

使用 opsnssl 命令

[root@etcd-1 ssl]#  openssl x509  -noout -text -in  kubernetes.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            74:81:e7:d1:0f:8b:2b:f4:ac:72:2a:9f:b4:d7:ec:ce:65:10:c6:cf
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CN, ST=SiChuan, L=chengdu, O=k8s, OU=System, CN=panjb-k8s
        Validity
            Not Before: Oct 11 18:07:00 2017 GMT
            Not After : Oct  9 18:07:00 2027 GMT
        Subject: C=CN, ST=SiChuan, L=chengdu, O=k8s, OU=System, CN=panjb-k8s
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:c0:98:5d:fb:d1:67:ee:af:11:33:02:51:4c:7c:
                    e2:be:c6:8f:5c:a6:e3:3d:e6:b1:e1:46:eb:3b:84:
                    7b:1a:fe:a5:49:df:9e:34:67:0f:00:c1:c1:06:d6:
                    6e:63:7e:9a:9e:14:0c:be:58:ca:90:f3:30:8e:e4:
                    c0:07:49:66:f0:65:4d:e7:2d:3b:67:60:88:e9:6d:
                    1c:02:b3:6f:4d:c1:63:8f:85:d3:f0:d1:b5:0e:ac:
                    5a:94:9c:da:2f:dc:1f:e8:bd:be:49:59:ef:b0:24:
                    ae:84:da:d8:b6:8e:f1:52:5c:ce:87:b8:ce:77:20:
                    bf:f5:a2:1b:1f:a5:43:2d:18:43:d1:14:30:06:06:
                    ed:c8:4c:1f:f0:e0:20:be:87:ac:dd:3e:2c:2c:c6:
                    63:32:0a:0b:84:a5:1e:8d:cc:a2:59:77:4d:09:8c:
                    b1:0a:c0:56:50:6e:69:59:cf:e0:fd:33:cf:44:4c:
                    9c:7e:d0:9c:d8:58:23:9e:ae:41:ae:6e:7d:51:d4:
                    60:f7:9f:66:00:77:04:45:5e:78:f4:0f:72:bd:da:
                    f2:76:57:34:6a:c2:33:39:01:51:b3:eb:ed:89:c9:
                    f8:be:a0:f6:10:18:16:17:ce:f4:be:98:90:30:6e:
                    fb:05:39:81:17:a3:18:de:36:d6:ac:a9:cf:d7:44:
                    e2:2b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                96:FD:84:03:18:8C:D2:D8:99:28:25:08:94:5F:80:F8:64:BC:02:74
            X509v3 Authority Key Identifier: 
                keyid:A1:98:B3:41:BB:16:75:15:AC:CA:BB:39:5A:A2:55:57:F8:31:51:27

            X509v3 Subject Alternative Name: 
                DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster, DNS:kubernetes.default.svc.cluster.local, IP Address:127.0.0.1, IP Address:192.168.19.101, IP Address:192.168.19.102, IP Address:192.168.19.103, IP Address:192.168.19.104, IP Address:10.254.0.1
    Signature Algorithm: sha256WithRSAEncryption
         5a:21:75:06:0b:6b:bf:cc:c7:49:3a:c2:1b:7c:d4:e5:8d:80:
         c3:af:e1:a0:c3:ae:46:ad:c3:a6:45:af:ba:be:82:e8:ec:3c:
         4f:03:f9:89:66:24:ad:f1:c9:cd:01:d8:0b:46:f4:a0:50:00:
         36:b5:a5:11:6b:fd:b9:99:3a:b9:cb:be:71:05:b1:0c:09:75:
         7c:e9:46:2e:8d:29:61:45:40:23:dd:e6:3d:fa:e7:86:a7:f2:
         36:ed:c0:41:48:4c:51:74:c7:47:2b:9d:af:00:08:a1:fd:4d:
         d5:e4:57:64:9d:f1:55:1a:78:16:5f:c9:22:d3:26:27:cc:fa:
         a7:12:ae:1c:22:a0:e0:d3:8e:03:8d:82:9b:93:7d:c0:c2:71:
         fd:8c:6c:c2:54:4c:af:06:4b:70:82:21:a1:d1:5c:48:1c:32:
         b5:bc:8c:77:fd:6b:9e:04:a0:34:3f:23:c1:13:6f:ac:f7:12:
         7e:3c:6b:ed:99:9e:bc:0c:58:42:bd:f1:7f:ea:8b:1e:93:9c:
         e8:b6:e5:03:38:3e:da:a7:1e:19:1c:67:4a:98:6b:e0:e4:45:
         bf:91:32:4b:6e:1b:4a:d9:80:ef:72:65:0a:91:ff:af:ed:68:
         dc:ea:de:0c:12:61:ff:95:6d:46:14:73:f4:5e:b1:81:51:f2:
         96:3b:47:a8

确认 Issuer 字段的内容和 ca-csr.json 一致

确认 Subject 字段的内容和 kubernetes-csr.json 一致；

确认 X509v3 Subject Alternative Name 字段的内容和 kubernetes-csr.json 一致；

确认 X509v3 Key Usage、Extended Key Usage 字段的内容和 ca-config.json 中 kubernetes profile 一致；

使用 cfssl-certinfo 命令

$  cfssl-certinfo -cert kubernetes.pem
{
  "subject": {
    "common_name": "panjb-k8s",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "chengdu",
    "province": "SiChuan",
    "names": [
      "CN",
      "SiChuan",
      "chengdu",
      "k8s",
      "System",
      "panjb-k8s"
    ]
  },
  "issuer": {
    "common_name": "panjb-k8s",
    "country": "CN",
    "organization": "k8s",
    "organizational_unit": "System",
    "locality": "chengdu",
    "province": "SiChuan",
    "names": [
      "CN",
      "SiChuan",
      "chengdu",
      "k8s",
      "System",
      "panjb-k8s"
    ]
  },
  "serial_number": "665139919623901799018181161602228019860390069967",
  "sans": [
    "kubernetes",
    "kubernetes.default",
    "kubernetes.default.svc",
    "kubernetes.default.svc.cluster",
    "kubernetes.default.svc.cluster.local",
    "127.0.0.1",
    "192.168.19.101",
    "192.168.19.102",
    "192.168.19.103",
    "192.168.19.104",
    "10.254.0.1"
  ],
  "not_before": "2017-10-11T18:07:00Z",
  "not_after": "2027-10-09T18:07:00Z",
  "sigalg": "SHA256WithRSA",
  "authority_key_id": "A1:98:B3:41:BB:16:75:15:AC:CA:BB:39:5A:A2:55:57:F8:31:51:27",
  "subject_key_id": "96:FD:84:3:18:8C:D2:D8:99:28:25:8:94:5F:80:F8:64:BC:2:74",
  "pem": "-----BEGIN CERTIFICATE-----
MIIEiTCCA3GgAwIBAgIUdIHn0Q+LK/SsciqftNfszmUQxs8wDQYJKoZIhvcNAQEL
BQAwZDELMAkGA1UEBhMCQ04xEDAOBgNVBAgTB1NpQ2h1YW4xEDAOBgNVBAcTB2No
ZW5nZHUxDDAKBgNVBAoTA2s4czEPMA0GA1UECxMGU3lzdGVtMRIwEAYDVQQDEwlw
YW5qYi1rOHMwHhcNMTcxMDExMTgwNzAwWhcNMjcxMDA5MTgwNzAwWjBkMQswCQYD
VQQGEwJDTjEQMA4GA1UECBMHU2lDaHVhbjEQMA4GA1UEBxMHY2hlbmdkdTEMMAoG
A1UEChMDazhzMQ8wDQYDVQQLEwZTeXN0ZW0xEjAQBgNVBAMTCXBhbmpiLWs4czCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMCYXfvRZ+6vETMCUUx84r7G
j1ym4z3mseFG6zuEexr+pUnfnjRnDwDBwQbWbmN+mp4UDL5YypDzMI7kwAdJZvBl
TectO2dgiOltHAKzb03BY4+F0/DRtQ6sWpSc2i/cH+i9vklZ77AkroTa2LaO8VJc
zoe4zncgv/WiGx+lQy0YQ9EUMAYG7chMH/DgIL6HrN0+LCzGYzIKC4SlHo3Moll3
TQmMsQrAVlBuaVnP4P0zz0RMnH7QnNhYI56uQa5ufVHUYPefZgB3BEVeePQPcr3a
8nZXNGrCMzkBUbPr7YnJ+L6g9hAYFhfO9L6YkDBu+wU5gRejGN421qypz9dE4isC
AwEAAaOCATEwggEtMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcD
AQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUlv2EAxiM0tiZKCUI
lF+A+GS8AnQwHwYDVR0jBBgwFoAUoZizQbsWdRWsyrs5WqJVV/gxUScwga0GA1Ud
EQSBpTCBooIKa3ViZXJuZXRlc4ISa3ViZXJuZXRlcy5kZWZhdWx0ghZrdWJlcm5l
dGVzLmRlZmF1bHQuc3Zjgh5rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXKC
JGt1YmVybmV0ZXMuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbIcEfwAAAYcEwKgT
ZYcEwKgTZocEwKgTZ4cEwKgTaIcECv4AATANBgkqhkiG9w0BAQsFAAOCAQEAWiF1
Bgtrv8zHSTrCG3zU5Y2Aw6/hoMOuRq3DpkWvur6C6Ow8TwP5iWYkrfHJzQHYC0b0
oFAANrWlEWv9uZk6ucu+cQWxDAl1fOlGLo0pYUVAI93mPfrnhqfyNu3AQUhMUXTH
RyudrwAIof1N1eRXZJ3xVRp4Fl/JItMmJ8z6pxKuHCKg4NOOA42Cm5N9wMJx/Yxs
wlRMrwZLcIIhodFcSBwytbyMd/1rngSgND8jwRNvrPcSfjxr7ZmevAxYQr3xf+qL
HpOc6LblAzg+2qceGRxnSphr4ORFv5EyS24bStmA73JlCpH/r+1o3OreDBJh/5Vt
RhRz9F6xgVHyljtHqA==
-----END CERTIFICATE-----
"
}

将证书拷贝到全部服务器

创建目录

$ mkdir -p  /etc/kubernetes/ssl
$ cp ./*pem  /etc/kubernetes/ssl

参考资料

http://www.cnblogs.com/lykops/p/7465016.html