分析kube-proxy的iptables规则

NodePort service

创建一个mysql的NodePort服务，对应两个pod实例，rc和service的配置如下：

1、rc配置

apiVersion: v1
kind: ReplicationController
metadata:
  name: wordpress-mysql
spec:
  replicas: 2
  selector:
    name: wordpress-mysql
  template:
    metadata:
      labels:
        name: wordpress-mysql
    spec:
      containers:
        - name: wordpress-mysql
          image: 172.16.114.201/library/mysql:v1
          ports:
            - containerPort: 3306
          volumeMounts:
            - name: "wordpress-mysql-data"
              mountPath: "/var/lib/mysql"
          env:
          - name: MYSQL_PASS
            value: "123456"
          - name: ON_CREATE_DB
            value: "wordpress"
      volumes:
        - name: "wordpress-mysql-data"
          hostPath:
            path: "/root/wordpress-mysql/data"

2、service配置

apiVersion: v1
kind: Service
metadata:
  name: wordpress-mysql
spec:
  ports:
    - port: 3306
      targetPort: 3306
      nodePort: 30010
      protocol: TCP
  type: NodePort
  selector:
    name: wordpress-mysql

3、创建的service情况

Name:            wordpress-mysql
Namespace:        default
Labels:            <none>
Selector:        name=wordpress-mysql
Type:            NodePort
IP:            10.254.67.85
Port:            <unset>    3306/TCP
NodePort:        <unset>    30010/TCP
Endpoints:        10.0.3.2:3306,10.0.45.6:3306
Session Affinity:    None
No events.

4、kube-proxy占用端口情况

[root@test-209 log]# netstat -anp | grep kube-proxy
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      10165/kube-proxy
tcp        0      0 172.16.114.209:46010    172.16.114.208:8080     ESTABLISHED 10165/kube-proxy
tcp        0      0 172.16.114.209:46014    172.16.114.208:8080     ESTABLISHED 10165/kube-proxy
tcp        0      0 172.16.114.209:46012    172.16.114.208:8080     ESTABLISHED 10165/kube-proxy
tcp6       0      0 :::30010                :::*                    LISTEN      10165/kube-proxy
unix  2      [ ]         DGRAM                    36395    10165/kube-proxy
unix  3      [ ]         STREAM     CONNECTED     36403    10165/kube-proxy

5、对应的iptables规则

iptables -S -t nat | grep mysql
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport 30010 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp --dport 30010 -j KUBE-SVC-GJ6HULPZPPQIKMS7
-A KUBE-SEP-7KXQQUXVSZ2LFV44 -s 10.0.45.6/32 -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ
-A KUBE-SEP-7KXQQUXVSZ2LFV44 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.45.6:3306
-A KUBE-SEP-J7SZJXRP24HRFT23 -s 10.0.3.2/32 -m comment --comment "default/wordpress-mysql:" -j KUBE-MARK-MASQ
-A KUBE-SEP-J7SZJXRP24HRFT23 -p tcp -m comment --comment "default/wordpress-mysql:" -m tcp -j DNAT --to-destination 10.0.3.2:3306
-A KUBE-SERVICES -d 10.254.67.85/32 -p tcp -m comment --comment "default/wordpress-mysql: cluster IP" -m tcp --dport 3306 -j KUBE-SVC-GJ6HULPZPPQIKMS7
-A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-J7SZJXRP24HRFT23
-A KUBE-SVC-GJ6HULPZPPQIKMS7 -m comment --comment "default/wordpress-mysql:" -j KUBE-SEP-7KXQQUXVSZ2LFV44

从以上信息看出，kube-proxy为mysql服务在node节点上单独起了一个端口30010，在iptables的规则中，目的端口30010被指向KUBE-SVC-GJ6HULPZPPQIKMS7，KUBE-SVC-GJ6HULPZPPQIKMS7又被指向KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44（他两各50%的几率），KUBE-SEP-J7SZJXRP24HRFT23和KUBE-SEP-7KXQQUXVSZ2LFV44定义了DNAT转换规则，将访问重定向到10.0.45.6:3306和10.0.3.2:3306这两个endpoint。因此，当外部访问30010端口时，根据iptables的规则会将该消息分发给10.0.45.6:3306和10.0.3.2:3306这两个地址（分发的几率是各50%）

ClusterIP service

创建一个zookeeper的ClusterIP服务，rc和service的配置如下：

1、rc配置

apiVersion: v1
kind: ReplicationController
metadata:
  name: zookeeper1
spec:
  replicas: 1
  selector:
    name: zookeeper1
  template:
    metadata:
      labels:
        name: zookeeper1
    spec:
      containers:
        - name: zookeeper1
          image: 10.10.30.166/public/zookeeper:v1
          ports:
            - containerPort: 2181
            - containerPort: 2888
            - containerPort: 3888
          env:
          - name: ZOOKEEPER_ID
            value: "1"
          - name: ZOOKEEPER_SERVER_1
            value: "zookeeper1"
          - name: ZOOKEEPER_SERVER_2
            value: "zookeeper2"
          - name: ZOOKEEPER_SERVER_3
            value: "zookeeper3"

2、service配置

apiVersion: v1
kind: Service
metadata:
  name: zookeeper1
spec:
  ports:
    - port: 2181
      targetPort: 2181
      protocol: TCP
      name: "1"
    - port: 2888
      targetPort: 2888
      protocol: TCP
      name: "2"
    - port: 3888
      targetPort: 3888
      protocol: TCP
      name: "3"
  type: ClusterIP
  selector:
    name: zookeeper1

3、创建service情况

Name:            zookeeper1
Namespace:        default
Labels:            <none>
Selector:        name=zookeeper1
Type:            ClusterIP
IP:            10.254.181.6
Port:            1    2181/TCP
Endpoints:        10.0.45.4:2181
Port:            2    2888/TCP
Endpoints:        10.0.45.4:2888
Port:            3    3888/TCP
Endpoints:        10.0.45.4:3888
Session Affinity:    None
No events.

4、iptables规则

iptables -S -t nat | grep zookeeper1
-A KUBE-SEP-BZJZKIUQRVYJVMQB -s 10.0.45.4/32 -m comment --comment "default/zookeeper1:3" -j KUBE-MARK-MASQ
-A KUBE-SEP-BZJZKIUQRVYJVMQB -p tcp -m comment --comment "default/zookeeper1:3" -m tcp -j DNAT --to-destination 10.0.45.4:3888
-A KUBE-SEP-C3J2QHMJ3LTD3GR7 -s 10.0.45.4/32 -m comment --comment "default/zookeeper1:2" -j KUBE-MARK-MASQ
-A KUBE-SEP-C3J2QHMJ3LTD3GR7 -p tcp -m comment --comment "default/zookeeper1:2" -m tcp -j DNAT --to-destination 10.0.45.4:2888
-A KUBE-SEP-RZ4H7H2HFI3XFCXZ -s 10.0.45.4/32 -m comment --comment "default/zookeeper1:1" -j KUBE-MARK-MASQ
-A KUBE-SEP-RZ4H7H2HFI3XFCXZ -p tcp -m comment --comment "default/zookeeper1:1" -m tcp -j DNAT --to-destination 10.0.45.4:2181
-A KUBE-SERVICES -d 10.254.181.6/32 -p tcp -m comment --comment "default/zookeeper1:1 cluster IP" -m tcp --dport 2181 -j KUBE-SVC-HHEJUKXW5P7DV7BX
-A KUBE-SERVICES -d 10.254.181.6/32 -p tcp -m comment --comment "default/zookeeper1:2 cluster IP" -m tcp --dport 2888 -j KUBE-SVC-2SVOYTXLXAXVV7L3
-A KUBE-SERVICES -d 10.254.181.6/32 -p tcp -m comment --comment "default/zookeeper1:3 cluster IP" -m tcp --dport 3888 -j KUBE-SVC-KAVJ7GO67HRSOAM3
-A KUBE-SVC-2SVOYTXLXAXVV7L3 -m comment --comment "default/zookeeper1:2" -j KUBE-SEP-C3J2QHMJ3LTD3GR7
-A KUBE-SVC-HHEJUKXW5P7DV7BX -m comment --comment "default/zookeeper1:1" -j KUBE-SEP-RZ4H7H2HFI3XFCXZ
-A KUBE-SVC-KAVJ7GO67HRSOAM3 -m comment --comment "default/zookeeper1:3" -j KUBE-SEP-BZJZKIUQRVYJVMQB

从iptables的规则来看，对目的ip是10.254.181.6，端口是2181、2888或者3888的消息，规则指向了KUBE-SVC-HHEJUKXW5P7DV7BX、KUBE-SVC-2SVOYTXLXAXVV7L3、KUBE-SVC-KAVJ7GO67HRSOAM3，他们三又分别指向了KUBE-SEP-C3J2QHMJ3LTD3GR7、KUBE-SEP-RZ4H7H2HFI3XFCXZ、KUBE-SEP-BZJZKIUQRVYJVMQB，这三条规则定义了DNAT转换规则，将访问重定向到了10.0.45.4:3888、10.0.45.4:2888、10.0.45.4:2181