docker使用 Flannel（etcd+flannel）网络

一、Flannel网络简介

Flannel是一种基于overlay网络的跨主机容器网络解决方案，也就是将TCP数据包封装在另一种网络包里面进行路由转发和通信，Flannel是CoreOS开发,专门用于docker多机互联的一个工具,让集群中的不同节点主机创建的容器都具有全集群唯一的虚拟ip地址，Flannel使用go语言编写

二、Flannel实现原理

2.1、原理说明

1、Flannel为每个host分配一个subnet，容器从这个subnet中分配IP，这些IP可以在host间路由，容器间无需使用nat和端口映射即可实现跨主机通信

2、每个subnet都是从一个更大的IP池中划分的，flannel会在每个主机上运行一个叫flanneld的agent，其职责就是从池子中分配subnet

3、Flannel使用etcd存放网络配置、已分配 的subnet、host的IP等信息

4、Flannel数据包在主机间转发是由backend实现的，目前已经支持UDP、VxLAN、host-gw、AWS VPC和GCE路由等多种backend

2.2、数据转发流程

1、容器直接使用目标容器的ip访问，默认通过容器内部的eth0发送出去。

2、报文通过veth pair被发送到vethXXX。

3、ethXXX是直接连接到虚拟交换机docker0的，报文通过虚拟bridge docker0发送出去。

4、查找路由表，外部容器ip的报文都会转发到flannel0虚拟网卡，这是一个P2P的虚拟网卡，然后报文就被转发到监听在另一端的flanneld。

5、flanneld通过etcd维护了各个节点之间的路由表，把原来的报文UDP封装一层，通过配置的iface发送出去。

6、报文通过主机之间的网络找到目标主机。

7、报文继续往上，到传输层，交给监听在8285端口的flanneld程序处理。

8、数据被解包，然后发送给flannel0虚拟网卡。

9、查找路由表，发现对应容器的报文要交给docker0。

10、docker0找到连到自己的容器，把报文发送过去。

三、部署etcd集群

3.1、环境准备

节点名称	IP地址	安装软件
node1	192.168.0.115	etcd
node2	192.168.0.116	etcd
node3	192.168.0.117	etcd

3.2、安装etcd

# yum -y install etcd

3.3、配置etcd

# cp /etc/etcd/etcd.conf{,_bak}

【注释：每个ETCD_NAME必须不同，绿色部分的ip为当前宿主机的ip】
# grep -v '^#' /etc/etcd/etcd.conf
ETCD_NAME="node1"
ETCD_DATA_DIR="/var/lib/etcd/node1.etcd"
ETCD_LISTEN_PEER_URLS="http://192.168.0.115:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.0.115:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.0.115:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.0.115:2379"
ETCD_INITIAL_CLUSTER="node1=http://192.168.0.115:2380,node2=http://192.168.0.116:2380,node3=http://192.168.0.117:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"

3.4、修改etcd启动文件

# cp /usr/lib/systemd/system/etcd.service{,_bak}

# cat /usr/lib/systemd/system/etcd.service
[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
User=etcd
# set GOMAXPROCS to number of processors
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /usr/bin/etcd 
--name="${ETCD_NAME}" 
--data-dir="${ETCD_DATA_DIR}" 
--listen-peer-urls="${ETCD_LISTEN_PEER_URLS}" 
--listen-client-urls="${ETCD_LISTEN_CLIENT_URLS}" 
--initial-advertise-peer-urls="${ETCD_INITIAL_ADVERTISE_PEER_URLS}" 
--advertise-client-urls="${ETCD_ADVERTISE_CLIENT_URLS}" 
--initial-cluster="${ETCD_INITIAL_CLUSTER}"  
--initial-cluster-token="${ETCD_INITIAL_CLUSTER_TOKEN}" 
--initial-cluster-state="${ETCD_INITIAL_CLUSTER_STATE}""
Restart=on-failure
LimitNOFILE=65536

3.5、启动etcd服务

【注释：另外两台服务器，操作也如上】
# systemctl start etcd.service

3.6、检测etcd集群状态，至此etcd安装完成

# 查看cluster状态
# etcdctl cluster-health
member 3e398d43ae9c8720 is healthy: got healthy result from http://192.168.0.116:2379
member 65368524050cc2e8 is healthy: got healthy result from http://192.168.0.115:2379
member d8ff06c8c9b413da is healthy: got healthy result from http://192.168.0.117:2379
cluster is healthy

# 列出etcd服务状态，从列出信息可以看出，目前是node2为主节点。
# etcdctl member list
3e398d43ae9c8720: name=node2 peerURLs=http://192.168.0.116:2380 clientURLs=http://192.168.0.116:2379 isLeader=true
65368524050cc2e8: name=node1 peerURLs=http://192.168.0.115:2380 clientURLs=http://192.168.0.115:2379 isLeader=false
d8ff06c8c9b413da: name=node3 peerURLs=http://192.168.0.117:2380 clientURLs=http://192.168.0.117:2379 isLeader=false

3.7、添加flannel网络配置信息到etcd

【注释： 此（flannel_use）目录自己可以定义，但是此处设置的目录必须与flannel配置文件中FLANNEL_ETCD_PREFIX="/flannel_use/network"配置保持一致，flannel启动程序只认带“config”的key，否则会报错Not a directory (/flannel_use/network)】 

# 固定配置方式
# etcdctl set /flannel_use/network/config '{"Network":"10.10.0.0/16"}'

四、部署flannel

4.1、安装flannel

# yum install -y flannel

4.2、修改flannel配置文件

# cp /etc/sysconfig/flanneld{,_bak}

# cat /etc/sysconfig/flanneld
# Flanneld configuration options  
# etcd url location.  Point this to the server where etcd runs
FLANNEL_ETCD_ENDPOINTS="http://192.168.0.115:2379,http://192.168.0.116:2379,http://192.168.0.117:2379"
# etcd config key.  This is the configuration key that flannel queries
# For address range assignment
FLANNEL_ETCD_PREFIX="/flannel_use/network"
# Any additional options that you want to pass
#FLANNEL_OPTIONS=""

4.3、启动flannel

# systemctl start flanneld


# systemctl status flanneld
● flanneld.service - Flanneld overlay address etcd agent
   Loaded: loaded (/usr/lib/systemd/system/flanneld.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2019-12-23 15:01:07 CST; 4s ago
  Process: 1828 ExecStartPost=/usr/libexec/flannel/mk-docker-opts.sh -k DOCKER_NETWORK_OPTIONS -d /run/flannel/docker (code=exited, status=0/SUCCESS)
 Main PID: 1817 (flanneld)
   Memory: 18.8M
   CGroup: /system.slice/flanneld.service
           └─1817 /usr/bin/flanneld -etcd-endpoints=http://192.168.0.115:2379,http://192.168.0.116:2379,http://192.168.0.117:2379 -etcd-prefix=/flannel_use/network

Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.365994    1817 main.go:132] Installing signal handlers
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366705    1817 manager.go:136] Determining IP address of default interface
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366916    1817 manager.go:149] Using interface with name eth0 and address 192.168.0.109
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.366933    1817 manager.go:166] Defaulting external address to interface address (192.168.0.109)
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.375600    1817 local_manager.go:179] Picking subnet in range 10.10.1.0 ... 10.10.255.0
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.383110    1817 manager.go:250] Lease acquired: 10.10.88.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.383333    1817 network.go:98] Watching for new subnet leases
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.388324    1817 network.go:191] Subnet added: 10.10.65.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 flanneld-start[1817]: I1223 15:01:07.388344    1817 network.go:191] Subnet added: 10.10.50.0/24
Dec 23 15:01:07 nanan-product-yanpan-bigdate01 systemd[1]: Started Flanneld overlay address etcd agent.

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:16:3e:2c:36:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.109/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 314756444sec preferred_lft 314756444sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:30:1b:b8:fd brd ff:ff:ff:ff:ff:ff
    inet 10.10.0.1/24 brd 10.10.0.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
    link/none 
    inet 10.10.88.0/16 scope global flannel0
       valid_lft forever preferred_lft forever

4.4、注释

启动Flannel后，一定要记得重启docker，这样Flannel配置分配的ip才能生效，即docker0虚拟网卡的ip会变成上面flannel设定的ip段

4.5、修改docker启动/配置文件使用flannel网络

# cat /usr/lib/systemd/system/docker.service 
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd --insecure-registry=172.17.29.74 -H fd:// --containerd=/run/containerd/containerd.sock $DOCKER_NETWORK_OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
TimeoutSec=0
RestartSec=2
Restart=always
# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
# Both the old, and new location are accepted by systemd 229 and up, so using the old location
# to make them work for either version of systemd.
StartLimitBurst=3
# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
# this option work for either version of systemd.
StartLimitInterval=60s
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not support it.
# Only systemd 226 and above support this option.
TasksMax=infinity
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
[Install]
WantedBy=multi-user.target


# cat /etc/docker/daemon.json 
{
  "registry-mirrors": ["https://registry.docker-cn.com"]
}

4.6、重启docker

# systemctl daemon-reload

# systemctl restart docker

4.7、查看docker是否使用flannel网络

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:16:3e:2c:36:be brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.109/24 brd 192.168.0.255 scope global dynamic eth0
       valid_lft 314756133sec preferred_lft 314756133sec
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:30:1b:b8:fd brd ff:ff:ff:ff:ff:ff
    inet 10.10.88.1/24 brd 10.10.88.255 scope global docker0
       valid_lft forever preferred_lft forever
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN qlen 500
    link/none 
    inet 10.10.88.0/16 scope global flannel0
       valid_lft forever preferred_lft forever

4.8、如果容器无法联通，是由于flannel.0网卡和docker0网卡通过iptables的forward转发，所以需确保如下设置

1、核中的forward功能开启(立即生效，重启后效果不再)
# echo "1" > /proc/sys/net/ipv4/ip_forward
　　
2、包不会被iptables的forward规则拦截
# iptables -P FORWARD ACCEPT