Calico
Calico 是一个纯三层的数据中心网络方案(不需要Overlay),并且与OpenStack、Kubernetes、AWS、GCE等IaaS和容器平台都有良好的集成。
Calico在每一个计算节点利用Linux Kernel实现了一个高效的vRouter来负责数据转发,而每个vRouter通过BGP协议负责把自己上运行的workload的路由信息像整个Calico网络内传播——小规模部署可以直接互联,大规模下可通过指定的BGP route reflector来完成。 这样保证最终所有的workload之间的数据流量都是通过IP路由的方式完成互联的。Calico节点组网可以直接利用数据中心的网络结构(无论是L2或者L3),不需要额外的NAT,隧道或者Overlay Network。
此外,Calico基于iptables还提供了丰富而灵活的网络Policy,保证通过各个节点上的ACLs来提供Workload的多租户隔离、安全组以及其他可达性限制等功能。
Calico架构
Calico主要由Felix、etcd、BGP client以及BGP Route Reflector组成
- Felix,Calico Agent,跑在每台需要运行Workload的节点上,主要负责配置路由及ACLs等信息来确保Endpoint的连通状态;
- etcd,分布式键值存储,主要负责网络元数据一致性,确保Calico网络状态的准确性;
- BGP Client(BIRD), 主要负责把Felix写入Kernel的路由信息分发到当前Calico网络,确保Workload间的通信的有效性;
- BGP Route Reflector(BIRD),大规模部署时使用,摒弃所有节点互联的 mesh 模式,通过一个或者多个BGP Route Reflector来完成集中式的路由分发。
- calico/calico-ipam,主要用作Kubernetes的CNI插件
IP-in-IP
Calico控制平面的设计要求物理网络得是L2 Fabric,这样vRouter间都是直接可达的,路由不需要把物理设备当做下一跳。为了支持L3 Fabric,Calico推出了IPinIP的选项。
Calico CNI
见https://github.com/projectcalico/cni-plugin。
Calico CNM
Calico通过Pool和Profile的方式实现了docker CNM网络:
- Pool,定义可用于Docker Network的IP资源范围,比如:10.0.0.0/8或者192.168.0.0/16;
- Profile,定义Docker Network Policy的集合,由tags和rules组成;每个 Profile默认拥有一个和Profile名字相同的Tag,每个Profile可以有多个Tag,以List形式保存。
具体实现见https://github.com/projectcalico/libnetwork-plugin,而使用方法可以参考http://docs.projectcalico.org/v3.0/getting-started/docker/。
Calico Kubernetes
对于使用 kubeadm 创建的 Kubernetes 集群,使用以下配置安装 calico 时需要配置
--pod-network-cidr=192.168.0.0/16
--service-cidr=10.96.0.0/12
(不能与 Calico 网络重叠)
各版本的安装方法如下:
- 对于 Kubernetes 1.7.x 或者更新的版本
kubectl apply -f https://docs.projectcalico.org/v3.0/getting-started/kubernetes/installation/hosted/kubeadm/1.7/calico.yaml
- 对于 Kubernetes 1.6.x:
kubectl apply -f http://docs.projectcalico.org/v2.3/getting-started/kubernetes/installation/hosted/kubeadm/1.6/calico.yaml
- 对于 Kubernetes 1.5.x:
kubectl apply -f http://docs.projectcalico.org/v2.3/getting-started/kubernetes/installation/hosted/kubeadm/1.5/calico.yaml
更详细的自定义配置方法见https://docs.projectcalico.org/v3.0/getting-started/kubernetes。
这会在Pod中启动Calico-etcd,在所有Node上启动bird6、felix以及confd,并配置CNI网络为calico插件:
Calico相关进程
root@ubuntu:~# ps -ef | grep calico | grep -v grep
root 39419 39415 0 Jul01 ? 00:00:26 calico-node -monitor-addresses
root 39420 39413 1 Jul01 ? 00:16:16 calico-node -felix
root 39421 39418 0 Jul01 ? 00:00:23 calico-node -allocate-tunnel-addrs
root 39422 39416 0 Jul01 ? 00:00:28 calico-node -confd
root 39782 39417 0 Jul01 ? 00:00:24 bird6 -R -s /var/run/calico/bird6.ctl -d -c /etc/calico/confd/config/bird6.cfg
root 39783 39414 0 Jul01 ? 00:00:30 bird -R -s /var/run/calico/bird.ctl -d -c /etc/calico/confd/config/bird.cfg
CNI网络插件配置
root@ubuntu:~# cat /etc/cni/net.d/10-calico.conflist { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "log_level": "info", "log_file_path": "/var/log/calico/cni/cni.log", "datastore_type": "kubernetes", "nodename": "ubuntu", "mtu": 0, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} }, { "type": "bandwidth", "capabilities": {"bandwidth": true} } ] }root@ubuntu:~#
root@cloud:~# cat /etc/cni/net.d/10-calico.conflist { "name": "k8s-pod-network", "cniVersion": "0.3.1", "plugins": [ { "type": "calico", "log_level": "info", "log_file_path": "/var/log/calico/cni/cni.log", "datastore_type": "kubernetes", "nodename": "cloud", "mtu": 0, "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" } }, { "type": "portmap", "snat": true, "capabilities": {"portMappings": true} }, { "type": "bandwidth", "capabilities": {"bandwidth": true} } ] }root@cloud:~# cat /etc/cni/net.d/calico-kubeconfig # Kubeconfig file for Calico CNI plugin. apiVersion: v1 kind: Config clusters: - name: local cluster: server: https://[10.96.0.1]:443 certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUN5RENDQWJDZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFWTVJNd0VRWURWUVFERXdwcmRXSmwKY201bGRHVnpNQjRYRFRJeE1EY3dNVEE1TVRZME5sb1hEVE14TURZeU9UQTVNVFkwTmxvd0ZURVRNQkVHQTFVRQpBeE1LYTNWaVpYSnVaWFJsY3pDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUtQCkIyeWJFWHRxcHVJN0UyaE01L3FXSnZKY0pqY1dyTDc2TGR3bDRCb295R2Z0cE94UFZTMGlXSmtFcGFmaHZMUjQKdUVwalpOYXllWjZlZWoxWjZOS1docWZWOUw5U1FISDVjdDg4VnVEMUhnUVZIT2NTbkxJNHozYkM2NldKeTcrcQo5TjFPMGpSb0ZZank4USs4cmxxWFVKUEJlY2ZXNnlOS21uWHd4Y2ZLOGlrTmNSVFZQTUtMOTA3aHcvZjFzVmlvCmNPSWZCVENWeXNSRmNTc0RBSm04MGRwVXpCaDE4WTNoczZWdFNNSTlmUzFobWhOSUp0aGhUTXV4aWZtQjRLSjYKclBWWmY2ekxDM3paNEVNam5UaVV5L1NWVXd4QlJCWWVuUURKc1BQS3VkWFE1a08zS0E5ZEpCNy9xekV5L05OWgo0Z1cwZDBnTldCMGc2UjFkOHowQ0F3RUFBYU1qTUNFd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCCi93UUZNQU1CQWY4d0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFEYmtyMjRzTXlobXk0TTh1LzR5S2I4cEYzM0cKeWs2UW9CQWcvcDJzQUVQM3N5d0pGYzlBbjZkZkl1RnVVbkllTXBZcXpWZzlnazMzdGhnR2cxZWwrTGpzb2pDOApSRkZoMEpqbVMyU25hdGx1Z2JDYnlobnB2K1dDaG9ERXJQaHREM1RlclV0NHV1bUxpbjNIcXc4STJ4OXBCakhwClErU05nT0NCN1ZTNHIycHZoc2tQenU5T3lnRmRCT1hHREdNWmRrMkJmeWI2MnhTMmhjcGlsK0pqWUNUckdtSXgKQ2RwSnpsKzBKbjRmZE4zRnppOEFuRGlVUlFiV1JkdnJlU2tQaVpidEUrMVhFUG5nVFdybE1YUjNFUlFzU1lhdQp0eitFOWY2SDZ2cW8xQXROZ3JQaW9YOXpPZDFtVUlXVUNKVndzRUFLOGQ1Q1g3VlBBWEUrdDZkcGh1UT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo= users: - name: calico user: token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjRScXhYMDBHcTBkOUo0OWJUMXBTNVR6MXRZOXFRRVp2RW5SWHQ1dnZLZVUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJjYWxpY28tbm9kZS10b2tlbi1tczlxOSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjYWxpY28tbm9kZSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjJmMWUyNDE1LWMyOTktNGMwNy1hNjM0LTU3Njk3NjQwNDllMyIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTpjYWxpY28tbm9kZSJ9.qnE0SeQWIb8cl1rclo6fjcmYU2pI34VzdI5enscouyQKiqDM9-nmjdUg2q_aMS3Mk7Reefuf1ZHOVrpPEnn6Q8C4d84WN5GnqegmFqYWxVZBlXxGUYZPK_T7AZn9oUJk_kJPAnz-Mz6TKh_UeFKwrOzGdkumo1WxdC4aeIbsWh0mByZNPXRyBGD0TccYuzidMLY5eWthyfDG-YVk1HVBWKSUw-76YEsmieSF-bzr6w3hlMwPi2vdeaHud3gZrIGHboe61MmV1B71joB2_kARATQKAH-pOD3M73ikxmqgvBX3qjju5LpGfxRlqlaF0ZawEKO_jIS12Dz6ybiKzxkj9Q contexts: - name: calico-context context: cluster: local user: calico current-context: calico-context