warden创建容器的过程

基础知识（本节转自http://www.54chen.com/architecture/cloud-foundry-warden-part1.html）

rv = unshare(CLONE_NEWNS);

     unshare这个调用，可以把挂载的文件系统设置成只在新的挂载命名空间（mount namespace）中可见。

execvp(argv[0], argv);

      execvp()会从PATH 环境变量所指的目录中查找符合参数file 的文件名，找到后便执行该文件，然后将第二个参数argv传给该欲执行的文件。

shopt -s nullglob

      设置shell环境变量nullglob的值为on，nullglob为on时对于通配符匹配时，若匹配不到时为空（相对应的为通配符本身）。

int stat(const char *restrict pathname, struct stat *restrict buf);

      提供文件名字，获取文件对应属性。

build-essential软件包

      作用是提供编译程序必须软件包的列表信息，也就是说编译程序有了这个软件包，它才知道 头文件在哪，才知道库函数在哪，还会下载依赖的软件包，最后才组成一个开发环境。

工具debootstrap

       可以用于在系统的某个目录中安装一套基本系统，这个基本系统除了一些配置项外，与ubuntu安装程序在安装的第一阶段安装的内容基本相同。这项功能有许多有趣的功能，例如，你可以从某个定制版本的ubuntu Live光盘上通过这个命令快速的在硬盘上安装ubuntu而不需要ubuntu的安装程序，也可以把创建在硬盘上的基本系统目录作为某些涉及系统安全性服务的chroot运行环境，通过chroot进入该目录并调试和运行一些可能修改系统配置的应用程序，作为定制小型系统模板等等。

aufs

       一种文件格式，可以mount到目录，同时控制只读和读写。

overlayfs

 

        另一种文件格式，在ubuntu 11.04后开始替代aufs作为官方livecd的文件格式。

 

cgroup的初始化

---

初始化的脚本为：

cgroup_path=/sys/fs/cgroup

mount -t tmpfs none $cgroup_path

for subsystem in cpu cpuacct devices memory

do

  mkdir -p $cgroup_path/$subsystem

 

  if ! grep -q "${cgroup_path}/$subsystem " /proc/mounts

  then

    mount -t cgroup -o $subsystem none $cgroup_path/$subsystem

  fi

done

 

上面的命令执行完后，可以看下挂载的结果：

 

# grep '^cgroup' /proc/mounts

cgroups /sys/fs/cgroup tmpfs rw,relatime,mode=755 0 0

cgroup /sys/fs/cgroup/cpu cgroup rw,relatime,cpu 0 0

cgroup /sys/fs/cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0

cgroup /sys/fs/cgroup/devices cgroup rw,relatime,devices 0 0

cgroup /sys/fs/cgroup/memory cgroup rw,relatime,memory 0 0

cgroup /sys/fs/cgroup/freezer cgroup rw,relatime,freezer 0 0

 

网络的控制

---

 Every container is assigned a network interface which is one side of a
 virtual ethernet pair created on the host. The other side of the virtual
 ethernet pair is only visible on the host (from the root namespace).
 The pair is configured to use IPs in a small and static subnet. Traffic
 from and to the container can be forwarded using NAT. Additionally, all
 traffic can be filtered and shaped as needed, using readily available
 tools such as `iptables`.网络初始化

 

echo 1 > /proc/sys/net/ipv4/ip_forward

 

# iptables-save | grep -E 'warden|\*'
*nat
:warden-instance-16al6hojp15 - [0:0]
:warden-prerouting - [0:0]
-A PREROUTING -i eth0 -j warden-prerouting
-A OUTPUT -o lo -j warden-prerouting
-A warden-prerouting -j warden-instance-16al6hojp15
*mangle
*filter
:warden-default - [0:0]
:warden-dispatch - [0:0]
:warden-instance-16al6hojp15 - [0:0]
-A INPUT -i w-+ -j warden-dispatch
-A FORWARD -i w-+ -j warden-dispatch
-A warden-dispatch -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j ACCEPT
-A warden-dispatch -i w-16al6hojp15-0 -g warden-instance-16al6hojp15
-A warden-dispatch -j DROP
-A warden-instance-16al6hojp15 -g warden-default

 

停止apparmor：

 

     /etc/init.d/apparmor teardown
 

# quotaon(8) exits with non-zero status when quotas are ENABLED

if quotaon -p $CONTAINER_DEPOT_MOUNT_POINT_PATH > /dev/null

then

  mount -o remount,usrjquota=aquota.user,grpjquota=aquota.group,jqfmt=vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH

  quotacheck -ugmb -F vfsv0 $CONTAINER_DEPOT_MOUNT_POINT_PATH

  quotaon $CONTAINER_DEPOT_MOUNT_POINT_PATH

fi

  

 

文件系统

---

Every container gets a private root filesystem. This filesystem is

created by stacking a read-only filesytem and a read-write filesystem.
This is implemented by using `aufs` on Ubuntu versions from 10.04 up to
11.10, and `overlayfs` on Ubuntu 12.04.

The read-only filesystem contains the minimal set of Ubuntu packages and
Warden-specific modifications common to all containers. The read-write
filesystem stores files overriding container-specific settings when
necessary. Because all writes are applied to the read-write filesystem,
containers can share the same read-only base filesystem.

The read-write filesystem is created by formatting a large sparse file.
Because the size of this file is fixed, the filesystem that it contains
cannot grow beyond this initial size.

 

创建一个容器

---

set -o nounset
set -o errexit
shopt -s nullglob

 

cp -r skeleton "${target}"
unshare -m "${target}"/setup.sh

 

"${target}"/setup.sh脚本

 

1. 首先生成一个配置文件

 

id=16al6hojp15
network_netmask=255.255.255.252
network_host_ip=10.254.0.17
network_host_iface=w-16al6hojp15-0
network_container_ip=10.254.0.18
network_container_iface=w-16al6hojp15-1
user_uid=10004

 

2. 调用setup_fs()

 

mkdir -p rootfs ${target}

mount -n -t overlayfs -o rw,upperdir=rootfs,lowerdir=${1} none ${target}

3. 调用prepare.sh 删除不需要的文件，并创建必要的设备

 

4. 配置系统参数

 

# cat etc/hosts
127.0.0.1 16al6hojp15 localhost
10.254.0.17 host
10.254.0.18 container

 

# cat etc/hostname
16al6hojp15

 

# cat etc/network/interfaces
auto lo
iface lo inet loopback
auto w-16al6hojp15-1
iface w-16al6hojp15-1 inet static
  gateway 10.254.0.17
  address 10.254.0.18
  netmask 255.255.255.252

 

cp /etc/resolv.conf ${target}/etc/

 

chroot并添加用户：useradd -mU -u ${user_uid} -s /bin/bash vcap

 

# Copy override directory     其实就是几个etc文件和一个sbin/warden-stop.sh文件
cp -r override/* ${target}/
chmod 700 ${target}/sbin/warden-*

 

5. 配置ssh

 

6. 配置mesg

 

# The `mesg` tool modifies permissions on stdin. Warden regularly passes a
# custom stdin, which makes `mesg` complain that stdin is not a tty. Instead of
# removing all occurances of `mesg`, we simply bind it to /bin/true.
chroot <<EOS
rm /usr/bin/mesg
ln -s /bin/true /usr/bin/mesg
EOS

 

7. 物理机上添加网卡作为容器的网关：

 

     ifconfig ${network_host_iface} ${network_host_ip} netmask ${network_netmask}

 

并添加tc规则控制带宽：

 

     qdisc tbf 8002: dev w-16al6hojp15-0 root refcnt 2 rate 8192bit burst 9b lat 24.4ms 

     qdisc ingress ffff: dev w-16al6hojp15-0 parent ffff:fff1 ---------------- 

 

8. 配置cgroup

 

# Add new group for every subsystem

for system_path in /sys/fs/cgroup/*

do

  instance_path=$system_path/instance-$id

 

  mkdir -p $instance_path

 

  if [ $(basename $system_path) == "cpuset" ]

  then

    cat $system_path/cpuset.cpus > $instance_path/cpuset.cpus

    cat $system_path/cpuset.mems > $instance_path/cpuset.mems

  fi

 

  echo 1 > $instance_path/cgroup.clone_children

  echo $PID > $instance_path/tasks

done

 

echo ${PPID} >> ppid

 

ip link add name ${network_host_iface} type veth peer name ${network_container_iface}

ip link set ${network_host_iface} netns 1

ip link set ${network_container_iface} netns ${PID}

 

创建好后，可以通过ssh -i access_key root@10.254.0.18 来登陆，密码是前面安装时设置的密码