Openvswitch是一个virutal swtich， 支持Open Flow协议，当然也有一些硬件Switch也支持Open Flow协议，他们都可以被统一的Controller管理，从而实现物理机和虚拟机的网络联通。
Open Switch定义了一系列Flow Table，通过它来控制包的流向和结构。
根据Open Flow协议，Flow Table包含match field，如果匹配上了，则执行Action。
其中Match Field涵盖TCP/IP协议各层：
- Layer 1 – Tunnel ID, In Port, QoS priority, skb mark
- Layer 2 – MAC address, VLAN ID, Ethernet type
- Layer 3 – IPv4/IPv6 fields, ARP
- Layer 4 – TCP/UDP, ICMP, ND
Action也主要包含下面的操作：
- Output to port (port range, flood, mirror)
- Discard, Resubmit to table x
- Packet Mangling (Push/Pop VLAN header, TOS, ...)
- Send to controller, Learn
可以设置Tunnel
可以支持下列的框架来监控流量。
- sFlow
- NetFlow
- Port Mirroring
- SPAN
- RSPAN
- ERSPAN
支持QoS
- Uses existing Traffic Control Layer
- Policer (Ingress rate limiter)
- HTB, HFSC (Egress traffic classes)
- Controller (Open Flow) can select Traffic Class
Openvswitch的架构
数据库结构和OVS-VSCTL
# ps aux | grep openvswitch
root 1117 0.0 0.0 21200 1580 ? S< Jun09 0:35 ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach --monitor
root 1153 0.6 1.1 169508 24016 ? S<Ll Jun09 16:24 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid --detach --monitor
我们发现有两个进程：
- ovsdb-server 维护数据库/etc/openvswitch/conf.db
- ovs-vswitchd 核心daemon
- 两者通过unix domain socket /var/run/openvswitch/db.sock 互相通信
ovs-vsctl就是通过和ovsdb-server通信，来修改数据库。
ovs-vswitchd会和ovsdb-server进行通信，来对虚拟设备做相应的修改。
所以ovs-vsctl里面的命令大多数都是对数据库的操作，因而我们必须好好的了解数据库。
如果我们cat /etc/openvswitch/conf.db，我们会发现它是json格式的。
数据库可以通过ovsdb-client dump将数据库内容打印出来
# ovsdb-client dump
Bridge table
_uuid controller datapath_id datapath_type external_ids fail_mode flood_vlans flow_tables ipfix mirrors name netflow other_config ports protocols sflow status stp_enable
------------------------------------ ---------- ------------------ ------------- ------------ --------- ----------- ----------- ----- ------- ------ ------- ------------ ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------- ----- ------ ----------
929ab1c2-1146-411d-8557-af7498a26444 [] "0000080027dfbff7" "" {} [] [] {} [] [] br-ex [] {} [52c02f9d-db2f-4c50-84ce-7a377530ad3b, 79f74c54-056d-4c60-82d7-46412ceee17e] [] [] {} false
12ebfe38-6dab-402a-8fb5-aa814a5a3f52 [] "00003afeeb122a40" "" {} [] [] {} [] [] br-int [] {} [02049620-d3b4-4ecf-9d9e-f8b40a039f64, 035eed84-ce54-44c4-97d5-92f9cc9d662e, 18eacfea-0982-4c89-b2aa-3e2c7dc2b6d3, d43963b0-be0b-4265-b0bb-4ebc928d3ee0, d564225b-60e4-4786-a5fc-6b1c8febd0fb] [] [] {} false
ccfc8a92-2d29-4a55-ae4b-21f59eeeaed7 [] "0000928afccc554a" "" {} [] [] {} [] [] br-tun [] {} [6d14a97d-28e8-42f8-b5d6-ec655ccc7b91, 8b0946ae-3177-4d5d-bc8e-6d04536701ad] [] [] {} false
Controller table
_uuid connection_mode controller_burst_limit controller_rate_limit enable_async_messages external_ids inactivity_probe is_connected local_gateway local_ip local_netmask max_backoff other_config role status target
----- --------------- ---------------------- --------------------- --------------------- ------------ ---------------- ------------ ------------- -------- ------------- ----------- ------------ ---- ------ ------
Flow_Sample_Collector_Set table
_uuid bridge external_ids id ipfix
----- ------ ------------ -- -----
Flow_Table table
_uuid flow_limit groups name overflow_policy
----- ---------- ------ ---- ---------------
IPFIX table
_uuid cache_active_timeout cache_max_flows external_ids obs_domain_id obs_point_id sampling targets
----- -------------------- --------------- ------------ ------------- ------------ -------- -------
Interface table
_uuid admin_state bfd bfd_status cfm_fault cfm_fault_status cfm_health cfm_mpid cfm_remote_mpids cfm_remote_opstate duplex external_ids ifindex ingress_policing_burst ingress_policing_rate lacp_current link_resets link_speed link_state mac mac_in_use mtu name ofport ofport_request options other_config statistics status type
------------------------------------ ----------- --- ---------- --------- ---------------- ---------- -------- ---------------- ------------------ ------ -------------------------------------------------------------------------------------------------------------------------------------------------------- ------- ---------------------- --------------------- ------------ ----------- ----------- ---------- --- ------------------- ---- ---------------- ------ -------------- ---------------- ------------ --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- --------------------------------------------------------------------------- --------
54ab6189-2611-40cc-884f-5ce20913bc32 down {} {} [] [] [] [] [] [] full {} 3 0 0 [] 0 1000000000 down [] "08:00:27:df:bf:f7" 1500 "eth1" 3 [] {} {} {collisions=0, rx_bytes=0, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=0, tx_bytes=558, tx_dropped=0, tx_errors=0, tx_packets=7} {driver_name="e1000", driver_version="7.3.21-k8-NAPI", firmware_version=""} ""
6da8e9cd-98df-451e-b7d9-fcddefe0325e up {} {} [] [] [] [] [] [] [] {} 0 0 0 [] 0 [] up [] "c2:9e:64:de:bd:db" [] patch-int 1 [] {peer=patch-tun} {} {collisions=0, rx_bytes=1340, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=18, tx_bytes=90, tx_dropped=0, tx_errors=0, tx_packets=1} {} patch
eda8218d-1d5e-4c52-ad89-24f8d0445ef9 up {} {} [] [] [] [] [] [] [] {} 0 0 0 [] 0 [] up [] "fa:91:44:32:65:f9" [] patch-tun 1 [] {peer=patch-int} {} {collisions=0, rx_bytes=90, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=1, tx_bytes=1340, tx_dropped=0, tx_errors=0, tx_packets=18} {} patch
ee54427c-133c-4b2a-a641-c2624732eb66 up {} {} [] [] [] [] [] [] [] {} 5 0 0 [] 0 [] up [] "08:00:27:df:bf:f7" 1500 br-ex 65534 [] {} {} {collisions=0, rx_bytes=648, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=8, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0} {driver_name=openvswitch} internal
a45575fd-5179-459e-8951-3553c51f4aaa up {} {} [] [] [] [] [] [] [] {} 6 0 0 [] 2 [] up [] "3a:fe:eb:12:2a:40" 1500 br-int 65534 [] {} {} {collisions=0, rx_bytes=648, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=8, tx_bytes=2078, tx_dropped=0, tx_errors=0, tx_packets=27} {driver_name=openvswitch} internal
801c0dec-44db-4161-b791-08828d542ecf up {} {} [] [] [] [] [] [] [] {} 8 0 0 [] 2 [] up [] "92:8a:fc:cc:55:4a" 1500 br-tun 65534 [] {} {} {collisions=0, rx_bytes=648, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=8, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0} {driver_name=openvswitch} internal
e8193562-c3bb-4b48-a9bf-00bba5f2a213 up {} {} [] [] [] [] [] [] full {attached-mac="fa:16:3e:37:28:e6", iface-id="7d18228b-476f-4b50-804a-18e60e6b0e6f", iface-status=active, vm-uuid="413d9fb4-34e5-4032-95ff-dea80f1f4adc"} 17 0 0 [] 0 10000000000 up [] "66:b8:10:c8:dc:df" 1500 "qvo7d18228b-47" 4 [] {} {} {collisions=0, rx_bytes=530, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=7, tx_bytes=0, tx_dropped=0, tx_errors=0, tx_packets=0} {driver_name=veth, driver_version="1.0", firmware_version=""} ""
b33cdfe2-dab1-449e-b2be-ce85e2d0ac79 up {} {} [] [] [] [] [] [] full {attached-mac="fa:16:3e:58:be:c6", iface-id="eea9263a-5e4b-4e5b-9923-3d59ca752082", iface-status=active, vm-uuid="98b592b1-7778-46f1-93df-3c5079650b71"} 14 0 0 [] 0 10000000000 up [] "7a:d4:a4:8d:60:b7" 1500 "qvoeea9263a-5e" 3 [] {} {} {collisions=0, rx_bytes=530, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=7, tx_bytes=390, tx_dropped=0, tx_errors=0, tx_packets=5} {driver_name=veth, driver_version="1.0", firmware_version=""} ""
c0a5afe4-b7a8-4230-b3bb-af11a33de14e up {} {} [] [] [] [] [] [] full {attached-mac="fa:16:3e:6f:5e:bf", iface-id="c64bd111-eff3-4c65-b096-53c3b3188a43", iface-status=active, vm-uuid="e2009bf8-de2b-4bff-a96a-627a61caf9e7"} 11 0 0 [] 0 10000000000 up [] "e2:d8:61:c4:1d:01" 1500 "qvoc64bd111-ef" 2 [] {} {} {collisions=0, rx_bytes=280, rx_crc_err=0, rx_dropped=0, rx_errors=0, rx_frame_err=0, rx_over_err=0, rx_packets=4, tx_bytes=250, tx_dropped=0, tx_errors=0, tx_packets=3} {driver_name=veth, driver_version="1.0", firmware_version=""} ""
Manager table
_uuid connection_mode external_ids inactivity_probe is_connected max_backoff other_config status target
----- --------------- ------------ ---------------- ------------ ----------- ------------ ------ ------
Mirror table
_uuid external_ids name output_port output_vlan select_all select_dst_port select_src_port select_vlan statistics
----- ------------ ---- ----------- ----------- ---------- --------------- --------------- ----------- ----------
NetFlow table
_uuid active_timeout add_id_to_interface engine_id engine_type external_ids targets
----- -------------- ------------------- --------- ----------- ------------ -------
Open_vSwitch table
_uuid bridges cur_cfg db_version external_ids manager_options next_cfg other_config ovs_version ssl statistics system_type system_version
------------------------------------ ------------------------------------------------------------------------------------------------------------------ ------- ---------- -------------------------------------------------- --------------- -------- ------------ ----------- --- ---------- ----------- --------------
ab19a0f9-1dc8-44d0-ac00-a9687bb43fdd [12ebfe38-6dab-402a-8fb5-aa814a5a3f52, 929ab1c2-1146-411d-8557-af7498a26444, ccfc8a92-2d29-4a55-ae4b-21f59eeeaed7] 273 "7.3.0" {system-id="6b963f5d-45e7-409f-b5ee-8e30006dcd73"} [] 273 {} "2.0.1" [] {} Ubuntu "14.04-trusty"
Port table
_uuid bond_downdelay bond_fake_iface bond_mode bond_updelay external_ids fake_bridge interfaces lacp mac name other_config qos statistics status tag trunks vlan_mode
------------------------------------ -------------- --------------- --------- ------------ ------------ ----------- -------------------------------------- ---- --- ---------------- ------------ --- ---------- ------ --- ------ ---------
52c02f9d-db2f-4c50-84ce-7a377530ad3b 0 false [] 0 {} false [ee54427c-133c-4b2a-a641-c2624732eb66] [] [] br-ex {} [] {} {} [] [] []
18eacfea-0982-4c89-b2aa-3e2c7dc2b6d3 0 false [] 0 {} false [a45575fd-5179-459e-8951-3553c51f4aaa] [] [] br-int {} [] {} {} [] [] []
8b0946ae-3177-4d5d-bc8e-6d04536701ad 0 false [] 0 {} false [801c0dec-44db-4161-b791-08828d542ecf] [] [] br-tun {} [] {} {} [] [] []
79f74c54-056d-4c60-82d7-46412ceee17e 0 false [] 0 {} false [54ab6189-2611-40cc-884f-5ce20913bc32] [] [] "eth1" {} [] {} {} [] [] []
6d14a97d-28e8-42f8-b5d6-ec655ccc7b91 0 false [] 0 {} false [6da8e9cd-98df-451e-b7d9-fcddefe0325e] [] [] patch-int {} [] {} {} [] [] []
035eed84-ce54-44c4-97d5-92f9cc9d662e 0 false [] 0 {} false [eda8218d-1d5e-4c52-ad89-24f8d0445ef9] [] [] patch-tun {} [] {} {} [] [] []
d564225b-60e4-4786-a5fc-6b1c8febd0fb 0 false [] 0 {} false [e8193562-c3bb-4b48-a9bf-00bba5f2a213] [] [] "qvo7d18228b-47" {} [] {} {} 1 [] []
02049620-d3b4-4ecf-9d9e-f8b40a039f64 0 false [] 0 {} false [c0a5afe4-b7a8-4230-b3bb-af11a33de14e] [] [] "qvoc64bd111-ef" {} [] {} {} 1 [] []
d43963b0-be0b-4265-b0bb-4ebc928d3ee0 0 false [] 0 {} false [b33cdfe2-dab1-449e-b2be-ce85e2d0ac79] [] [] "qvoeea9263a-5e" {} [] {} {} 1 [] []
QoS table
_uuid external_ids other_config queues type
----- ------------ ------------ ------ ----
Queue table
_uuid dscp external_ids other_config
----- ---- ------------ ------------
SSL table
_uuid bootstrap_ca_cert ca_cert certificate external_ids private_key
----- ----------------- ------- ----------- ------------ -----------
sFlow table
_uuid agent external_ids header polling sampling targets
----- ----- ------------ ------ ------- -------- -------
数据库表之间的关系如图所示
其中Open_vSwitch是根，结构如下：
这个表示ovs-vswitchd的配置，包含下面的几方面:
- 对bridge设备的配置，bridges指向bridge表，我们能看到的openvswitch的主要功能都是在bridge上实现的，在bridge表中详细叙述。
- 对本身的一些配置;
- other_config : stats-update-interval ：将统计信息写入数据库的间隔时间
- other_config : flow-restore-wait ： 为hot-upgrade使用的，如果设为true则不处理任何的包。一般使用的过程为，先停掉ovs-vswitchd，然后将这个值设为true，启动ovs-vswitchd，这个时候不处理任何包，然后使用ovs-ofctl将flow table restore到一个正确的状态，最后设置这个值为false，开始处理包
- other_config : flow-limit ：在flow table中flow entry的数量
- other_config : n-handler-threads ：用于处理新flow的线程数
- other_config : n-revalidator-threads ：用于验证flow的线程数.
- other_config : enable-statistics 是否统计
- statistics : cpu 统计cpu数量，线程
- statistics : load_average system load
- statistics : memory 总RAM，swap
- statistics : process_NAME ：with NAME replaced by a process name，统计memory size, cpu time等
- statistics : file_systems：mount point, size, used
- client request id: 也即cur_cfg和next_cfg，当一个client修改了数据库的之后，增加next_cfg，然后等待openvswitch应用这些修改，当修改应用完毕，则cur_cfg = next_cfg。如果我们打开/etc/openvswitch/conf.db文件，我们发现，随着我们队openvswitch的配置，cur_cfg是不断++的
- 对SSL的配置：指向SSL表，主要配置private key, certificate(里面是public key)，已经CA的certificate
- 对ovsdb-server的配置，指向Manager表，ovs-vswitchd作为它的client，配置一下db connection的选项
在SSL表中，包含了经典的SSL connection的所有配置：
openvswitch本身的private key和public key对，其中public key放在certificate中，并且需要CA使用自己的private key进行签名，CA来担保这个certificate是合法的，为了验证这个CA签名，当然需要CA的public key，而CA的public key是放在ca cert里面的，当然也需要被签名，被更高级的CA担保，或者自己担保自己。
bootstrap_ca_cert是一个boolean，如果是true，则每次启动的时候，都会向controller去拿最新的ca cert。
我们如果仔细观察ovsdb-server进程，据发现这些配置被使用了。
ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/var/run/openvswitch/db.sock --private-key=db:Open_vSwitch,SSL,private_key --certificate=db:Open_vSwitch,SSL,certificate --bootstrap-ca-cert=db:Open_vSwitch,SSL,ca_cert --no-chdir --log-file=/var/log/openvswitch/ovsdb-server.log --pidfile=/var/run/openvswitch/ovsdb-server.pid --detach –monitor
我们还可以配置ovs-vswitchd和ovs-controller之间通过ssl相互通信，使用的是双向SSL互信
配置ovs-vswitchd，使用switch的private key(sc-privkey.pem)和public key(sc-cert.pem)，但是指定的是controller的CA Cert(cacert.pem)
cd /etc/openvswitch
sudo ovs-pki req+sign sc switch
sudo ovs-vsctl set-ssl
/etc/openvswitch/sc-privkey.pem
/etc/openvswitch/sc-cert.pem
/var/lib/openvswitch/pki/controllerca/cacert.pem配置并启动controller
cd /etc/openvswitch
sudo ovs-pki req+sign ctl controllersudo ovs-controller -v pssl:6633
-p /etc/openvswitch/ctl-privkey.pem
-c /etc/openvswitch/ctl-cert.pem
-C /var/lib/openvswitch/pki/switchca/cacert.pem这里配置的是controller的的private key(ctl-privkey.pem)和public key(ctl-cert.pem)，但是指定的是switch的CA Cert(cacert.pem)
这个SSL机制有关，当建立SSL连接的时候，
当client连接Server的时候，Server会将自己的certificate发给client，将来Client会用Server的certificate来解密Server用private key加密的信息，然而这个certificate是否合法，需要CA进行验证，由于这个certificate是Server的，当然client要使用Server的CA cert进行验证。同理Server会向client请求Client的certificate，来解密Client的private key加密的信息，则会用Client的CA cert进行验证。
当建立ssl连接后，我们可以看到下面的样子：
mininet@mininet:~$ sudo ovs-vsctl show
902d6aa3-6a0a-4708-a286-3301c8b36430
Bridge "s1"
Controller "ssl:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "s1"
Interface "s1"
type: internal
Port "s1-eth1"
Interface "s1-eth1"
Port "s1-eth2"
Interface "s1-eth2"
ovs_version: "2.0.1"连接controller时某一个switch连接到某个controller.
然而连接manager时某一个openvswitch daemon连接到某个manager.
Manager表配置的是ovsdb-server的，他配置了ovsdb-server使用manager_options中的配置来监听端口，等待client来连接。
其中最重要的一项就是target：
- ssl:ip[:port]：ovsdb-server会监听在ip的port上，协议为ssl
- tcp:ip[:port]：会监听在ip的port上，协议为tcp
- pssl:[port][:ip]：会监听在端口port上，协议为ssl
- ptcp:[port][:ip]：会监听在端口port上，协议为ssl
可以通过下面的命令进行设置：
ovs-vsctl set-manager…
从架构图中我们看出，ovs-vswitchd是ovsdb-server的client，两者是通过unix domain sock /var/run/openvswitch/db.sock进行通信的。
ovs-vsctl也是ovsdb-server的客户端，默认情况下，ovs-vsctl是运行在ovsdb-server同一台机器上，也是通过/var/run/openvswitch/db.sock进行通信的。
我们看ovs-vsctl有参数--db，默认是unix:file，然而也可以是tcp:ip:port或者ssl:ip:port
这就使得ovs-vsctl在另外一台机器上，也能远程控制ovsdb-server.
我们有两台机器，一台16.158.165.153，我们设置manager
sudo ovs-vsctl set-manager ptcp:8881
$ sudo ovs-vsctl show
ab19a0f9-1dc8-44d0-ac00-a9687bb43fdd
Manager "ptcp:8881"
Bridge br-int
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvoeea9263a-5e"
tag: 1
Interface "qvoeea9263a-5e"
Port "qvo7d18228b-47"
tag: 1
Interface "qvo7d18228b-47"
Port "qvoc64bd111-ef"
tag: 1
Interface "qvoc64bd111-ef"
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "eth1"
Interface "eth1"
Bridge br-tun
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
ovs_version: "2.0.1"
另一台机器16.158.165.102，我们在上面运行
root@openstackcliu8:~# ovs-vsctl show
21c19aae-c278-4c65-9c6b-9b2c4d67b6dd
ovs_version: "2.0.1"
发现表是空的。
# ovs-vsctl --db=tcp:16.158.165.153:8881 show
ab19a0f9-1dc8-44d0-ac00-a9687bb43fdd
Manager "ptcp:8881"
Bridge br-int
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvoeea9263a-5e"
tag: 1
Interface "qvoeea9263a-5e"
Port "qvo7d18228b-47"
tag: 1
Interface "qvo7d18228b-47"
Port "qvoc64bd111-ef"
tag: 1
Interface "qvoc64bd111-ef"
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "eth1"
Interface "eth1"
Bridge br-tun
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
ovs_version: "2.0.1"
Open_vSwitch指向Bridge表
Bridge表无疑是最重要的表，所谓的virtual switch，多是用bridge来实现的。
Bridge有下面几项配置：
- 核心功能配置项
- name
- ports：指向Port表，
- mirrors：指向mirror表
- netflow
- sflow
- ipfix
- flood_vlans：是一些vlan id，对于这些vlan id，mac address learning是不做的，每次寻找mac都进行arp
- OpenFlow配置项：从架构图中我们可以看出，openvwitch的一个bridge可以通过openflow协议，被一个统一的controller管理的
- Controller
- flow_tables
- fail_mode:
- 一旦一个bridge连到一个openflow controller，则flow table就由controller统一管理，如果连接断了
- secure： 这个bridge会试图一直连接controller，并不自己建立flow table
- standalone：一旦bridge三次连不上controller，就自己建立和管理flow table
- datapath_id：
- Spaning Tree配置
单纯创建一个bridge 很简单
ovs-vsctl add-br helloworld，但是里面的若干配置相对比较复杂。
下一节，我们来看Controller