Set up your Open Stack
There are several ways to deploy openstack, Devstack is easily for developer to deploy Open Stack.
Please follow this URL to set up your Open Stack.
http://docs.openstack.org/developer/devstack/
nested KVM support:
- Kernel
- First check if your system has already configured nested kvm:
root@minicloud-allinone-controller0-i7bnc6baumzl:~# dpkg -S /etc/modprobe.d/qemu-system-x86.conf qemu-system-x86: /etc/modprobe.d/qemu-system-x86.conf root@minicloud-allinone-controller0-i7bnc6baumzl:~# cat /etc/modprobe.d/qemu-system-x86.conf options kvm_intel nested=1
- If not, run:
echo “options kvm-intel nested=1″ | sudo tee /etc/modprobe.d/kvm-intel.conf
- then reload kvm_intel.ko
rmmod kvm_intel modprobe kvm_intel
- Nova
- Optional: Modify nova.conf and add host-passthrough. Please note it may impact migration.
libvirt_cpu_mode = host-passthrough
- Restart nova services.
PCI PassThrough Support
There are two ways to support PCI PassThrough.
- legacy UIO framework(PCI Assign);
- VFIO, it is recommended http://lwn.net/Articles/509153/
For PCI SR-IOV device we have a specially way to setup.
Preparation for PCI PassThrough
(1) Enable the Intel VT-d extensions http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
Setup in your bios. Check you have enable it.
$sudo dmesg | grep -e DMAR -e IOMMU
(2) Activate Intel VT-d in the kernel http://www.zerg.sc/technote/?p=241
$ sudo echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts
update the the kernel cmdline.
https://www.gnu.org/software/grub/manual/html_node/Simple-configuration.html
$ sudo su $ sudo sed -i -e 's/(^GRUB_CMDLINE_LINUX_DEFAULT.*)"$/1 intel_iommu=on"/' /etc/default/grub $ sudo update-grub $ sudo grep intel_iommu=on /boot/grub/grub.cfg
Use legacy UIO framework instead of VFIO
Why we still need the legacy UIO framework, for there are some thing wrong with vfio and igb in some PC.
http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM
Set parameter KVM allow_unsafe_assigned_interrupts
Transient configure (take effect Immediately)
$ sudo echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts
Or persistent
$ sudo echo "options kvm allow_unsafe_assigned_interrupts=1" >> /etc/modprobe.d/kvm_iommu.conf
Search the PCI device
$ lspci -n 01:00.0 0200: 8086:10fb (rev 01) 01:00.1 0200: 8086:10fb (rev 01) $ lspci 01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
Bind pci_stub driver to PCI device
$ sudo modprobe pci_stub $ sudo su(take effect Immediately) $ echo "8086 10fb" > /sys/bus/pci/drivers/pci-stub/new_id $ echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind $ echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind $ echo 0000:01:00.1 > /sys/bus/pci/devices/0000:01:00.1/driver/unbind $ echo 0000:01:00.1 > /sys/bus/pci/drivers/pci-stub/bind
Start attach PCI device to guest and start
- in this way, if you want to use virsh to start your guest, you must assign the driver name, or you will failed for libvirt will use vfio driver as default.
- name='kvm' means: the legacy pci device assignment handled directly by the KVM kernel module
$ echo "
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='kvm'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
</hostdev>" > new-pci.xml
$ virsh attach-device guest-name --config new-pci.xml
$ virsh start guest-name
Use VFIO
There are tons of tutorials on how to get VT-d working with KVM/libvirt/Qemu/etc. It’s still a fairly new and unused feature, which means that it is getting updated pretty frequently. Unfortunately, the Internet doesn’t update as quickly, so most of the documentation out there is not accurate or the best approach. readlink You can refer this when any trouble.
We need this command to load the kernel modules at boot time
$ sudo echo "vfio vfio_iommu_type1 vfio_pci" >> /etc/modules
Set vfio_iommu_type1 allow_unsafe_interrupts parameters If you're overriding the ivrs table to enable intremap you dont need this line Persistent (need reboot.)
$ echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf
Or transient (take effect Immediately)
$ sudo echo 1 > /sys/module/vfio_iommu_type1/parameters/allow_unsafe_interrupts
Disable AppArmor for libvirtd
Actually, we do not need to disable Apparmor, this is a bug on ubuntu, we need to work around it. (It may can not works)
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1276719
http://libvirt.org/drvqemu.html#securitysvirtaa
$ apt-get install apparmor-utils $ sudo aa-complain /usr/sbin/libvirtd $ sudo su $ for x in /etc/apparmor.d/libvirt/libvirt-*[0-9a-b];do aa-complain $x; done
Or (It may can not works)
$ sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/ $ sudo ln -s /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper /etc/apparmor.d/disable/ $ sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.libvirtd $ sudo apparmor_parser -R /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper $ sudo service libvirt-bin restart
Or remove it (the best way)
http://www.techytalk.info/disable-and-remove-apparmor-on-ubuntu-based-linux-distributions/
$ sudo /etc/init.d/apparmor stop $ sudo update-rc.d -f apparmor remove $ sudo apt-get --purge remove apparmor apparmor-utils libapparmor-perl libapparmor1
It may cause the ubuntu-desktop crash, please reinstall it.
$ sudo apt-get install ubuntu-desktop
Find the all the pci devices in sam iommu group
We must binding all devices in same group to the vfio-pci driver creates.
$ readlink /sys/bus/pci/devices/0000:01:00.0/iommu_group ../../../../kernel/iommu_groups/1 $ ls -l /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices
Bind vfio driver to PCI device
$ sudo su $ echo "8086 10fb" > /sys/bus/pci/drivers/vfio-pci/new_id $ echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind $ echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind $ echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind $ echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
Start attach PCI device to guest and start
$ echo "
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
</hostdev>" > new-pci.xml
$ virsh attach-device guest-name --config new-pci.xml
$ virsh start guest-name
Configure for igb module for SR-IOV support.
Persistent configure for igb module.
$ sudo echo 'options igb max_vfs=7' > /etc/modprobe.d/igb.conf $ sudo echo "igb" >> /etc/modules $ sudo update-initramfs -k all -t -u
Transient configure for igb module.
$ sudo su $ sudo rmmod igb && sudo modprobe igb max_vfs=7
It maill failed as follow on PC:
Jan 10 04:42:18 localhost kernel: [ 8441.914658] ixgbe 0000:01:00.1: SR-IOV: bus number out of range Jan 10 04:42:18 localhost kernel: [ 8441.914662] ixgbe 0000:01:00.1 0000:01:00.1 (uninitialized): Failed to enable PCI sriov: -12
This is intel official document, it is some different from the above.
Work around if pci-passthrough can not work.
Upgrade your kernel.
Download the latest stable kernel. https://www.kernel.org/
$ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.18.1.tar.xz $ tar -xvf linux-3.18.1.tar.xz $ cd linux-3.18.1 $ cp /boot/config-3.13.0-43-generic ./.config $ echo "" | make oldconfig
Please enable CONFIG_IP_NF_NAT the config, kernel 3.17 change the macro define for nat.
$ sed -i -e "s/# CONFIG_IP_NF_NAT.*/CONFIG_IP_NF_NAT=m/" ./.config $ make -j8 $ make modules_install $ make install $ sudo reboot
compile your owen qemu
$ git clone git://git.qemu.org/qemu.git $ apt-get install libglib2.0-dev zlib1g-dev libspice-server-dev libspice-protocol-dev $ ./configure '--target-list=x86_64-softmmu' '--enable-debug' '--enable-kvm' '--enable-spice' '--prefix=/home/shhfeng/qemu/' $ make
re make you initrd
$ gunzip # uncompress your initrd-*.img file $ mkdir tmp $ cd tmp/ $ cpio -i --make-directories < "your uncompress initrd direction." $ tree -C .
# tree: option -C color show，-n shutdown color show.
or
$ cd tmp/ $ gzip -dc ../sth.gz |cpio -id
# edit your initrd，
$ cd tmp/
# add or remove *.ko
$ find ./ | cpio -H newc -o > /boot/initrd-2.6.22.6.img-ahci $ gzip /boot/initrd-2.6.22.6.img-ahci $ mv initrd-2.6.22.6.img-ahci.gz /boot/initrd-2.6.22.6.img-ahci
Build up your skills to develop Open stack
Doc In Repository
Please read the docs in /opt/stack/nova/doc/source/devref first. Then you will get some ideas about Open Stack. Read up these files, you can ignore them if you already know them.
http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start
- gerrit.rst
It introduce the gerrit work flow.
- development.environment.rst
It introduce how to set up development environment.
- architecture.rst
Introduce the profile of the Open Stack.
- rpc.rst
Open Stack rpc is different with the traditional SUN IPC, it base on a AMQP broker. http://www.rabbitmq.com/getstarted.html
Learn more about implementation of rpc. Please seen the Open Stack Common Library.
$ git clone git://git.openstack.org/openstack/oslo.git
Also learn more about kombu:
http://kombu.readthedocs.org/en/latest/
- threading.rst
All OpenStack services use *green thread* model of threading, If your know coroutine, you will learn greenthread quickly.
All the above files also can be access by this url link.
http://docs.openstack.org/developer/nova/devref/rpc.html
import component
You had better do some exercise about some technologies that Open Stack uses. This is helpful for you to know Open Stack.
- Rabbitmq python tutorials
Tutorial:
https://github.com/rabbitmq/rabbitmq-tutorials/ http://wenku.baidu.com/view/800285ea0975f46527d3e19b.html (a Chinese Document for Rabbitmq) http://www.openstack.cn/p136.html (a Chinese Simple Document for Rabbitmq)
Tutorial Source Code:
https://github.com/rabbitmq/rabbitmq-tutorials/
- sqlalchemy (http://www.sqlalchemy.org/)
Tutorial Source Code:
http://docs.sqlalchemy.org/en/rel_0_9/orm/tutorial.html
- WSGI And eventlet
Read the Red hat REST API Guide. It is useful to understand REST API. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Virtualization/3.0/html-single/REST_API_Guide/index.html#sect-REST_API_Guide-Common_Features-Resources-Creating_Resources
Join The Open Stack Community to do Contributions
- Join the IRC
IRC: https://wiki.openstack.org/wiki/IRC
- Subscribe the mail-list
Dev: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
General: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
- To submit the patch
Workflow: https://wiki.openstack.org/wiki/Gerrit_Workflow
Play with Open Stack
DashBoard
http://docs.openstack.org/user-guide/content/log_in_dashboard.html
Try to access http://localhost/
CURL
http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start
I have write a small script to get token ID, as follow, you can name it as “get-token”:
#!/bin/bash return=`curl -i 'http://127.0.0.1:5000/v2.0/tokens' -X POST -H "Content-Type: application/json" -H "Accept: application/json" -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "123"}}}'` echo $return echo "-----------------------------------------------------------------------------" ID="" for x in $return; do if [ "x$ID" = "xtokenID" ] ;then id=`echo $x | tr -d '[",]'` export TOKEN=$id echo "TOKEN=$id" ID="" break fi if [ "x$x" = 'x"id":' ] ;then ID="tokenID" fi done echo ""
This script just is a sample, you can improve it.
Run it by
$ . get-token
Then you can run other commands use the TOKEN environment value, such as get all the tenents, as follow:
$ curl -i -X GET http://127.0.0.1:35357/v2.0/tenants -H "User-Agent: python-keystoneclient" -H "X-Auth-Token: $TOKEN"
Open Stack Client
http://docs.openstack.org/api/quick-start/content/index.html#Compute_API_Quick_Start
You should export the OS_USERNAME and OS_TENANT_NAME environment by the follow command firstly.
$ cd devstack/ $ . openrc OS_USERNAME OS_TENANT_NAME
http://docs.openstack.org/juno/config-reference/content/section_compute-scheduler.html
update your /etc/nova/nova.conf
Add the follow option to “[DEFAULT]” section.
$ grep -e DEFAULT -e pci_passthrough -e pci_alias -e scheduler_available_filters /etc/nova/nova.conf
[DEFAULT]
pci_passthrough_whitelist={"product_id":"10fb", "vendor_id":"8086", "deviceids":"0000:01:00.0, 0000:01:00.1"}
pci_passthrough_whitelist={"product_id":"8c20", "vendor_id":"8086", "deviceids":"0000:00:1b.0"}
pci_alias = {"vendor_id":"8086", "product_id":"10fb", "name":"intelnic"}
scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter
Create a instance with pci-passthrough device.
https://wiki.openstack.org/wiki/Pci_passthrough
$ nova flavor-list $ nova flavor-key m1.tiny set pci_passthrough:alias=intelnic:1 $ nova boot pci --flavor m1.tiny --image=cirros-0.3.2-x86_64-uec
Debug with Open Stack
I usually use “print” for debugging. The “print” is on the stack screen terminal.
$ cd devstack/ $ ./rejoin-stack.sh
Or
$ screen –ls There is a screen on: 13846.stack $ screen -r stack
Some useful operation:
run a command by “Ctrl + A + :”, you can input help command to show more info. Swith to any window by “Ctrl + A + ' ”, input the windows number you want to switch. Swith to any window by “Ctrl + A + " ”, select the windows you want to switch.