之前介绍了《OpenStack Swift All In One安装部署流程与简单使用》，那么接下来就说一说Swift集群部署吧。
1. 简介
本文档详细描述了使用两台PC部署一个小型Swift集群的过程，并给出一些简单的使用实例。本文档假定如下前提条件：
- 使用Ubuntu操作系统。
- 每台机器都运行Swift的所有服务，既是Proxy Server，又是Storage Server，用户可以向任何一台机器发起存储服务请求。
- 采用Swift自带的TempAuth作为用户的身份与权限认证。
- 所有机器构成memcached集群来提供Token缓存服务。
- 所有操作均在root用户下进行，并使用root作为Swift的用户和组。
- 所有机器都运行在局域网中。
- 使用回环设备和XFS文件系统作为Swift底层存储。
阅读本文档前，可以先阅读文档《Swift All In One安装部署流程》，学习Swift单机部署的相关知识。
2. 安装部署
2.1 准备环境
|
|
PC 1 |
PC 2 |
|
机器类型： |
PC物理机 |
PC物理机 |
|
操作系统： |
Ubuntu-12.04-desktop-64位 |
Ubuntu-12.04-desktop-64位 |
|
用户类型： |
root |
root |
|
数据库： |
sqlite3 |
sqlite3 |
|
IP地址： |
192.168.3.52（局域网） |
192.168.3.53（局域网） |
|
Proxy Server： |
是 |
是 |
|
Storage Server： |
是 |
是 |
|
Auth： |
TempAuth |
TempAuth |
|
Token缓存： |
memcached |
memcached |
2.2 版本说明
本文档基于：
- 官方文档：Swift 1.7.7-dev documentation -> Instructions for a Multiple Server Swift Installation (Ubuntu)
- Swift版本：1.7.6
- python-swiftclient版本：1.2.0
请确保安装的Swift版本与本文档中的版本相同。如有问题，请参考官网的更新文档。
2.3 安装软件环境
首先，在PC1和PC2上安装Swift所需的软件环境（确保你的机器可以访问互联网），例如，sqlite3作为本地数据库，memcached作为Token缓存。Ubuntu-12.04已自带rsync工具，因此不用另行安装。
2.4 安装Swift 在PC1和PC2上执行以下操作，安装Swift服务： 1. 在主目录（root用户）下创建swift目录。然后在该下创建bin目录，用于存放我们手动创建的Swift相关脚本文件。进入~/swift目录，然后从git上获取Swift和python-swiftclient源代码，下载到本地。当然也可以使用以前下载的1.7.6版本的Swift代码和1.2.0版本的python-swiftclient代码，将代码目录放至~/swift目录下即可。 3. 然后使用上述代码以开发的方式安装Swift和python-swiftclient（假设Swift的代码目录为~/swift/swift_1.7.6，python-swiftclient的代码目录为~/swift/python-swiftclient_1.2.0）。最终，两者都会被安装到python的dist-packages中。安装过程中，会自动检查其所需的依赖项，并自动进行下载安装。文件~/swift/swift_1.7.6/tools/pip-requires中（内容如下所示）记录了Swift所需的依赖项，setup.py就是根据该文件来检查依赖项的。 5. 类似的，文件~/swift/python-swiftclient_1.2.0/tools/pip-requires中（内容如下所示）记录了python-swiftclient所需的依赖项。修改~/.bashrc文件，在文件尾部添加如下内容：（该文件包含当前用户Bash Shell的环境变量信息，用以标明Swift测试配置文件路径和启动程序路径） /var/run/swift目录在操作系统关闭后会消失，因此需要在操作系统再次启动时进行创建。我们可以编辑/etc/rc.local文件，在exit 0 之前添加如下内容来实现该目录的自动创建。 2.5 配置Storage Server2.5.1 配置存储空间 Swift能够运行在任何支持扩展属性的现代文件系统之上，Swift官方推荐用户使用XFS文件系统。经过官方的验证，认为XFS文件系统能为Swift的用例提供最佳的性能，并且通过了完整的稳定性测试。 对于任何一台PC，我们可以选择使用一个分区作为存储（Using a partition for storage），也可以使用一个回环设备作为存储（Using a loopback device for storage）。由于实验环境所限，本文档使用回环设备作为存储。若希望使用独立分区作为存储，请参考官方文档。我们需要在每一台PC上创建回环设备，作为每一个Swift节点的数据存储空间。在PC1和PC2上执行以下操作： 1. 选择一个位置创建存储文件夹。在存储文件夹中创建XFS格式的回环设备，即/srv/swift-disk文件。
|
# cd ~/swift # git clone https://github.com/openstack/swift.git # git clone https://github.com/openstack/python-swiftclient.git |
|
eventlet>=0.9.15 greenlet>=0.3.1 netifaces>=0.6 pastedeploy>=1.3.3 simplejson>=2.0.9 xattr>=0.4 python-swiftclient |
|
mkdir -p /var/run/swift chown root:root /var/run/swift |
|
/srv/swift-disk /srv/node/sdb1 xfs loop,noatime,nodiratime,nobarrier,logbufs=8 0 0 |
|
# chown -R root:root /srv/node |
|
# mkdir -p /etc/swift # chown -R root:root /etc/swift/ |
|
[swift-hash] # random unique string that can never change (DO NOT LOSE) swift_hash_path_suffix = jtangfs |
|
uid = root gid = root log file = /var/log/rsyncd.log pid file = /var/run/rsyncd.pid address = 192.168.3.52
[account] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/account.lock
[container] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/container.lock
[object] max connections = 2 path = /srv/node/ read only = false lock file = /var/lock/object.lock |
|
# perl -pi -e 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync # service rsync restart |
|
[DEFAULT] devices = /srv/node mount_check = false bind_ip = 192.168.3.52 bind_port = 6002 workers = 4 user = root log_facility = LOG_LOCAL4
[pipeline:main] pipeline = account-server
[app:account-server] use = egg:swift#account
[account-replicator]
[account-auditor]
[account-reaper] |
|
[DEFAULT] devices = /srv/node mount_check = false bind_ip = 192.168.3.52 bind_port = 6001 workers = 4 user = root log_facility = LOG_LOCAL3
[pipeline:main] pipeline = container-server
[app:container-server] use = egg:swift#container
[container-replicator]
[container-updater]
[container-auditor]
[container-sync] |
|
[DEFAULT] devices = /srv/node mount_check = false bind_ip = 192.168.3.52 bind_port = 6000 workers = 4 user = root log_facility = LOG_LOCAL2
[pipeline:main] pipeline = object-server
[app:object-server] use = egg:swift#object
[object-replicator]
[object-updater]
[object-auditor] |
|
# Uncomment the following to have a log containing all logs together #local1,local2,local3,local4,local5.* /var/log/swift/all.log
# Uncomment the following to have hourly proxy logs for stats processing $template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%" #local1.*;local1.!notice ?HourlyProxyLog
local2.*;local2.!notice /var/log/swift/object.log local2.notice /var/log/swift/ object.error local2.* ~
local3.*;local3.!notice /var/log/swift/container.log local3.notice /var/log/swift/ container.error local3.* ~
local4.*;local4.!notice /var/log/swift/account.log local4.notice /var/log/swift/ account.error local4.* ~ |
|
[DEFAULT] bind_port = 8080 user = root workers = 8 log_facility = LOG_LOCAL1
[pipeline:main] pipeline = healthcheck cache tempauth proxy-logging proxy-server
[app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true
[filter:tempauth] use = egg:swift#tempauth user_admin_admin = admin .admin .reseller_admin user_test_tester = testing .admin user_test2_tester2 = testing2 .admin user_test_tester3 = testing3 reseller_prefix = AUTH # account和token的命名前缀，注意此处不可以加“_”。 # 例如X-Storage-Url为http://192.168.3.52:8080/v1/AUTH_test # 例如X-Auth-Token为AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1 token_life = 86400 # token的有效期，单位：秒。
[filter:healthcheck] use = egg:swift#healthcheck
[filter:cache] use = egg:swift#memcache memcache_servers = 192.168.3.52:11211,192.168.3.53:11211
[filter:proxy-logging] use = egg:swift#proxy_logging |
|
# Uncomment the following to have a log containing all logs together #local1,local2,local3,local4,local5.* /var/log/swift/all.log
# Uncomment the following to have hourly proxy logs for stats processing $template HourlyProxyLog,"/var/log/swift/hourly/%$YEAR%%$MONTH%%$DAY%%$HOUR%" #local1.*;local1.!notice ?HourlyProxyLog
local1.*;local1.!notice /var/log/swift/proxy.log local1.notice /var/log/swift/ proxy.error local1.* ~
local2.*;local2.!notice /var/log/swift/object.log local2.notice /var/log/swift/ object.error local2.* ~
local3.*;local3.!notice /var/log/swift/container.log local3.notice /var/log/swift/ container.error local3.* ~
local4.*;local4.!notice /var/log/swift/account.log local4.notice /var/log/swift/ account.error local4.* ~ |
|
# cd /etc/swift # swift-ring-builder account.builder create 18 2 1 # swift-ring-builder container.builder create 18 2 1 # swift-ring-builder object.builder create 18 2 1 |
|
# cd /etc/swift # swift-ring-builder account.builder add z1-192.168.3.52:6002/sdb1 100 # swift-ring-builder container.builder add z1-192.168.3.52:6001/sdb1 100 # swift-ring-builder object.builder add z1-192.168.3.52:6000/sdb1 100
# swift-ring-builder account.builder add z2-192.168.3.53:6002/sdb1 100 # swift-ring-builder container.builder add z2-192.168.3.53:6001/sdb1 100 # swift-ring-builder object.builder add z2-192.168.3.53:6000/sdb1 100 |
|
#!/bin/bash
cd /etc/swift
rm -f *.builder *.ring.gz backups/*.builder backups/*.ring.gz
swift-ring-builder account.builder create 18 2 1 swift-ring-builder container.builder create 18 2 1 swift-ring-builder object.builder create 18 2 1
swift-ring-builder account.builder add z1-192.168.3.52:6002/sdb1 100 swift-ring-builder container.builder add z1-192.168.3.52:6001/sdb1 100 swift-ring-builder object.builder add z1-192.168.3.52:6000/sdb1 100
swift-ring-builder account.builder add z2-192.168.3.53:6002/sdb1 100 swift-ring-builder container.builder add z2-192.168.3.53:6001/sdb1 100 swift-ring-builder object.builder add z2-192.168.3.53:6000/sdb1 100
swift-ring-builder account.builder rebalance swift-ring-builder container.builder rebalance swift-ring-builder object.builder rebalance |
|
#!/bin/bash
swift-init all stop find /var/log/swift -type f -exec rm -f {} ; sudo umount /srv/node/sdb1 sudo mkfs.xfs -f -i size=1024 /srv/swift-disk sudo mount /srv/node/sdb1 sudo chown root:root /srv/node/sdb1 sudo rm -f /var/log/debug /var/log/messages /var/log/rsyncd.log /var/log/syslog sudo service rsyslog restart sudo service rsync restart sudo service memcached restart |
|
#!/bin/bash
swift-init main start |
|
#!/bin/bash
swift-init main stop |
|
#!/bin/bash
swift-init proxy start swift-init account-server start swift-init account-replicator start swift-init account-auditor start swift-init container-server start swift-init container-replicator start swift-init container-updater start swift-init container-auditor start swift-init object-server start swift-init object-replicator start swift-init object-updater start swift-init object-auditor start |
|
#!/bin/bash
swift-init proxy stop swift-init account-server stop swift-init account-replicator stop swift-init account-auditor stop swift-init container-server stop swift-init container-replicator stop swift-init container-updater stop swift-init container-auditor stop swift-init object-server stop swift-init object-replicator stop swift-init object-updater stop swift-init object-auditor stop |
|
# chmod +x ~/swift/bin/* |
|
# cd ~/swift/swift_1.7.6 # ./.unittests |
|
# startmain 或 # startall |
|
# stopmain 或 # stopall |
|
# swift --help Usage: swift <command> [options] [args]
Commands: stat [container] [object] Displays information for the account, container, or object depending on the args given (if any). list [options] [container] Lists the containers for the account or the objects for a container. -p or --prefix is an option that will only list items beginning with that prefix. -d or --delimiter is option (for container listings only) that will roll up items with the given delimiter (see Cloud Files general documentation for what this means). upload [options] container file_or_directory [file_or_directory] [...] Uploads to the given container the files and directories specified by the remaining args. -c or --changed is an option that will only upload files that have changed since the last upload. -S <size> or --segment-size <size> and --leave-segments are options as well (see --help for more). post [options] [container] [object] Updates meta information for the account, container, or object depending on the args given. If the container is not found, it will be created automatically; but this is not true for accounts and objects. Containers also allow the -r (or --read-acl) and -w (or --write-acl) options. The -m or --meta option is allowed on all and used to define the user meta data items to set in the form Name:Value. This option can be repeated. Example: post -m Color:Blue -m Size:Large download --all OR download container [options] [object] [object] ... Downloads everything in the account (with --all), or everything in a container, or a list of objects depending on the args given. For a single object download, you may use the -o [--output] <filename> option to redirect the output to a specific file or if "-" then just redirect to stdout. delete [options] --all OR delete container [options] [object] [object] ... Deletes everything in the account (with --all), or everything in a container, or a list of objects depending on the args given. Segments of manifest objects will be deleted as well, unless you specify the --leave-segments option.
Example: swift -A https://auth.api.rackspacecloud.com/v1.0 -U user -K key stat
Options: --version show program's version number and exit -h, --help show this help message and exit -s, --snet Use SERVICENET internal network -v, --verbose Print more info -q, --quiet Suppress status output -A AUTH, --auth=AUTH URL for obtaining an auth token -V AUTH_VERSION, --auth-version=AUTH_VERSION Specify a version for authentication. Defaults to 1.0. -U USER, --user=USER User name for obtaining an auth token. -K KEY, --key=KEY Key for obtaining an auth token. --os-username=<auth-user-name> Openstack username. Defaults to env[OS_USERNAME]. --os-password=<auth-password> Openstack password. Defaults to env[OS_PASSWORD]. --os-tenant-id=<auth-tenant-id> OpenStack tenant ID. Defaults to env[OS_TENANT_ID] --os-tenant-name=<auth-tenant-name> Openstack tenant name. Defaults to env[OS_TENANT_NAME]. --os-auth-url=<auth-url> Openstack auth URL. Defaults to env[OS_AUTH_URL]. --os-auth-token=<auth-token> Openstack token. Defaults to env[OS_AUTH_TOKEN] --os-storage-url=<storage-url> Openstack storage URL. Defaults to env[OS_STORAGE_URL] --os-region-name=<region-name> Openstack region name. Defaults to env[OS_REGION_NAME] --os-service-type=<service-type> Openstack Service type. Defaults to env[OS_SERVICE_TYPE] --os-endpoint-type=<endpoint-type> Openstack Endpoint type. Defaults to env[OS_ENDPOINT_TYPE] --insecure Allow swiftclient to access insecure keystone server. The keystone's certificate will not be verified. |
|
# curl -k -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' http://192.168.3.52:8080/auth/v1.0 |
|
* About to connect() to 192.168.3.52 port 8080 (#0) * Trying 192.168.3.52... connected > GET /auth/v1.0 HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: 192.168.3.52:8080 > Accept: */* > X-Storage-User: test:tester > X-Storage-Pass: testing > < HTTP/1.1 200 OK < X-Storage-Url: http://192.168.3.52:8080/v1/AUTH_test < X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1 < Content-Type: text/html; charset=UTF-8 < X-Storage-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1 < Content-Length: 0 < Date: Wed, 20 Mar 2013 06:13:15 GMT < * Connection #0 to host 192.168.3.52 left intact * Closing connection #0 |
|
# curl -k -v -H 'X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1' http://192.168.3.52:8080/v1/AUTH_test |
|
* About to connect() to 192.168.3.52 port 8080 (#0) * Trying 192.168.3.52... connected > GET /v1/AUTH_test HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: 192.168.3.52:8080 > Accept: */* > X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1 > < HTTP/1.1 204 No Content < Content-Length: 0 < Accept-Ranges: bytes < X-Timestamp: 1363760036.52552 < X-Account-Bytes-Used: 0 < X-Account-Container-Count: 0 < Content-Type: text/html; charset=UTF-8 < X-Account-Object-Count: 0 < Date: Wed, 20 Mar 2013 06:13:56 GMT < * Connection #0 to host 192.168.3.52 left intact * Closing connection #0 |
|
# curl -k -v -H 'X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1' http://192.168.3.53:8080/v1/AUTH_test |
|
* About to connect() to 192.168.3.53 port 8080 (#0) * Trying 192.168.3.53... connected > GET /v1/AUTH_test HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: 192.168.3.53:8080 > Accept: */* > X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1 > < HTTP/1.1 204 No Content < Content-Length: 0 < Accept-Ranges: bytes < X-Timestamp: 1363760036.52552 < X-Account-Bytes-Used: 0 < X-Account-Container-Count: 0 < Content-Type: text/html; charset=UTF-8 < X-Account-Object-Count: 0 < Date: Wed, 20 Mar 2013 06:15:19 GMT < * Connection #0 to host 192.168.3.53 left intact * Closing connection #0 |
|
# curl -k -v -H 'X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1' http://192.168.3.52:8080/v1/AUTH_test |
|
* About to connect() to 192.168.3.52 port 8080 (#0) * Trying 192.168.3.52... connected > GET /v1/AUTH_test HTTP/1.1 > User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3 > Host: 192.168.3.52:8080 > Accept: */* > X-Auth-Token: AUTH_tk440e9bd9a9cb46d6be07a5b6a585f7d1 > < HTTP/1.1 204 No Content < Content-Length: 0 < Accept-Ranges: bytes < X-Timestamp: 1363760036.52552 < X-Account-Bytes-Used: 0 < X-Account-Container-Count: 0 < Content-Type: text/html; charset=UTF-8 < X-Account-Object-Count: 0 < Date: Wed, 20 Mar 2013 06:17:01 GMT < * Connection #0 to host 192.168.3.52 left intact * Closing connection #0 |
|
# swift -A http://192.168.3.53:8080/auth/v1.0 -U test:tester -K testing stat |
|
Account: AUTH_test Containers: 0 Objects: 0 Bytes: 0 Accept-Ranges: bytes X-Timestamp: 1363760036.52552 Content-Type: text/plain; charset=utf-8 |
|
# swift -A http://192.168.3.52:8080/auth/v1.0 -U test:tester -K testing post myfiles |
|
|
|
# swift -A http://192.168.3.53:8080/auth/v1.0 -U test:tester -K testing list |
|
myfiles |
|
# swift -A http://192.168.3.52:8080/auth/v1.0 -U test:tester -K testing list |
|
myfiles |
|
# swift -A http://192.168.3.53:8080/auth/v1.0 -U test:tester -K testing upload myfiles ~/file |
|
root/file |
|
# swift -A http://192.168.3.52:8080/auth/v1.0 -U test:tester -K testing list myfiles |
|
root/file |