一、对象存储
1.1 介绍
通过对象存储，将数据存储为对象，每个对象除了包含数据，还包含数据自身的元数据
对象通过Object ID来检索，无法通过普通文件系统操作来直接访问对象，只能通过API来访问，或者第三方客户端（实际上也是对API的封装）
对象存储中的对象不整理到目录树中，而是存储在扁平的命名空间中，Amazon S3将这个扁平命名空间称为bucket。而swift则将其称为容器
无论是bucket还是容器，都不能嵌套
bucket需要被授权才能访问到，一个帐户可以对多个bucket授权，而权限可以不同
对象存储的优点：易扩展、快速检索
1.2 Rados网关介绍
RADOS网关也称为Ceph对象网关、RADOSGW、RGW，是一种服务，使客户端能够利用标准对象存储API来访问Ceph集群。它支持S3和Swift API
rgw运行于librados之上，事实上就是一个称之为Civetweb的web服务器来响应api请求
客户端使用标准api与rgw通信，而rgw则使用librados与ceph集群通信
rgw客户端通过s3或者swift api使用rgw用户进行身份验证。然后rgw网关代表用户利用cephx与ceph存储进行身份验证
二、RADOS网关部署
2.1 配置radosgw
[root@ceph5 ~]# ceph auth get-or-create client.rgw.ceph5 mon 'allow rwx' osd 'allow rwx' -o /etc/ceph/backup.client.rgw.ceph5.keyring --cluster backup
[root@ceph5 ~]# vim /etc/ceph/backup.conf
fsid = 51dda18c-7545-4edb-8ba9-27330ead81a7 mon_initial_members = ceph5 mon_host = 172.25.250.14 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx public_network = 172.25.250.0/24 cluster_network = 172.25.250.0/24 [mgr] mgr modules = dashboard [client.rgw.ceph5] host = ceph5 keyring = /etc/ceph/backup.client.rgw.ceph5.keyring rgw_frontends = civetweb port=80
[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5
[root@ceph5 ~]# ps -ef|grep rados
root 13828 1 0 18:07 ? 00:00:00 /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph
[root@ceph5 ~]# netstat -ntlp|grep 80
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 13828/radosgw
[root@ceph5 ~]# ceph osd pool ls
[root@ceph5 ~]# ceph -s
[root@ceph5 ~]# ceph osd pool application enable rbd rbd
[root@ceph5 ~]# ceph osd pool application enable rbdmirror rbd
[root@ceph5 ~]# ceph -s
[root@ceph5 ~]# cat /usr/lib/systemd/system/ceph-radosgw@.service
[Unit] Description=Ceph rados gateway After=network-online.target local-fs.target time-sync.target Wants=network-online.target local-fs.target time-sync.target PartOf=ceph-radosgw.target [Service] LimitNOFILE=1048576 LimitNPROC=1048576 EnvironmentFile=-/etc/sysconfig/ceph Environment=CLUSTER=ceph ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i --setuser ceph --setgroup ceph PrivateDevices=yes ProtectHome=true ProtectSystem=full PrivateTmp=true TasksMax=infinity Restart=on-failure StartLimitInterval=30s StartLimitBurst=5 [Install] WantedBy=ceph-radosgw.target
2.2 定义监听的线程数
[root@ceph5 ~]# vim /etc/ceph/backup.conf
[client.rgw.ceph5] host = ceph5 keyring = /etc/ceph/backup.client.rgw.ceph5.keyring rgw_frontends = civetweb port=80 num_threads=100 log = /var/log/ceph/$cluster.$name.log
[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5
[root@ceph5 ~]# ps -ef|grep rados
ceph 15553 1 1 20:26 ? 00:00:00 /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph
2.3 访问rados网关
[root@ceph5 ~]# curl http://ceph5
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>
三、S3对象存储
3.1 S3简介
S3由Amazon于2006年推出，全称为Simple Storage Service
S3定义了对象存储，是对象存储事实上的标准，从某种意义上说，S3就是对象存储，对象存储就是S3
S3是对象存储市场的霸主，后续的对象存储都是对S3的模仿
3.2 用户以及权限设置
创建radosgw的用户
[root@ceph5 ~]# radosgw-admin user create --uid joy --display-name 'Joy Ning'
{ "user_id": "joy", "display_name": "Joy Ning", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "joy", "access_key": "X0CVIF04TAJVTN9D29UL", "secret_key": "vMmPqPap0FC0IRC5J3t9AIPgXNoiw1H9TOWELd5B" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
修改信息
[root@ceph5 ~]# radosgw-admin user modify --uid joy --display-name 'joy Ningrui' --max_buckets 2000
禁用suspend
[root@ceph5 ~]# radosgw-admin user suspend --uid joy
启用
[root@ceph5 ~]# radosgw-admin user enable --uid joy
列出用户
[root@ceph5 ~]# radosgw-admin user list
删除用户
[root@ceph5 ~]# radosgw-admin user rm --uid joy
[root@ceph5 ~]# radosgw-admin user list
[root@ceph5 ~]# radosgw-admin user create --uid joy --display-name 'Joy Ning'
{ "user_id": "joy", "display_name": "Joy Ning", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "joy", "access_key": "5XCV68WUQJFFJPVM3UHK", "secret_key": "xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG" } ], "swift_keys": [], "caps": [], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
[root@ceph5 ~]# radosgw-admin key create --uid joy --display-name 'Joy Ning' --key-type=s3 --gen-access-key --gen-secret
删除key
[root@ceph5 ~]# radosgw-admin key rm --uid joy --display-name 'Joy Ning' --key-type=s3 --access-key HPT1SBAXCXW46ZACKPY0
3.3 设置配额
基于用户的配额
[root@ceph5 ~]# radosgw-admin quota set --quota-scope=user --uid=joy --max-size 1
[root@ceph5 ~]# radosgw-admin user info --uid joy
开启配额
[root@ceph5 ~]# radosgw-admin quota enable --quota-scope=user --uid joy
[root@ceph5 ~]# radosgw-admin user info --uid joy
[root@ceph5 ~]# radosgw-admin quota set --quota-scope=bucket --uid=joy --max-size 1
[root@ceph5 ~]# radosgw-admin quota enable --quota-scope=bucket --uid=joy
[root@ceph5 ~]# radosgw-admin user info --uid joy
注：如果两个都进行配置，则那个先到，使用哪一个
关闭配额
可以disable
[root@ceph5 ~]# radosgw-admin quota disable --quota-scope=bucket --uid=joy
也可以参数设为1
[root@ceph5 ~]# radosgw-admin quota set --quota-scope=user --uid joy --max-size -1
[root@ceph5 ~]# radosgw-admin user info --uid joy
3.4 统计数据
统计所有
[root@ceph5 ~]# radosgw-admin usage show --uid joy
[root@ceph5 ~]# radosgw-admin usage show --uid joy --start-date 2019-03-19 21:00:00 --end-date 2019-03-19 22:00:00
3.5 利用rados网关来访问s3对象
[root@ceph5 ~]# vim /etc/ceph/backup.conf
[root@ceph5 ~]# systemctl restart ceph-radosgw@rgw.ceph5
[root@ceph5 ~]# ps -ef|grep rados
ceph 18072 1 2 21:52 ? 00:00:00 /usr/bin/radosgw -f --cluster backup --name client.rgw.ceph5 --setuser ceph --setgroup ceph
四 验证配置
4.1 配置s3cmd
[root@ceph1 ceph]# yum -y install s3cmd
[root@ceph1 ceph]# s3cmd --configure
root@ceph6's password: Permission denied, please try again. root@ceph6's password: hosts 100% 786 1.6MB/s 00:00 [root@ceph1 ceph]# s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: 5XCV68WUQJFFJPVM3UHK Secret Key: xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG Default Region [US]: Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: redhat Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP, and can only be proxied with Python 2.7 or newer Use HTTPS protocol [Yes]: no On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: ceph5.lab.example.com HTTP Proxy server port [3128]: 80 New settings: Access Key: 5XCV68WUQJFFJPVM3UHK Secret Key: xhaA2YB1CA3xH54xLbmwPcglqjDyuFez36F8XGuG Default Region: US Encryption password: redhat Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: ceph5.lab.example.com HTTP Proxy server port: 80 Test access with supplied credentials? [Y/n] n Save settings? [y/N] y Configuration saved to '/root/.s3cfg'
[root@ceph1 ceph]# vim /root/.s3cfg
host_base = ceph5 host_bucket = %(bucket)s.ceph5.lab.example.com cloudfront_host = cloudfront.amazonaws.com website_endpoint = http://%(bucket)s.ceph5.lab.example.com/
4.2 创建bucket
[root@ceph1 ceph]# s3cmd mb s3://test
4.3 传送数据
[root@ceph1 ceph]# echo 11111 >/tmp/demoobject
[root@ceph1 ceph]# s3cmd put --acl-public /tmp/demoobject s3://test/demoobject
[root@ceph1 ceph]# vim /etc/hosts
172.25.250.10 ceph1 ceph1.lab.example.com servera 172.25.250.11 ceph2 ceph2.lab.example.com serverb 172.25.250.12 ceph3 ceph3.lab.example.com serverc 172.25.250.13 ceph4 ceph4.lab.example.com serverd 172.25.250.14 ceph5 ceph5.lab.example.com servere test.ceph5.lab.example.com
4.4 访问bucket
[root@ceph1 ceph]# curl http://test.ceph5.lab.example.com/demoobject
4.5 查看bucket
到服务端看
[root@ceph5 ~]# radosgw-admin bucket list
[root@ceph5 ~]# radosgw-admin bucket stats --bucket=test
{ "bucket": "test", "zonegroup": "e80133e1-a513-44f5-ba90-e25b6c987b26", "placement_rule": "default-placement", "explicit_placement": { "data_pool": "", "data_extra_pool": "", "index_pool": "" }, "id": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1", "marker": "1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1", "index_type": "Normal", "owner": "joy", "ver": "0#3", "master_ver": "0#0", "mtime": "2019-03-19 22:02:50.726716", "max_marker": "0#", "usage": { "rgw.main": { "size": 6, "size_actual": 4096, "size_utilized": 6, "size_kb": 1, "size_kb_actual": 4, "size_kb_utilized": 1, "num_objects": 1 } }, "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 } }
[root@ceph5 ~]# radosgw-admin bucket check --bucket=test
删除
[root@ceph5 ~]# radosgw-admin bucket rm --bucket=test
[root@ceph1 ceph]# s3cmd put --acl-public /etc/ceph/ceph.conf s3://test/ceph
upload: '/etc/ceph/ceph.conf' -> 's3://test/ceph' [1 of 1] 589 of 589 100% in 0s 20.96 kB/s done Public URL of the object is: http://test.ceph5/ceph
[root@ceph1 ceph]# curl http://test.ceph5.lab.example.com/ceph
# Please do not change this file directly since it is managed by Ansible and will be overwritten [global] fsid = 35a91e48-8244-4e96-a7ee-980ab989d20d mon initial members = ceph2,ceph3,ceph4 mon host = 172.25.250.11,172.25.250.12,172.25.250.13 public network = 172.25.250.0/24 cluster network = 172.25.250.0/24 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx [osd] osd mkfs type = xfs osd mkfs options xfs = -f -i size=2048 osd mount options xfs = noatime,largeio,inode64,swalloc osd journal size = 5120 [mon] mon_allow_pool_delete = true
[root@ceph1 ceph]# s3cmd get s3://test/demoobject ./demoobject
download: 's3://test/demoobject' -> './demoobject' [1 of 1] 6 of 6 100% in 0s 1346.20 B/s done
[root@ceph1 ceph]# cat ./demoobject
4.6 查看底层数据
[root@ceph5 ~]# ceph osd pool ls rbd rbdmirror .rgw.root default.rgw.control default.rgw.meta default.rgw.log default.rgw.buckets.index default.rgw.buckets.data [root@ceph5 ~]# rados -p default.rgw.buckets.index ls --cluster backup .dir.1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1 [root@ceph5 ~]# rados -p default.rgw.buckets.data ls error opening pool default.rgw.buckets.data: (2) No such file or directory [root@ceph5 ~]# rados -p default.rgw.buckets.data ls --cluster backup 1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1_demoobject 1b85c5b1-19d2-48a1-bb45-3ac75895aeed.4235.1_ceph
实验完成
博主声明：本文的内容来源主要来自誉天教育晏威老师，由本人实验完成操作验证，需要的博友请联系誉天教育（http://www.yutianedu.com/），获得官方同意或者晏老师（https://www.cnblogs.com/breezey/）本人同意即可转载，谢谢！