20210821第二天：Ceph账号管理（普通用户挂载）、mds高可用

下面主要内容：

1. 用户权限管理和授权流程
2. 用普通用户挂载rbd和cephfs
3. mds高可用
   • 多mds active
   • 多mds active加standby

一、Ceph的用户权限管理和授权流程

一般系统的身份认真无非三点：账号、角色和认真鉴权，Ceph 的用户可以是一个具体的人或系统角色（e.g. 应用程序），Ceph 管理员通过创建用户并设置权限来控制谁可以访问、操作 Ceph Cluster、Pool 或 Objects 等资源。

 

 Ceph 的用户类型可以分为以下几类：

• 客户端用户
  • 操作用户（e.g. client.admin）
  • 应用程序用户（e.g. client.cinder）
• 其他用户
  • Ceph 守护进程用户（e.g. mds.ceph-node1、osd.0）

用户命名遵循 <TYPE.ID> 的命名规则，其中 Type 有 mon,osd,client 三者，L 版本以后添加了 mgr 类型。

NOTE：为 Ceph 守护进程创建用户，是因为 MON、OSD、MDS 等守护进程同样遵守 CephX 协议，但它们不能属于真正意义上的客户端。

Ceph 用户的密钥：相当于用户的密码，本质是一个唯一字符串。e.g. key: AQB3175c7fuQEBAAMIyROU5o2qrwEghuPwo68g==

Ceph 用户的授权（Capabilities, caps）：即用户的权限，通常在创建用户的同时进行授权。只有在授权之后，用户才可以使用权限范围内的 MON、OSD、MDS 的功能，也可以通过授权限制用户对 Ceph 集群数据或命名空间的访问范围。以 client.admin 和 client.jack为例：

client.admin
    key: AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    # 允许访问 MDS
    caps: [mds] allow *
    # 允许访问 Mgr
    caps: [mgr] allow *
    # 允许访问 MON
    caps: [mon] allow *
    # 允许访问 OSD
    caps: [osd] allow *

client.tom
    key: AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    # 允许对 MON 读的权限
    caps: [mon] allow r
    # 允许对rbd-data资源池进行读写
    caps: [osd] allow rwx pool=rbd-date

授权类型：

• allow：在守护进程进行访问设置之前就已经具有特定权限，常见于管理员和守护进程用户。
• r：授予用户读的权限，读取集群各个组件（MON/OSD/MDS/CRUSH/PG）的状态，但是不能修改。
• w：授予用户写对象的权限，与 r 配合使用，修改集群的各个组件的状态，可以执行组件的各个动作指令。
• x：授予用户调用类方法的能力，仅仅和 ceph auth 操作相关。
• class-read：授予用户调用类读取方法的能力，是 x 的子集。
• class-write：授予用户调用类写入方法的能力，是 x 的子集。
• *：授予用户 rwx 权限。
• profile osd：授权用户以 OSD 身份连接到其它 OSD 或 MON，使得 OSD 能够处理副本心跳和状态汇报。
• profile mds：授权用户以 MDS 身份连接其它 MDS 或 MON。
• profile bootstrap-osd：授权用户引导 OSD 守护进程的能力，通常授予部署工具（e.g. ceph-deploy），让它们在引导 OSD 时就有增加密钥的权限了。
• profile bootstrap-mds：授权用户引导 MDS 守护进程的能力。同上。

NOTE：可见 Ceph 客户端不直接访问 Objects，而是必须要经过 OSD 守护进程的交互。

用户管理常规操作

Ceph 用户管理指令 auth：

root@node02:~# ceph auth --help
...
auth add <entity> [<caps>...]                                                                            add auth info for <entity> from input file, or random key if no input is given, and/or any caps 
                                                                                                          specified in the command
auth caps <entity> <caps>...                                                                             update caps for <name> from caps specified in the command
auth export [<entity>]                                                                                   write keyring for requested entity, or master keyring if none given
auth get <entity>                                                                                        write keyring file with requested key
auth get-key <entity>                                                                                    display requested key
auth get-or-create <entity> [<caps>...]                                                                  add auth info for <entity> from input file, or random key if no input given, and/or any caps specified 
                                                                                                          in the command
auth get-or-create-key <entity> [<caps>...]                                                              get, or add, key for <name> from system/caps pairs specified in the command.  If key already exists, any 
                                                                                                          given caps must match the existing caps for that key.
auth import                                                                                              auth import: read keyring file from -i <file>
auth ls                                                                                                  list authentication state
auth print-key <entity>                                                                                  display requested key
auth print_key <entity>                                                                                  display requested key
auth rm <entity>                                                                                         remove all caps for <name>

获取指定用户的权限信息：

root@node02:~# ceph auth get client.admin
[client.admin]
    key = AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
exported keyring for client.admin
root@node02:~# 

新建用户：

# 授权的格式为：`{守护进程类型} 'allow {权限}'`

# 创建 client.jack 用户，并授权可读 MON、可读写 OSD Pool livepool
ceph auth add client.jack mon 'allow r' osd 'allow rw pool=rbd-data'

# 获取或创建 client.tom，若创建，则授权可读 MON，可读写 OSD Pool livepool
ceph auth get-or-create client.tom mon 'allow r' osd 'allow rw pool=rbd-data'

# 获取或创建 client.jerry，若创建，则授权可读 MON，可读写  OSD Pool livepool，输出用户秘钥环文件
ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rw pool=rbd-data' -o jerry.keyring
ceph auth get-or-create-key client.ringo mon 'allow r' osd 'allow rw pool=rbd-data' -o ringo.key

删除用户：

ceph auth del {TYPE}.{ID}

CephX 认证系统

Ceph 提供了两种身份认证方式：None 和 CephX。前者表示客户端不需要通过密钥访问即可访问 Ceph 存储集群，显然这种方式是不被推荐的。所以我们一般会启用 CephX 认证系统，通过编辑 ceph.conf 开启：

[global]
...
# 表示存储集群（mon,osd,mds）相互之间需要通过 keyring 认证
auth_cluster_required = cephx
# 表示客户端（比如gateway）到存储集群（mon,osd,mds）需要通过 keyring 认证
auth_service_required = cephx
# 表示存储集群（mon,osd,mds）到客户端（e.g. gateway）需要通过 keyring 认证
auth_client_required = cephx

　　CephX 的本质是一种对称加密协议，加密算法为 AES，用于识别用户的身份、对用户在客户端上的操作进行认证，以此防止中间人攻击、数据篡改等网络安全问题。应用 CephX 的前提是创建一个用户，当我们使用上述指令创建一个用户时，MON 会将用户的密钥返回给客户端，同时自己也会保存一份副本。这样客户端和 MON 就共享了一份密钥。CephX 使用共享密钥的方式进行认证，客户端和 MON 集群都会持有一份用户密钥，它提供了一个相互认证的机制，MON 集群确定客户端持有用户密钥，而客户端确定 MON 集群持有用户密钥的的副本。

 

身份认证原理

1. 用户通过客户端向 MON 发起请求。
2. 客户端将用户名传递到 MON。
3. MON 对用户名进行检查，若用户存在，则通过加密用户密钥生成一个 session key 并返回客户端。
4. 客户端通过共享密钥解密 session key，只有拥有相同用户密钥环文件的客户端可以完成解密。
5. 客户端得到 session key 后，客户端持有 session key 再次向 MON 发起请求
6. MON 生成一个 ticket，同样使用用户密钥进行加密，然后发送给客户端。
7. 客户端同样通过共享密钥解密得到 ticket。
8. 往后，客户端持有 ticket 向 MON、OSD 发起请求。

 这就是共享密钥认证的好处，客户端、MON、OSD、MDS 共同持有用户的密钥，只要客户端与 MON 完成验证之后，客户端就可以与任意服务进行交互。并且只要客户端拥有任意用户的密钥环文件，客户端就可以执行特定用户所具有权限的所有操作。当我们执行 ceph -s 时候，实际上执行的是 ceph -s --conf /etc/ceph/ceph.conf --name client.admin --keyring /etc/ceph/ceph.client.admin.keyring。客户端会从下列默认路径查看 admin 用户的 keyring：

/etc/ceph/ceph.client.admin.keyring
/etc/ceph/ceph.keyring
/etc/ceph/keyring
/etc/ceph/keyring.bin

使用 ceph-authtool 进行密钥环管理

NOTE：密钥环文件 .keyring 是 Ceph 用户的载体，当客户端拥有了密钥环文件就相当于可以通过对应的用户来执行客户端操作。

MON、OSD、admin client 的初始密钥环文件：

• /var/lib/ceph/mon/ceph-$hostname/keyring
• /var/lib/ceph/osd/ceph-$hostname/keyring
• /etc/ceph/ceph.client.admin.keyring

指定密钥环（admin 用户）来执行集群操作：

ceph -n client.admin --keyring=/etc/ceph/ceph.client.admin.keyring health
rbd create -p rbd volume01 --size 1G --image-feature layering --conf /etc/ceph/ceph.conf --name client.admin --keyring /etc/ceph/ceph.client.admin.keyring

查看用户密钥：

ceph auth print-key {TYPE}.{ID}

将用户密钥导入到密钥环：

ceph auth get client.admin -o /etc/ceph/ceph.client.admin.keyring

创建用户并生产密钥环文件：

ceph-authtool -n client.user1 --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.keyring
ceph-authtool -C /etc/ceph/ceph.keyring -n client.user1 --cap osd 'allow rwx' --cap mon 'allow rwx' --gen-key

修改用户授权：

ceph-authtool /etc/ceph/ceph.keyring -n client.user1 --cap osd 'allow rwx' --cap mon 'allow rwx'

从密钥环文件导入一个用户:

ceph auth import -i /path/to/keyring

将用户导出到一个密钥环文件：

ceph auth get client.admin -o /etc/ceph/ceph.client.admin.keyring

不废话了，下面是我本次作业的情况：

1、创建块、cephfs和创建普通用户挂载

1.1、创建rbd-data存储池并创建rbdtest1.img块：

root@node01:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
hdd    240 GiB  240 GiB  41 MiB    41 MiB       0.02
TOTAL  240 GiB  240 GiB  41 MiB    41 MiB       0.02
 
--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS    USED  %USED  MAX AVAIL
device_health_metrics   1    1      0 B        0     0 B      0     76 GiB
mypool1                 2   32  3.1 KiB        3  21 KiB      0     76 GiB
root@node01:~# 
root@node01:~# ceph osd pool create rbd-data 32 32
pool 'rbd-data' created
root@node02:~# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node02:~# 
root@node01:~# ceph osd pool ls
device_health_metrics
mypool1
rbd-data
root@node01:~# ceph osd pool application enable rbd-data rbd
enabled application 'rbd' on pool 'rbd-data'
root@node01:~# rb
rbash            rbd              rbdmap           rbd-replay       rbd-replay-many  rbd-replay-prep  
root@node01:~# rbd pool init -p rbd-data
root@node01:~# rbd create rbdtest1.img --pool rbd-data --size 1G
root@node01:~# rbd ls --pool rbd-data
rbdtest1.img
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# rbd feature disable rbd-data/rbdtest1.img exclusive-lock object-map fast-diff  deep-flatten
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# 

 下面是具体主要步骤命令：

root@node01:~# ceph osd pool create rbd-data 32 32
root@node01:~# ceph osd pool application enable rbd-data rbd
root@node01:~# rbd pool init -p rbd-data
root@node01:~# rbd create rbdtest1.img --pool rbd-data --size 1G
root@node01:~# rbd ls --pool rbd-data
root@node01:~# rbd info rbd-data/rbdtest1.img
root@node01:~# rbd feature disable rbd-data/rbdtest1.img exclusive-lock object-map fast-diff  deep-flatten
root@node01:~# rbd info rbd-data/rbdtest1.img

1.2、创建删除tom、Jerry等账号的情况：

root@node01:~# ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=rbd-date'
added key for client.tom
root@node01:~# ceph auth get client.tom
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
exported keyring for client.tom
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth get
get                get-key            get-or-create      get-or-create-key  
root@node01:~# ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# 
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth print-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==root@node01:~# 
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth  get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth caps client.jerry mon 'allow rw' osd 'allow r pool=rbd-data'
updated caps for client.jerry
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow rw"
    caps osd = "allow r pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth  del client.jerry
updated
root@node01:~# ceph auth get client.jerry
Error ENOENT: failed to find client.jerry in keyring
root@node01:~# 

 下面是具体主要步骤命令：

root@node01:~# ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=rbd-date'
root@node01:~# ceph auth get client.tom
root@node01:~# ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
root@node01:~# ceph auth get-or-create-key client.jerry
root@node01:~# ceph auth get-or-create-key client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
root@node01:~# ceph auth print-key client.jerry
root@node01:~# ceph auth caps client.jerry mon 'allow rw' osd 'allow r pool=rbd-data'

1.3、通过秘钥环文件备份与恢复用

root@node01:~# ceph auth get-or-create client.user1 mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ls
ceph-deploy  cluster  snap
root@node01:~# ceph-authtool --help
usage: ceph-authtool keyringfile [OPTIONS]...
where the options are:
  -l, --list                    will list all keys and capabilities present in
                                the keyring
  -p, --print-key               will print an encoded key for the specified
                                entityname. This is suitable for the
                                'mount -o secret=..' argument
  -C, --create-keyring          will create a new keyring, overwriting any
                                existing keyringfile
  -g, --gen-key                 will generate a new secret key for the
                                specified entityname
  --gen-print-key               will generate a new secret key without set it
                                to the keyringfile, prints the secret to stdout
  --import-keyring FILE         will import the content of a given keyring
                                into the keyringfile
  -n NAME, --name NAME          specify entityname to operate on
  -a BASE64, --add-key BASE64   will add an encoded key to the keyring
  --cap SUBSYSTEM CAPABILITY    will set the capability for given subsystem
  --caps CAPSFILE               will set all of capabilities associated with a
                                given key, for all subsystems
  --mode MODE                   will set the desired file mode to the keyring
                                e.g: '0644', defaults to '0600'
root@node01:~# ceph-authtool -C ceph.client.user1.keyring
creating ceph.client.user1.keyring
root@node01:~# cat ceph.client.user1.keyring 
root@node01:~# file ceph.client.user1.keyring 
ceph.client.user1.keyring: empty
root@node01:~# 
root@node01:~# ceph auth  get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ceph auth  get client.user1 -o ceph.client.user1.keyring 
exported keyring for client.user1
root@node01:~# cat ceph.client.user1.keyring 
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 

从 keyring 文件恢复用户认证信息:


root@node01:~# 
root@node01:~# ceph auth del client.user1
updated
root@node01:~# ceph auth get client.user1
Error ENOENT: failed to find client.user1 in keyring
root@node01:~# 
root@node01:~# ceph auth import -i ceph.client.user1.keyring 
imported keyring
root@node01:~# 
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# 

 下面是具体主要步骤命令：

root@node01:~# ceph auth get-or-create client.user1 mon 'allow r' osd 'allow rwx pool=rbd-data'
root@node01:~# ceph auth get client.user1
root@node01:~# ceph-authtool -C ceph.client.user1.keyring
root@node01:~# ceph auth  get client.user1 -o ceph.client.user1.keyring 

从keyring 文件恢复用户认证信息:
root@node01:~# ceph auth del client.user1
root@node01:~# ceph auth get client.user1
root@node01:~# ceph auth import -i ceph.client.user1.keyring 
root@node01:~# ceph auth get client.user1

再加一点：一个配置文件中多个账号情况：

root@node01:~# ceph-authtool -C ceph.client.tom.keyring
creating ceph.client.tom.keyring
root@node01:~# ceph auth get client.tom -o ceph.client.tom.keyring 
exported keyring for client.tom
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# ceph-authtool -l ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# ceph-authtool ceph.client.user1.keyring --import-keyring ceph.client.tom.keyring 
importing contents of ceph.client.tom.keyring into ceph.client.user1.keyring
root@node01:~# echo $?
0
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# cat ceph.client.user1.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 

1.4、普通用户挂载RBD情况：

服务上同步配置文件到客户端：
root@node01:~# ls
ceph.client.tom.keyring  ceph.client.user1.keyring  ceph-deploy  client.client.user1.keyring  client.user1.keyring  cluster  snap
root@node01:~# scp ceph.client.user1.keyring ceph-deploy/ceph.conf root@node04:/etc/ceph/
root@node04's password: 
ceph.client.user1.keyring                                                                                                                                                       100%  244   263.4KB/s   00:00    
ceph.conf                                                                                                                                                                       100%  265    50.5KB/s   00:00    
root@node01:~#

客户端操作

root@node04:/etc/netplan# apt install ceph-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4 librados2
  libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd python3-rgw
Suggested packages:
  ceph-base ceph-mds
The following NEW packages will be installed:
  ceph-common ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4
  librados2 libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd
  python3-rgw
0 upgraded, 30 newly installed, 0 to remove and 79 not upgraded.
Need to get 37.3 MB of archives.
After this operation, 152 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific focal/main amd64 libjaeger amd64 16.2.5-1focal [3,780 B]

中间太长了，我就删除了点

Setting up ceph-common (16.2.5-1focal) ...
Adding group ceph....done
Adding system user ceph....done
Setting system user ceph properties....done
chown: cannot access '/var/log/ceph/*.log*': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target.
Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
root@node04:/etc/netplan# ls /etc/ceph/
rbdmap
root@node04:/etc/netplan# ls /etc/ceph/
ceph.client.user1.keyring  ceph.conf  rbdmap
root@node04:/etc/netplan# cd /etc/ceph/
root@node04:/etc/ceph# ll
total 20
drwxr-xr-x  2 root root 4096 Aug 24 21:10 ./
drwxr-xr-x 96 root root 4096 Aug 24 21:09 ../
-rw-------  1 root root  244 Aug 24 21:10 ceph.client.user1.keyring
-rw-r--r--  1 root root  265 Aug 24 21:10 ceph.conf
-rw-r--r--  1 root root   92 Jul  8 22:16 rbdmap
root@node04:/etc/ceph# ceph --user user1 -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4225 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 70m)
    mgr: node01(active, since 70m), standbys: node02, node03
    osd: 6 osds: 6 up (since 70m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 7 objects, 54 B
    usage:   44 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --user user1 -p rbd-data map rbdtest1.img
/dev/rbd0
rbd: --user is deprecated, use --id
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
rbd: warning: image already mapped as /dev/rbd0
/dev/rbd1
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
rbd: rbd-data/rbdtest1.img: mapped more than once, unmapping /dev/rbd0 only
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd1                      252:16   0    1G  0 disk 
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
/dev/rbd0
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=32768 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@node04:/etc/ceph# mount /dev/rbd0 /mnt/
root@node04:/etc/ceph# cp /etc/passwd /mnt/
root@node04:/etc/ceph# cd /mnt/
root@node04:/mnt# ls
passwd
root@node04:/mnt# tail passwd 
uuidd:x:107:112::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:113::/nonexistent:/usr/sbin/nologin
landscape:x:109:115::/var/lib/landscape:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
usbmux:x:111:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
sshd:x:112:65534::/run/sshd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
vmuser:x:1000:1000:vmuser:/home/vmuser:/bin/bash
lxd:x:998:100::/var/snap/lxd/common/lxd:/bin/false
ceph:x:64045:64045:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
root@node04:/mnt# 

 下面是具体主要步骤命令：

服务上操作同步配置文件到客户端：
root@node01:~# scp ceph.client.user1.keyring ceph-deploy/ceph.conf root@node04:/etc/ceph/

客户端操作：
root@node04:/etc/netplan# apt install ceph-common
root@node04:/etc/netplan# ls /etc/ceph/
root@node04:/etc/ceph# ceph --user user1 -s
root@node04:/etc/ceph# rbd --user user1 -p rbd-data map rbdtest1.img（这台命令少用吧，用下面的）
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
root@node04:/etc/ceph# mkfs.xfs /dev/rbd0
root@node04:/etc/ceph# mount /dev/rbd0 /mnt/

1.5、普通用户挂载CephFS情况：

root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4986 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 83m)
    mgr: node01(active, since 83m), standbys: node02, node03
    osd: 6 osds: 6 up (since 83m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 18 objects, 14 MiB
    usage:   132 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node01:~/ceph-deploy# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph osd pool create cephfs-metadata 16 16
pool 'cephfs-metadata' created
root@node01:~/ceph-deploy# ceph osd pool create cephfs-data 32 32
pool 'cephfs-data' created
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5126 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 85m)
    mgr: node01(active, since 85m), standbys: node02, node03
    osd: 6 osds: 6 up (since 85m), 6 in (since 7d)
 
  data:
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs new mycephfs ceph-metadata cephfs-data
Error ENOENT: pool 'ceph-metadata' does not exist
root@node01:~/ceph-deploy# ceph fs new mycephfs cephfs-metadata cephfs-data
new fs with metadata pool 4 and data pool 5
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5221 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 87m)
    mgr: node01(active, since 87m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 87m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ceph mds status mycephfs
no valid command found; 10 closest matches:
mds metadata [<who>]
mds count-metadata <property>
mds versions
mds compat show
mds ok-to-stop <ids>...
mds fail <role_or_gid>
mds repaired <role>
mds rm <gid:int>
mds compat rm_compat <feature:int>
mds compat rm_incompat <feature:int>
Error EINVAL: invalid command
root@node01:~/ceph-deploy# ceph fs status mycephfs
mycephfs - 0 clients
========
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata     0   75.9G  
  cephfs-data      data       0   75.9G  
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5291 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 88m)
    mgr: node01(active, since 88m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 88m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-mgr.keyring  ceph.bootstrap-osd.keyring  ceph.bootstrap-rgw.keyring  ceph.client.admin.keyring  ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring
root@node01:~/ceph-deploy# ceph-de
ceph-debugpack  ceph-dencoder   ceph-deploy     
root@node01:~/ceph-deploy# ceph-deploy mds create node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9007af8a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f9007acddd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node01', 'node01')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node01:node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node01
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node01][WARNIN] mds keyring does not exist yet, creating one
[node01][DEBUG ] create a keyring file
[node01][DEBUG ] create path if it doesn't exist
[node01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node01/keyring
[node01][INFO  ] Running command: systemctl enable ceph-mds@node01
[node01][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node01.service → /lib/systemd/system/ceph-mds@.service.
[node01][INFO  ] Running command: systemctl start ceph-mds@node01
[node01][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active}
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth add client.jack mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
added key for client.jack
root@node01:~/ceph-deploy# ceph auth get client.jack
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
exported keyring for client.jack
root@node01:~/ceph-deploy# ceph auth get client.admin
[client.admin]
    key = AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
exported keyring for client.admin
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get client.jack -o ceph.client.jack.keyring
exported keyring for client.jack
root@node01:~/ceph-deploy# cat ceph.client.jack.keyring
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# scp ceph.client.jack.keyring node04:/etc/ceph/
root@node04's password: 
ceph.client.jack.keyring                                                                                                                                                        100%  148   118.7KB/s   00:00    
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get-or-create-key cliet.jack
Error EINVAL: bad entity name
root@node01:~/ceph-deploy# ceph auth print-key cliet.jack
Error EINVAL: invalid entity_auth cliet.jack
root@node01:~/ceph-deploy# ceph auth print-key client.jack 
AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==root@node01:~/ceph-deploy# ceph auth print-key client.jack > jack.key
root@node01:~/ceph-deploy# scp jack.key node04:/etc/ceph/
root@node04's password: 
jack.key                                                                                                                                                                        100%   40    26.3KB/s   00:00    
root@node01:~/ceph-deploy#


root@node04:~# ceph --user jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5671 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# ceph --id jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5676 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# 
root@node04:~# mkdir /data
root@node04:~# 
root@node04:~# mount -t ceph node01:6789,node02:6789,node03:6789:/ /data/ -o name=jack,secretfile=/etc/ceph/jack.key
root@node04:~# df -hT
Filesystem                                                    Type      Size  Used Avail Use% Mounted on
udev                                                          devtmpfs  936M     0  936M   0% /dev
tmpfs                                                         tmpfs     196M  1.2M  195M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv                             ext4       20G  4.5G   15G  25% /
tmpfs                                                         tmpfs     980M     0  980M   0% /dev/shm
tmpfs                                                         tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                                                         tmpfs     980M     0  980M   0% /sys/fs/cgroup
/dev/sda2                                                     ext4      976M  106M  804M  12% /boot
/dev/loop0                                                    squashfs   56M   56M     0 100% /snap/core18/2128
/dev/loop1                                                    squashfs   56M   56M     0 100% /snap/core18/1944
/dev/loop2                                                    squashfs   32M   32M     0 100% /snap/snapd/10707
/dev/loop3                                                    squashfs   70M   70M     0 100% /snap/lxd/19188
tmpfs                                                         tmpfs     196M     0  196M   0% /run/user/0
/dev/loop4                                                    squashfs   33M   33M     0 100% /snap/snapd/12704
/dev/loop5                                                    squashfs   71M   71M     0 100% /snap/lxd/21029
/dev/rbd0                                                     xfs      1014M   40M  975M   4% /mnt
192.168.11.210:6789,192.168.11.220:6789,192.168.11.230:6789:/ ceph       76G     0   76G   0% /data
root@node04:~# cd /data/
root@node04:/data# echo 'test content' > testfile.txt
root@node04:/data# cat testfile.txt 
test content
root@node04:/data# 

 下面是具体主要步骤命令：

服务器端操作：
root@node01:~/ceph-deploy# ceph-deploy mds create node01
root@node01:~/ceph-deploy# ceph osd pool create cephfs-metadata 16 16
root@node01:~/ceph-deploy# ceph osd pool create cephfs-data 32 32
root@node01:~/ceph-deploy# ceph fs new mycephfs cephfs-metadata cephfs-data

root@node01:~/ceph-deploy# ceph auth add client.jack mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
root@node01:~/ceph-deploy# ceph auth get client.jack -o ceph.client.jack.keyring

root@node01:~/ceph-deploy# scp ceph.client.jack.keyring node04:/etc/ceph/
root@node01:~/ceph-deploy# ceph auth print-key client.jack > jack.key
root@node01:~/ceph-deploy# scp jack.key node04:/etc/ceph/

客户端操作：
root@node04:~# ceph --user jack -s
root@node04:~# mkdir /data
root@node04:~# mount -t ceph node01:6789,node02:6789,node03:6789:/ /data/ -o name=jack,secretfile=/etc/ceph/jack.key

 2、CephFS的mds高可用情况：

 本次实验的节点情况：

192.168.11.210    node01
192.168.11.220    node02
192.168.11.230    node03
192.168.11.240    node04  这个节点就是新加的作为上面客户端和这里的mds

2.1、开始创建多个mds吧

root@node01:~/ceph-deploy# ceph-deploy mds create node02
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node02
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7febd49fda50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7febd49d2dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node02', 'node02')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node02:node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node02
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node02][WARNIN] mds keyring does not exist yet, creating one
[node02][DEBUG ] create a keyring file
[node02][DEBUG ] create path if it doesn't exist
[node02][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node02 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node02/keyring
[node02][INFO  ] Running command: systemctl enable ceph-mds@node02
[node02][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node02.service → /lib/systemd/system/ceph-mds@.service.
[node02][INFO  ] Running command: systemctl start ceph-mds@node02
[node02][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy mds create node03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f1d42589a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f1d4255edd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node03', 'node03')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node03:node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node03
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node03][WARNIN] mds keyring does not exist yet, creating one
[node03][DEBUG ] create a keyring file
[node03][DEBUG ] create path if it doesn't exist
[node03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node03 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node03/keyring
[node03][INFO  ] Running command: systemctl enable ceph-mds@node03
[node03][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node03.service → /lib/systemd/system/ceph-mds@.service.
[node03][INFO  ] Running command: systemctl start ceph-mds@node03
[node03][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 2 up:standby
root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f72374d4a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f72374a9dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
bash: python2: command not found
[ceph_deploy.mds][ERROR ] connecting to host: node04 resulted in errors: IOError cannot send (already closed?)
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MDSs

root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fa90baa50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f4fa908fdd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node04
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node04][WARNIN] mds keyring does not exist yet, creating one
[node04][DEBUG ] create a keyring file
[node04][DEBUG ] create path if it doesn't exist
[node04][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node04 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node04/keyring
[node04][INFO  ] Running command: systemctl enable ceph-mds@node04
[node04][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node04.service → /lib/systemd/system/ceph-mds@.service.
[node04][INFO  ] Running command: systemctl start ceph-mds@node04
[node04][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   108k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
   node02    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    5
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:29:55.774998+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    1
in    0
up    {0=65409}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs set mycephfs max_mds 2
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    12
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:44:44.248039+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    2
in    0,1
up    {0=65409,1=64432}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
[mds.node02{1:64432} state up:active seq 41 addr [v2:192.168.11.220:6808/4242415336,v1:192.168.11.220:6809/4242415336]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 

 下面是具体主要步骤命令：

部署节点上（node01）上添加另外三个mds：
root@node01:~/ceph-deploy# ceph-deploy mds create node02
root@node01:~/ceph-deploy# ceph-deploy mds create node03
root@node01:~/ceph-deploy# ceph-deploy mds create node04
查看状态：
root@node01:~/ceph-deploy# ceph mds stat
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   108k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
   node02    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 
root@node01:~/ceph-deploy# ceph fs get mycephfs
root@node01:~/ceph-deploy# ceph fs set mycephfs max_mds 2
root@node01:~/ceph-deploy# ceph fs status

此时两主，两备，任何一个主出现问题，两个备节点中的一个会自动顶上去：

root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)

重启node01节点

root@node01:~/ceph-deploy# systemctl restart ceph
ceph-crash.service       ceph-mds@node01.service  ceph-mgr@node01.service  ceph-mon@node01.service  ceph-osd@0.service       ceph-osd.target          ceph.service             
ceph-fuse.target         ceph-mds.target          ceph-mgr.target          ceph-mon.target          ceph-osd@3.service       ceph-radosgw.target      ceph.target              
root@node01:~/ceph-deploy# systemctl restart ceph-m
ceph-mds@node01.service  ceph-mds.target          ceph-mgr@node01.service  ceph-mgr.target          ceph-mon@node01.service  ceph-mon.target          
root@node01:~/ceph-deploy# systemctl restart ceph-mds
ceph-mds@node01.service  ceph-mds.target          
root@node01:~/ceph-deploy# systemctl restart ceph-mds@node01.service 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s     1      4      2      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 

如果想做成一对一的绑定——node01和node02互为主备，node03和node04互为主备，下面这样整一下吧（主要是配置文件）：

root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node04
mds_standby_replay = true
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# ceph-deploy config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fdbb800f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f4fdbbd6350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to config 1 hosts

root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f201b0280f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f201b07e350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f549232a0f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node02', 'node02', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f5492380350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart ceph-mds@node01.service 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9fa57ccf50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f9fa58a54d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
Permission denied, please try again.
root@node04's password: 忘了做免密同步，不过也没上面，就输入一下密码即可
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy][ERROR ] GenericError: Failed to configure 1 admin hosts

root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04 --overwrite-conf 
usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME]
                   [--overwrite-conf] [--ceph-conf CEPH_CONF]
                   COMMAND ...
ceph-deploy: error: unrecognized arguments: --overwrite-conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf admin node01 node02 node03 node04 
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f41f97a5f50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f41f987e4d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-osd.keyring  ceph.client.admin.keyring  ceph.conf             ceph.mon.keyring
ceph.bootstrap-mgr.keyring  ceph.bootstrap-rgw.keyring  ceph.client.jack.keyring   ceph-deploy-ceph.log  jack.key
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s     1      4      2      0   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# cat ceph.c
cat: ceph.c: No such file or directory
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart ceph-mds@node01.service 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node04  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 

好了就这样吧！！！

================================ 下面是我的整个过程的log ===================================

root@node01:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL    USED  RAW USED  %RAW USED
hdd    240 GiB  240 GiB  41 MiB    41 MiB       0.02
TOTAL  240 GiB  240 GiB  41 MiB    41 MiB       0.02
 
--- POOLS ---
POOL                   ID  PGS   STORED  OBJECTS    USED  %USED  MAX AVAIL
device_health_metrics   1    1      0 B        0     0 B      0     76 GiB
mypool1                 2   32  3.1 KiB        3  21 KiB      0     76 GiB
root@node01:~# 
root@node01:~# ceph osd pool create rbd-data 32 32
pool 'rbd-data' created
root@node02:~# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node02:~# 
root@node01:~# ceph osd pool ls
device_health_metrics
mypool1
rbd-data
root@node01:~# ceph osd pool application enable rbd-data rbd
enabled application 'rbd' on pool 'rbd-data'
root@node01:~# rb
rbash            rbd              rbdmap           rbd-replay       rbd-replay-many  rbd-replay-prep  
root@node01:~# rbd pool init -p rbd-data
root@node01:~# rbd create rbdtest1.img --pool rbd-data --size 1G
root@node01:~# rbd ls --pool rbd-data
rbdtest1.img
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# rbd feature disable rbd-data/rbdtest1.img exclusive-lock object-map fast-diff  deep-flatten
root@node01:~# rbd info rbd-data/rbdtest1.img
rbd image 'rbdtest1.img':
    size 1 GiB in 256 objects
    order 22 (4 MiB objects)
    snapshot_count: 0
    id: fb9a70e1dad6
    block_name_prefix: rbd_data.fb9a70e1dad6
    format: 2
    features: layering
    op_features: 
    flags: 
    create_timestamp: Tue Aug 24 20:17:40 2021
    access_timestamp: Tue Aug 24 20:17:40 2021
    modify_timestamp: Tue Aug 24 20:17:40 2021
root@node01:~# 







root@node01:~# ceph auth add client.tom mon 'allow r' osd 'allow rwx pool=rbd-date'
added key for client.tom
root@node01:~# ceph auth get client.tom
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
exported keyring for client.tom
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth get
get                get-key            get-or-create      get-or-create-key  
root@node01:~# ceph auth get-or-create client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# 
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry mon 'allow r' osd 'allow rwx pool=rbd-data'
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# ceph auth get-or-create-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
root@node01:~# 
root@node01:~# ceph auth print-key client.jerry
AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==root@node01:~# 
root@node01:~# 
root@node01:~# 
root@node01:~# ceph auth  get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth caps client.jerry mon 'allow rw' osd 'allow r pool=rbd-data'
updated caps for client.jerry
root@node01:~# ceph auth get client.jerry
[client.jerry]
    key = AQBb5SRhY64XORAAHS+d0M/q8UixCa013knHQw==
    caps mon = "allow rw"
    caps osd = "allow r pool=rbd-data"
exported keyring for client.jerry
root@node01:~# ceph auth  del client.jerry
updated
root@node01:~# ceph auth get client.jerry
Error ENOENT: failed to find client.jerry in keyring
root@node01:~# 



5.6.1:通 通 过 秘 钥 环 文 件 备 份 与 恢 复 用

root@node01:~# ceph auth get-or-create client.user1 mon 'allow r' osd 'allow rwx pool=rbd-data'
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ls
ceph-deploy  cluster  snap
root@node01:~# ceph-authtool --help
usage: ceph-authtool keyringfile [OPTIONS]...
where the options are:
  -l, --list                    will list all keys and capabilities present in
                                the keyring
  -p, --print-key               will print an encoded key for the specified
                                entityname. This is suitable for the
                                'mount -o secret=..' argument
  -C, --create-keyring          will create a new keyring, overwriting any
                                existing keyringfile
  -g, --gen-key                 will generate a new secret key for the
                                specified entityname
  --gen-print-key               will generate a new secret key without set it
                                to the keyringfile, prints the secret to stdout
  --import-keyring FILE         will import the content of a given keyring
                                into the keyringfile
  -n NAME, --name NAME          specify entityname to operate on
  -a BASE64, --add-key BASE64   will add an encoded key to the keyring
  --cap SUBSYSTEM CAPABILITY    will set the capability for given subsystem
  --caps CAPSFILE               will set all of capabilities associated with a
                                given key, for all subsystems
  --mode MODE                   will set the desired file mode to the keyring
                                e.g: '0644', defaults to '0600'
root@node01:~# ceph-authtool -C ceph.client.user1.keyring
creating ceph.client.user1.keyring
root@node01:~# cat ceph.client.user1.keyring 
root@node01:~# file ceph.client.user1.keyring 
ceph.client.user1.keyring: empty
root@node01:~# 
root@node01:~# ceph auth  get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# ceph auth  get client.user1 -o ceph.client.user1.keyring 
exported keyring for client.user1
root@node01:~# cat ceph.client.user1.keyring 
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 

从 keyring 文件恢复用户认证信息:


root@node01:~# 
root@node01:~# ceph auth del client.user1
updated
root@node01:~# ceph auth get client.user1
Error ENOENT: failed to find client.user1 in keyring
root@node01:~# 
root@node01:~# ceph auth import -i ceph.client.user1.keyring 
imported keyring
root@node01:~# 
root@node01:~# ceph auth get client.user1
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
exported keyring for client.user1
root@node01:~# 





root@node01:~# ceph-authtool -C ceph.client.tom.keyring
creating ceph.client.tom.keyring
root@node01:~# ceph auth get client.tom -o ceph.client.tom.keyring 
exported keyring for client.tom
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# ceph-authtool -l ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# ceph-authtool ceph.client.user1.keyring --import-keyring ceph.client.tom.keyring 
importing contents of ceph.client.tom.keyring into ceph.client.user1.keyring
root@node01:~# echo $?
0
root@node01:~# cat ceph.client.tom.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
root@node01:~# 
root@node01:~# cat ceph.client.user1.keyring 
[client.tom]
    key = AQAQ5SRhNPftJBAA3lKYmTsgyeA1OQOVo2AwZQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-date"
[client.user1]
    key = AQDR5yRhiTkqJhAAY5ZmSnVKf/1/BGr/q0OTaQ==
    caps mon = "allow r"
    caps osd = "allow rwx pool=rbd-data"
root@node01:~# 





普通用户挂载
同步配置文件：
root@node01:~# ls
ceph.client.tom.keyring  ceph.client.user1.keyring  ceph-deploy  client.client.user1.keyring  client.user1.keyring  cluster  snap
root@node01:~# scp ceph.client.user1.keyring ceph-deploy/ceph.conf root@node04:/etc/ceph/
root@node04's password: 
ceph.client.user1.keyring                                                                                                                                                       100%  244   263.4KB/s   00:00    
ceph.conf                                                                                                                                                                       100%  265    50.5KB/s   00:00    
root@node01:~#

客户端操作

root@node04:/etc/netplan# apt install ceph-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4 librados2
  libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd python3-rgw
Suggested packages:
  ceph-base ceph-mds
The following NEW packages will be installed:
  ceph-common ibverbs-providers libbabeltrace1 libcephfs2 libdw1 libgoogle-perftools4 libibverbs1 libjaeger libleveldb1d liblttng-ust-ctl4 liblttng-ust0 liblua5.3-0 libnl-route-3-200 liboath0 librabbitmq4
  librados2 libradosstriper1 librbd1 librdkafka1 librdmacm1 librgw2 libsnappy1v5 libtcmalloc-minimal4 python3-ceph-argparse python3-ceph-common python3-cephfs python3-prettytable python3-rados python3-rbd
  python3-rgw
0 upgraded, 30 newly installed, 0 to remove and 79 not upgraded.
Need to get 37.3 MB of archives.
After this operation, 152 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-pacific focal/main amd64 libjaeger amd64 16.2.5-1focal [3,780 B]

中间太长了，我就删除了点

Setting up ceph-common (16.2.5-1focal) ...
Adding group ceph....done
Adding system user ceph....done
Setting system user ceph properties....done
chown: cannot access '/var/log/ceph/*.log*': No such file or directory
Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target.
Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
root@node04:/etc/netplan# ls /etc/ceph/
rbdmap
root@node04:/etc/netplan# ls /etc/ceph/
ceph.client.user1.keyring  ceph.conf  rbdmap
root@node04:/etc/netplan# cd /etc/ceph/
root@node04:/etc/ceph# ll
total 20
drwxr-xr-x  2 root root 4096 Aug 24 21:10 ./
drwxr-xr-x 96 root root 4096 Aug 24 21:09 ../
-rw-------  1 root root  244 Aug 24 21:10 ceph.client.user1.keyring
-rw-r--r--  1 root root  265 Aug 24 21:10 ceph.conf
-rw-r--r--  1 root root   92 Jul  8 22:16 rbdmap
root@node04:/etc/ceph# ceph --user user1 -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4225 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 70m)
    mgr: node01(active, since 70m), standbys: node02, node03
    osd: 6 osds: 6 up (since 70m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 7 objects, 54 B
    usage:   44 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --user user1 -p rbd-data map rbdtest1.img
/dev/rbd0
rbd: --user is deprecated, use --id
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
rbd: warning: image already mapped as /dev/rbd0
/dev/rbd1
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
rbd: rbd-data/rbdtest1.img: mapped more than once, unmapping /dev/rbd0 only
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd1                      252:16   0    1G  0 disk 
root@node04:/etc/ceph# rbd --id user1 unmap rbd-data/rbdtest1.img
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
root@node04:/etc/ceph# 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# rbd --id user1 -p rbd-data map rbdtest1.img
/dev/rbd0
root@node04:/etc/ceph# lsblk 
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0                       7:0    0 55.4M  1 loop /snap/core18/2128
loop1                       7:1    0 55.4M  1 loop /snap/core18/1944
loop2                       7:2    0 31.1M  1 loop /snap/snapd/10707
loop3                       7:3    0 69.9M  1 loop /snap/lxd/19188
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12704
loop5                       7:5    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   40G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   39G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   20G  0 lvm  /
sr0                        11:0    1  1.1G  0 rom  
rbd0                      252:0    0    1G  0 disk 
root@node04:/etc/ceph# 
root@node04:/etc/ceph# mkfs.xfs /dev/rbd0
meta-data=/dev/rbd0              isize=512    agcount=8, agsize=32768 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=16     swidth=16 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
root@node04:/etc/ceph# mount /dev/rbd0 /mnt/
root@node04:/etc/ceph# cp /etc/passwd /mnt/
root@node04:/etc/ceph# cd /mnt/
root@node04:/mnt# ls
passwd
root@node04:/mnt# tail passwd 
uuidd:x:107:112::/run/uuidd:/usr/sbin/nologin
tcpdump:x:108:113::/nonexistent:/usr/sbin/nologin
landscape:x:109:115::/var/lib/landscape:/usr/sbin/nologin
pollinate:x:110:1::/var/cache/pollinate:/bin/false
usbmux:x:111:46:usbmux daemon,,,:/var/lib/usbmux:/usr/sbin/nologin
sshd:x:112:65534::/run/sshd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
vmuser:x:1000:1000:vmuser:/home/vmuser:/bin/bash
lxd:x:998:100::/var/snap/lxd/common/lxd:/bin/false
ceph:x:64045:64045:Ceph storage service:/var/lib/ceph:/usr/sbin/nologin
root@node04:/mnt# 



cephfs挂载：
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 4986 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 83m)
    mgr: node01(active, since 83m), standbys: node02, node03
    osd: 6 osds: 6 up (since 83m), 6 in (since 7d)
 
  data:
    pools:   3 pools, 65 pgs
    objects: 18 objects, 14 MiB
    usage:   132 MiB used, 240 GiB / 240 GiB avail
    pgs:     65 active+clean
 
root@node01:~/ceph-deploy# ceph osd lspools
1 device_health_metrics
2 mypool1
3 rbd-data
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph osd pool create cephfs-metadata 16 16
pool 'cephfs-metadata' created
root@node01:~/ceph-deploy# ceph osd pool create cephfs-data 32 32
pool 'cephfs-data' created
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5126 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 85m)
    mgr: node01(active, since 85m), standbys: node02, node03
    osd: 6 osds: 6 up (since 85m), 6 in (since 7d)
 
  data:
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs new mycephfs ceph-metadata cephfs-data
Error ENOENT: pool 'ceph-metadata' does not exist
root@node01:~/ceph-deploy# ceph fs new mycephfs cephfs-metadata cephfs-data
new fs with metadata pool 4 and data pool 5
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5221 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 87m)
    mgr: node01(active, since 87m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 87m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ceph mds status mycephfs
no valid command found; 10 closest matches:
mds metadata [<who>]
mds count-metadata <property>
mds versions
mds compat show
mds ok-to-stop <ids>...
mds fail <role_or_gid>
mds repaired <role>
mds rm <gid:int>
mds compat rm_compat <feature:int>
mds compat rm_incompat <feature:int>
Error EINVAL: invalid command
root@node01:~/ceph-deploy# ceph fs status mycephfs
mycephfs - 0 clients
========
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata     0   75.9G  
  cephfs-data      data       0   75.9G  
root@node01:~/ceph-deploy# ceph -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_ERR
            1 filesystem is offline
            1 filesystem is online with fewer MDS than max_mds
            1 slow ops, oldest one blocked for 5291 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 88m)
    mgr: node01(active, since 88m), standbys: node02, node03
    mds: 0/0 daemons up
    osd: 6 osds: 6 up (since 88m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 18 objects, 14 MiB
    usage:   137 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:0
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-mgr.keyring  ceph.bootstrap-osd.keyring  ceph.bootstrap-rgw.keyring  ceph.client.admin.keyring  ceph.conf  ceph-deploy-ceph.log  ceph.mon.keyring
root@node01:~/ceph-deploy# ceph-de
ceph-debugpack  ceph-dencoder   ceph-deploy     
root@node01:~/ceph-deploy# ceph-deploy mds create node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9007af8a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f9007acddd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node01', 'node01')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node01:node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node01
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node01][WARNIN] mds keyring does not exist yet, creating one
[node01][DEBUG ] create a keyring file
[node01][DEBUG ] create path if it doesn't exist
[node01][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node01 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node01/keyring
[node01][INFO  ] Running command: systemctl enable ceph-mds@node01
[node01][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node01.service → /lib/systemd/system/ceph-mds@.service.
[node01][INFO  ] Running command: systemctl start ceph-mds@node01
[node01][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active}
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth add client.jack mon 'allow r' mds 'allow rw' osd 'allow rwx pool=cephfs-data'
added key for client.jack
root@node01:~/ceph-deploy# ceph auth get client.jack
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
exported keyring for client.jack
root@node01:~/ceph-deploy# ceph auth get client.admin
[client.admin]
    key = AQBErhthY4YdIhAANKTOMAjkzpKkHSkXSoNpaQ==
    caps mds = "allow *"
    caps mgr = "allow *"
    caps mon = "allow *"
    caps osd = "allow *"
exported keyring for client.admin
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get client.jack -o ceph.client.jack.keyring
exported keyring for client.jack
root@node01:~/ceph-deploy# cat ceph.client.jack.keyring
[client.jack]
    key = AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==
    caps mds = "allow rw"
    caps mon = "allow r"
    caps osd = "allow rwx pool=cephfs-data"
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# scp ceph.client.jack.keyring node04:/etc/ceph/
root@node04's password: 
ceph.client.jack.keyring                                                                                                                                                        100%  148   118.7KB/s   00:00    
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph auth get-or-create-key cliet.jack
Error EINVAL: bad entity name
root@node01:~/ceph-deploy# ceph auth print-key cliet.jack
Error EINVAL: invalid entity_auth cliet.jack
root@node01:~/ceph-deploy# ceph auth print-key client.jack 
AQD29CRh1IjhChAAjYT5Ydmp/cVYuVfKeAaBfw==root@node01:~/ceph-deploy# ceph auth print-key client.jack > jack.key
root@node01:~/ceph-deploy# scp jack.key node04:/etc/ceph/
root@node04's password: 
jack.key                                                                                                                                                                        100%   40    26.3KB/s   00:00    
root@node01:~/ceph-deploy#


root@node04:~# ceph --user jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5671 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# ceph --id jack -s
  cluster:
    id:     9138c3cf-f529-4be6-ba84-97fcab59844b
    health: HEALTH_WARN
            1 slow ops, oldest one blocked for 5676 sec, mon.node02 has slow ops
 
  services:
    mon: 3 daemons, quorum node01,node02,node03 (age 94m)
    mgr: node01(active, since 95m), standbys: node02, node03
    mds: 1/1 daemons up
    osd: 6 osds: 6 up (since 94m), 6 in (since 7d)
 
  data:
    volumes: 1/1 healthy
    pools:   5 pools, 113 pgs
    objects: 40 objects, 14 MiB
    usage:   138 MiB used, 240 GiB / 240 GiB avail
    pgs:     113 active+clean
 
root@node04:~# 
root@node04:~# mkdir /data
root@node04:~# 
root@node04:~# mount -t ceph node01:6789,node02:6789,node03:6789:/ /data/ -o name=jack,secretfile=/etc/ceph/jack.key
root@node04:~# df -hT
Filesystem                                                    Type      Size  Used Avail Use% Mounted on
udev                                                          devtmpfs  936M     0  936M   0% /dev
tmpfs                                                         tmpfs     196M  1.2M  195M   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv                             ext4       20G  4.5G   15G  25% /
tmpfs                                                         tmpfs     980M     0  980M   0% /dev/shm
tmpfs                                                         tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                                                         tmpfs     980M     0  980M   0% /sys/fs/cgroup
/dev/sda2                                                     ext4      976M  106M  804M  12% /boot
/dev/loop0                                                    squashfs   56M   56M     0 100% /snap/core18/2128
/dev/loop1                                                    squashfs   56M   56M     0 100% /snap/core18/1944
/dev/loop2                                                    squashfs   32M   32M     0 100% /snap/snapd/10707
/dev/loop3                                                    squashfs   70M   70M     0 100% /snap/lxd/19188
tmpfs                                                         tmpfs     196M     0  196M   0% /run/user/0
/dev/loop4                                                    squashfs   33M   33M     0 100% /snap/snapd/12704
/dev/loop5                                                    squashfs   71M   71M     0 100% /snap/lxd/21029
/dev/rbd0                                                     xfs      1014M   40M  975M   4% /mnt
192.168.11.210:6789,192.168.11.220:6789,192.168.11.230:6789:/ ceph       76G     0   76G   0% /data
root@node04:~# cd /data/
root@node04:/data# echo 'test content' > testfile.txt
root@node04:/data# cat testfile.txt 
test content
root@node04:/data# 

上面是用户创建及挂载，下面是：mds高可用情况：

root@node01:~/ceph-deploy# ceph-deploy mds create node02
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node02
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7febd49fda50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7febd49d2dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node02', 'node02')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node02:node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node02
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node02][WARNIN] mds keyring does not exist yet, creating one
[node02][DEBUG ] create a keyring file
[node02][DEBUG ] create path if it doesn't exist
[node02][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node02 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node02/keyring
[node02][INFO  ] Running command: systemctl enable ceph-mds@node02
[node02][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node02.service → /lib/systemd/system/ceph-mds@.service.
[node02][INFO  ] Running command: systemctl start ceph-mds@node02
[node02][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy mds create node03
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node03
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f1d42589a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f1d4255edd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node03', 'node03')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node03:node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node03
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node03][WARNIN] mds keyring does not exist yet, creating one
[node03][DEBUG ] create a keyring file
[node03][DEBUG ] create path if it doesn't exist
[node03][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node03 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node03/keyring
[node03][INFO  ] Running command: systemctl enable ceph-mds@node03
[node03][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node03.service → /lib/systemd/system/ceph-mds@.service.
[node03][INFO  ] Running command: systemctl start ceph-mds@node03
[node03][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 1 up:standby
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 2 up:standby
root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f72374d4a50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f72374a9dd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
bash: python2: command not found
[ceph_deploy.mds][ERROR ] connecting to host: node04 resulted in errors: IOError cannot send (already closed?)
[ceph_deploy][ERROR ] GenericError: Failed to create 1 MDSs

root@node01:~/ceph-deploy# ceph-deploy mds create node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy mds create node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : create
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fa90baa50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function mds at 0x7f4fa908fdd0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  mds                           : [('node04', 'node04')]
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.mds][DEBUG ] Deploying mds, cluster ceph hosts node04:node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[ceph_deploy.mds][INFO  ] Distro info: Ubuntu 20.04 focal
[ceph_deploy.mds][DEBUG ] remote host will use systemd
[ceph_deploy.mds][DEBUG ] deploying mds bootstrap to node04
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[node04][WARNIN] mds keyring does not exist yet, creating one
[node04][DEBUG ] create a keyring file
[node04][DEBUG ] create path if it doesn't exist
[node04][INFO  ] Running command: ceph --cluster ceph --name client.bootstrap-mds --keyring /var/lib/ceph/bootstrap-mds/ceph.keyring auth get-or-create mds.node04 osd allow rwx mds allow mon allow profile mds -o /var/lib/ceph/mds/ceph-node04/keyring
[node04][INFO  ] Running command: systemctl enable ceph-mds@node04
[node04][WARNIN] Created symlink /etc/systemd/system/ceph-mds.target.wants/ceph-mds@node04.service → /lib/systemd/system/ceph-mds@.service.
[node04][INFO  ] Running command: systemctl start ceph-mds@node04
[node04][INFO  ] Running command: systemctl enable ceph.target
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph mds stat
mycephfs:1 {0=node01=up:active} 3 up:standby
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   108k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
   node02    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    5
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:29:55.774998+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    1
in    0
up    {0=65409}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs set mycephfs max_mds 2
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs get mycephfs
Filesystem 'mycephfs' (1)
fs_name    mycephfs
epoch    12
flags    12
created    2021-08-24T21:27:30.730136+0800
modified    2021-08-24T21:44:44.248039+0800
tableserver    0
root    0
session_timeout    60
session_autoclose    300
max_file_size    1099511627776
required_client_features    {}
last_failure    0
last_failure_osd_epoch    0
compat    compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2}
max_mds    2
in    0,1
up    {0=65409,1=64432}
failed    
damaged    
stopped    
data_pools    [5]
metadata_pool    4
inline_data    disabled
balancer    
standby_count_wanted    1
[mds.node01{0:65409} state up:active seq 2 addr [v2:192.168.11.210:6810/3443284017,v1:192.168.11.210:6811/3443284017]]
[mds.node02{1:64432} state up:active seq 41 addr [v2:192.168.11.220:6808/4242415336,v1:192.168.11.220:6809/4242415336]]
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node01  Reqs:    0 /s    11     14     12      2   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 



root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart ceph
ceph-crash.service       ceph-mds@node01.service  ceph-mgr@node01.service  ceph-mon@node01.service  ceph-osd@0.service       ceph-osd.target          ceph.service             
ceph-fuse.target         ceph-mds.target          ceph-mgr.target          ceph-mon.target          ceph-osd@3.service       ceph-radosgw.target      ceph.target              
root@node01:~/ceph-deploy# systemctl restart ceph-m
ceph-mds@node01.service  ceph-mds.target          ceph-mgr@node01.service  ceph-mgr.target          ceph-mon@node01.service  ceph-mon.target          
root@node01:~/ceph-deploy# systemctl restart ceph-mds
ceph-mds@node01.service  ceph-mds.target          
root@node01:~/ceph-deploy# systemctl restart ceph-mds@node01.service 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s     1      4      2      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node04
mds_standby_replay = true
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# ceph-deploy config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4fdbb800f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f4fdbbd6350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy][ERROR ] GenericError: Failed to config 1 hosts

root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node01
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node01
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f201b0280f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f201b07e350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf config push node02 node02 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  subcommand                    : push
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f549232a0f0>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node02', 'node02', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function config at 0x7f5492380350>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.config][DEBUG ] Pushing config to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart ceph-mds@node01.service 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f9fa57ccf50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f9fa58a54d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
Permission denied, please try again.
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy][ERROR ] GenericError: Failed to configure 1 admin hosts

root@node01:~/ceph-deploy# ceph-deploy admin node01 node02 node03 node04 --overwrite-conf 
usage: ceph-deploy [-h] [-v | -q] [--version] [--username USERNAME]
                   [--overwrite-conf] [--ceph-conf CEPH_CONF]
                   COMMAND ...
ceph-deploy: error: unrecognized arguments: --overwrite-conf
root@node01:~/ceph-deploy# ceph-deploy --overwrite-conf admin node01 node02 node03 node04 
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /usr/bin/ceph-deploy --overwrite-conf admin node01 node02 node03 node04
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : True
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f41f97a5f50>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  client                        : ['node01', 'node02', 'node03', 'node04']
[ceph_deploy.cli][INFO  ]  func                          : <function admin at 0x7f41f987e4d0>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node01
[node01][DEBUG ] connected to host: node01 
[node01][DEBUG ] detect platform information from remote host
[node01][DEBUG ] detect machine type
[node01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node02
[node02][DEBUG ] connected to host: node02 
[node02][DEBUG ] detect platform information from remote host
[node02][DEBUG ] detect machine type
[node02][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node03
[node03][DEBUG ] connected to host: node03 
[node03][DEBUG ] detect platform information from remote host
[node03][DEBUG ] detect machine type
[node03][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node04
root@node04's password: 
root@node04's password: 
[node04][DEBUG ] connected to host: node04 
[node04][DEBUG ] detect platform information from remote host
[node04][DEBUG ] detect machine type
[node04][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ls
ceph.bootstrap-mds.keyring  ceph.bootstrap-osd.keyring  ceph.client.admin.keyring  ceph.conf             ceph.mon.keyring
ceph.bootstrap-mgr.keyring  ceph.bootstrap-rgw.keyring  ceph.client.jack.keyring   ceph-deploy-ceph.log  jack.key
root@node01:~/ceph-deploy# vim ceph.conf 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node04  Reqs:    0 /s    11     14     12      1   
 1    active  node02  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s     1      4      2      0   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node01  Reqs:    0 /s    10     13     11      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node04    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# cat ceph.c
cat: ceph.c: No such file or directory
root@node01:~/ceph-deploy# cat ceph.conf 
[global]
fsid = 9138c3cf-f529-4be6-ba84-97fcab59844b
public_network = 192.168.11.0/24
cluster_network = 192.168.22.0/24
mon_initial_members = node01
mon_host = 192.168.11.210
auth_cluster_required = cephx
auth_service_required = cephx
auth_client_required = cephx

[mds.node04]
mds_standby_for_name = node03
mds_standby_replay = true
[mds.node02]
mds_standby_for_name = node01
mds_standby_replay = true
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# systemctl restart ceph-mds@node01.service 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy# ceph fs status 
mycephfs - 1 clients
========
RANK  STATE    MDS       ACTIVITY     DNS    INOS   DIRS   CAPS  
 0    active  node02  Reqs:    0 /s    11     14     12      1   
 1    active  node04  Reqs:    0 /s    10     13     12      0   
      POOL         TYPE     USED  AVAIL  
cephfs-metadata  metadata   180k  75.9G  
  cephfs-data      data    12.0k  75.9G  
STANDBY MDS  
   node03    
   node01    
MDS version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)
root@node01:~/ceph-deploy# 
root@node01:~/ceph-deploy#