cs6.8-oracle挂载ceph

https://ceph-users.ceph.narkive.com/EgcYJhbG/hammer-0-94-1-still-getting-feature-set-mismatch-for-cephfs-mount-requests

https://blog.51cto.com/michaelkang/1675822 

Client挂载方式有两种：内核kernal方式和fuse方式。（内核挂载方式需要2.6.34及其以后的版本才支持）

使用fuse方式挂载（注意：此方式读写效率很低）

centos6.8要成功挂载cephfs
1需要升级内核
2需要在ceph节点执行ceph osd crush tunables hammer
3然后用mount挂载

设置好源

[root@cu-dbs-161 yum.repos.d]# cat ceph.repo
[ceph]
name=ceph
baseurl=http://download.ceph.com/rpm-hammer/el6/x86_64/
gpgcheck=0

[ceph-noarch]
name=cephnoarch
baseurl=http://download.ceph.com/rpm-hammer/el6/noarch/
gpgcheck=0
[root@cu-dbs-161 yum.repos.d]# cat 163.repo
[ceph]
name=Ceph noarch packages
baseurl=http://mirrors.163.com/ceph/rpm-hammer/el6/x86_64/
enabled=1
gpgcheck=1
gpgkey=http://mirrors.163.com/ceph/keys/release.asc

安装软件

yum install ceph-fuse

升级内核
[root@cu-dbs-161 ceph]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
[root@cu-dbs-161 ceph]#
[root@cu-dbs-161 ceph]# rpm -Uvh http://www.elrepo.org/elrepo-release-6-8.el6.elrepo.noarch.rpm

[root@cu-dbs-161 ceph]# yum --enablerepo=elrepo-kernel -y install kernel-lt

[root@cu-dbs-161 ceph]# vi /boot/grub/grub.conf
将内核修改为4.0的，其实就是将defaults=0就可以了
[root@cu-dbs-161 ceph]# reboot

[root@cu-dbs-151 ~]# modprobe ceph
[root@cu-dbs-151 ~]# lsmod |grep ceph
ceph 175251 0
libceph 161546 1 ceph
libcrc32c 1388 1 libceph
fscache 57792 1 ceph

在ceph节点上执行下面这个命令，客户端就可以挂载了
root@cu-pve04:/etc/ceph# ceph osd crush tunables hammer
adjusted tunables profile to hammer

==============================================
ceph osd crush tunables optimal

ceph osd crush show-tunables
ceph osd crush tunables legacy|argonaut|bobtail|firefly|hammer|jewel|optimal|default set crush tunables values to <profile>
ceph osd crush get-tunable straw_calc_version

May 13 17:52:24 cu-dbs-161 kernel: libceph: mon0 192.168.7.4:6789 feature set mismatch, my 107b84a842aca < server's 40107b84a842aca, missing 400000000000000
May 13 17:52:24 cu-dbs-161 kernel: libceph: mon0 192.168.7.4:6789 missing required protocol features
May 13 17:57:07 cu-dbs-161 kernel: libceph: no secret set (for auth_x protocol)
May 13 17:57:07 cu-dbs-161 kernel: libceph: error -22 on auth protocol 2 init
May 13 17:57:07 cu-dbs-161 kernel: libceph: client2428503 fsid 6e731e21-ac3a-493b-84bf-95f54481449c
May 13 17:58:07 cu-dbs-161 kernel: libceph: client2428569 fsid 6e731e21-ac3a-493b-84bf-95f54481449c
May 13 17:58:07 cu-dbs-161 kernel: libceph: mon0 192.168.7.4:6789 session established

==============================================

[root@cu-dbs-161 ceph]# mount -t ceph 192.168.7.4:6789:/ /ceph -o name=admin,secretfile=admin.keyring
[root@cu-dbs-161 ceph]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 83G 3.8G 75G 5% /
tmpfs 7.9G 72K 7.9G 1% /dev/shm
/dev/sda1 190M 74M 102M 42% /boot
192.168.7.4:6789:/ 53T 693G 52T 2% /ceph
[root@cu-dbs-161 ceph]# cd /ceph/
[root@cu-dbs-161 ceph]# ls
dump fileserver template
[root@cu-dbs-161 fileserver]# lsmod |grep ceph
ceph 174958 1
libceph 161546 1 ceph
libcrc32c 1388 1 libceph
fscache 57792 1 ceph

添加到开机启动项
vi /etc/fstab
192.168.7.4:6789,192.168.7.5:6789,192.168.7.6:6789:/ /ceph ceph name=admin,secretfile=/etc/ceph/admin.keyring 0 0

[root@cu-dbs-151 ceph]# cat admin.keyring
AQBR8NRcZL6tLhAAnuB+OPbBdI+t3G5B1beLHA==

这是个bug，不能用secretfile参数，或者直接在命令行上跟密钥。或者禁用ceph的认证

https://bugzilla.redhat.com/show_bug.cgi?id=1030402

[root@cu-dbs-151 ceph]# mount -t ceph 192.168.7.4:6789:/ /ceph -o name=admin,secretfile=admin.keyring
mount: wrong fs type, bad option, bad superblock on 192.168.7.4:6789:/,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so

[root@cu-dbs-151 ceph]# dmesg |tail
EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
Adding 16777212k swap on /dev/sda2. Priority:-1 extents:1 across:16777212k FS
audit: type=1305 audit(1561542914.822:3): audit_pid=3210 old=0 auid=4294967295 ses=4294967295 res=1
FS-Cache: Loaded
Key type ceph registered
libceph: loaded (mon/osd proto 15/24)
FS-Cache: Netfs 'ceph' registered for caching
ceph: loaded (mds proto 32)
libceph: bad option at 'secretfile=admin.keyring'
libceph: bad option at 'secretfile=admin.keyring'

[root@cu-dbs-151 ceph]# mount -t ceph 192.168.7.4:6789:/ /ceph -o name=admin,secret=AQBR8NRcZL6tLhAAnuB+OPbBdI+t3G5B1beLHA==
[root@cu-dbs-151 ceph]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 83G 3.6G 75G 5% /
tmpfs 16G 72K 16G 1% /dev/shm
/dev/sda1 190M 74M 103M 42% /boot
192.168.7.4:6789:/ 53T 945G 52T 2% /ceph