• 003 ansible部署ceph集群


    介绍:在上一次的deploy部署ceph,虽然出了结果,最后的结果并没有满足最初的目的,现在尝试使用ansible部署一遍,看是否会有问题

    一、环境准备

    ceph1充当部署节点,ceph2,ceph3,ceph4充当ceph集群节点

    IP主机名节点系统
    172.25.250.10 ceph1 ceph-ansible Red Hat release 7.4
    172.25.250.11 ceph2 mon、mgr、osd Red Hat release 7.4
    172.25.250.12 ceph3 mon、mgr、osd Red Hat release 7.4
    172.25.250.10 ceph4 mon、mgr、osd Red Hat release 7.4

    1.1 ceph1配置hosts

    172.25.250.10  ceph1     
    172.25.250.11  ceph2     
    172.25.250.12  ceph3   
    172.25.250.13  ceph4

    1.2 配置免密登录

    [root@ceph1 ~]#  ssh-keygen

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph1

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph2

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph3

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph4

    二,部署ceph集群

    2.1部署节点安装ansible

    注:再部署之前,一般需要配置yum源和时间服务器ntp配置

    [root@ceph1 ~]# yum -y install ansible

    [root@ceph1 ~]# cd /etc/ansible/

    [root@ceph1 ansible]# ll

    2.2 定义hosts

    [root@ceph1 ansible]# vim hosts

    [mons]
    ceph2
    ceph3
    ceph4

    [mgrs]
    ceph2
    ceph3
    ceph4

    [osds]
    ceph2
    ceph3
    ceph4

    [clients]
    ceph1

    2.3 安装ceph-ansible

    [root@ceph1 ansible]#  yum -y install ceph-ansible

    [root@ceph1 ansible]#  cd /usr/share/ceph-ansible/

    [root@ceph1 ceph-ansible]# ls

    [root@ceph1 ceph-ansible]#  ansible all -m ping

    [root@ceph1 ceph-ansible]# cd /usr/share/ceph-ansible/group_vars/

    [root@ceph1 group_vars]# cp mons.yml.sample mons.yml

    [root@ceph1 group_vars]# cp mgrs.yml.sample mgrs.yml

    [root@ceph1 group_vars]# cp osds.yml.sample osds.yml

    [root@ceph1 group_vars]# cp clients.yml.sample clients.yml

    [root@ceph1 group_vars]# cp all.yml.sample all.yml

    [root@ceph1 group_vars]# vim all.yml

    fetch_directory: ~/ceph-ansible-keys
    ceph_repository_type: "cdn"
    ceph_origin: repository
    ceph_repository: rhcs
    ceph_rhcs_version: 3
    monitor_interface: eth0
    journal_size: 5120
    public_network: 172.25.250.0/24
    cluster_network: "{{ public_network }}"

    2.4 定义OSD

    [root@ceph1 group_vars]# vim osds.yml

    devices:
       - /dev/vdb
       - /dev/vdc
       - /dev/vdd
    osd_scenario: collocated

     [root@ceph1 group_vars]# grep -Ev "^$|^s*#"  *.yml

    2.5 定义ansible的入口文件

    [root@ceph1 group_vars]# cd ..

    [root@ceph1 ceph-ansible]# cp site.yml.sample site.yml

    [root@ceph1 ceph-ansible]# vim site.yml

    - hosts:
      - mons
    #  - agents
      - osds
    #  - mdss
    #  - rgws
    #  - nfss
    #  - restapis
    #  - rbdmirrors
      - clients
      - mgrs
    #  - iscsi-gws

     2.6 安装

     [root@ceph1 ceph-ansible]# ansible-playbook  site.yml

    安装如果有问题,会有报错,根据报错信息去修改yml文件,然后重新执行,出现下面的结果表示已经安装完成

    PLAY RECAP ****************************************************************
    ceph1                      : ok=42   changed=4    unreachable=0    failed=0   
    ceph2                      : ok=213  changed=9    unreachable=0    failed=0   
    ceph3                      : ok=196  changed=3    unreachable=0    failed=0   
    ceph4                      : ok=190  changed=28   unreachable=0    failed=0   

    2.7 检测验证

    [root@ceph2 ~]# ceph -s

    [root@ceph2 ~]# ceph osd tree

     

    所有的文件都存放在/etc/ceph下

    [root@ceph2 ~]# cd /etc/ceph/

    [root@ceph2 ceph]# ll

    ceph.conf是配置文件的名字,是因为没有指定集群的名字默认为ceph,配置文件的名字应为:cluster_name.conf

    [root@ceph2 ceph]# cat ceph.conf

    [global]
    fsid = 35a91e48-8244-4e96-a7ee-980ab989d20d
    mon initial members = ceph2,ceph3,ceph4
    mon host = 172.25.250.11,172.25.250.12,172.25.250.13
    public network = 172.25.250.0/24
    cluster network = 172.25.250.0/24
    [osd]
    osd mkfs type = xfs
    osd mkfs options xfs = -f -i size=2048
    osd mount options xfs = noatime,largeio,inode64,swalloc
    osd journal size = 5120

    查看磁盘,有一个日志分区和数据分区

    [root@ceph2 ceph]# fdisk -l

    查看ceph进程

    [root@ceph2 ceph]# ps aux|grep ceph

    2.8 节点扩充

    如果需要扩充节点,可以直接在部署节点的hosts文件,添加上主机名,执行部署,就会自己在原来的基础上继续扩充

    如,把ceph1的三个磁盘扩充到osd,则配置如下:

    [root@ceph1 ansible]# vim hosts

    [mons]
    ceph2
    ceph3
    ceph4
    
    [mgrs]
    ceph2
    ceph3
    ceph4
    
    [osds]
    ceph2
    ceph3
    ceph4
    ceph1
    
    [clients]
    ceph1

    再执行部署

    [root@ceph1 ceph-ansible]# ansible-playbook  site.yml

    如果部署错误,删除需要下面这个yml文件

    [root@ceph1 ceph-ansible]#  vim infrastructure-playbooks/purge-cluster.yml

    部分注释内容:
    # This playbook purges Ceph # It removes: packages, configuration files and ALL THE DATA # # Use it like this: # ansible
    -playbook purge-cluster.yml # Prompts for confirmation to purge, defaults to no and # doesn't purge the cluster. yes purges the cluster. # # ansible-playbook -e ireallymeanit=yes|no purge-cluster.yml # Overrides the prompt using -e option. Can be used in # automation scripts to avoid interactive prompt.

     执行删除:

    [root@ceph1 ceph-ansible]# ansible-playbook infrastructure-playbooks/purge-cluster.yml


     另一个环境安装

     一、环境准备,

    每个虚拟机只有一块多余的磁盘做osd

    IP主机名节点系统硬盘

    172.25.254.130

    ceph1 ceph-ansible CentOS Linux release 7.4.1708 (Core) /dev/sdb,dev/sdc,dev/sdd

    172.25.254.131

    ceph2 mon、mgr、osd CentOS Linux release 7.4.1708 (Core) /dev/sdb,dev/sdc,dev/sdd

    172.25.254.132

    ceph3 mon、mgr、osd CentOS Linux release 7.4.1708 (Core) /dev/sdb,dev/sdc,dev/sdd
    172.25.254.133 ceph4 mon、mgr、osd
    CentOS Linux release 7.4.1708 (Core)
    /dev/sdb,dev/sdc,dev/sdd

    1.1 配置hosts

    172.25.254.130  ceph1     
    172.25.254.131  ceph2     
    172.25.254.132  ceph3   
    172.25.254.133  ceph4

    1.2 配置免密登录

    root@ceph1 ~]#  ssh-keygen

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph1

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph2

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph3

    [root@ceph1 ~]# ssh-copy-id -i .ssh/id_rsa.pub ceph4

    1.3 内核进行升级

    rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
    rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
    yum --disablerepo=* --enablerepo=elrepo-kernel repolist
    yum --disablerepo=* --enablerepo=elrepo-kernel list kernel*
    yum --disablerepo=* --enablerepo=elrepo-kernel install -y kernel-ml.x86_64
    awk -F' '$1=="menuentry " {print $2}' /etc/grub2.cfg
    grub2-set-default 0
    reboot
    rpm -qa|grep kernel|grep 3.10|xargs yum remove -y 
    yum --disablerepo=* --enablerepo=elrepo-kernel install -y kernel-ml-tools.x86_64 
    rpm -qa|grep kernel
    systemctl stop firewalld
    systemctl disable firewalld 

    二、部署

    2.1 下载包

    [root@ceph1 ~]# wget -c https://github.com/ceph/ceph-ansible/archive/v3.1.7.tar.gz

    [root@ceph1 ~]# tar xf v3.1.7.tar.gz

    [root@ceph1 ~]# cd ceph-ansible-3.1.7

    2.3 修改inventory,添加主机信息

    [root@ceph1 ceph-ansible-3.1.7]# vim hosts

    [mons]
    ceph2
    ceph3
    ceph4
    
    [osds]
    ceph2
    ceph3
    ceph4
    
    [mgrs]
    ceph2
    ceph3
    ceph4
    
    [mdss]
    ceph2
    ceph3
    ceph4
    
    [clients]
    ceph1
    ceph2
    ceph3
    ceph4

    [root@ceph1 ceph-ansible-3.1.7]# ansible all -m ping -i hosts

    2.4 修改all.yml写入如下内容

    [root@ceph1 ceph-ansible-3.1.7]# cp group_vars/all.yml.sample group_vars/all.yml

    [root@ceph1 ceph-ansible-3.1.7]# cp group_vars/osds.yml.sample group_vars/osds.yml

    [root@ceph1 ceph-ansible-3.1.7]# cp site.yml.sample site.yml

    [root@ceph1 ceph-ansible-3.1.7]# vim group_vars/all.yml

    ceph_origin: repository
    ceph_repository: community
    ceph_mirror: http://mirrors.aliyun.com/ceph
    ceph_stable_key: http://mirrors.aliyun.com/ceph/keys/release.asc
    ceph_stable_release: luminous
    ceph_stable_repo: "{{ ceph_mirror }}/rpm-{{ ceph_stable_release }}"
    
    fsid: 54d55c64-d458-4208-9592-36ce881cbcb7 ##通过uuidgen生成
    generate_fsid: false
    
    cephx: true
    
    public_network: 172.25.254.0/20
    cluster_network: 172.25.254.0/20
    monitor_interface: ens33
    
    ceph_conf_overrides:
        global:
          rbd_default_features: 7
          auth cluster required: cephx
          auth service required: cephx
          auth client required: cephx
          osd journal size: 2048
          osd pool default size: 3
          osd pool default min size: 1
          mon_pg_warn_max_per_osd: 1024
          osd pool default pg num: 128
          osd pool default pgp num: 128
          max open files: 131072
          osd_deep_scrub_randomize_ratio: 0.01
    
        mgr:
          mgr modules: dashboard
    
        mon:
          mon_allow_pool_delete: true
    
        client:
          rbd_cache: true
          rbd_cache_size: 335544320
          rbd_cache_max_dirty: 134217728
          rbd_cache_max_dirty_age: 10
    
        osd:
          osd mkfs type: xfs
          ms_bind_port_max: 7100
          osd_client_message_size_cap: 2147483648
          osd_crush_update_on_start: true
          osd_deep_scrub_stride: 131072
          osd_disk_threads: 4
          osd_map_cache_bl_size: 128
          osd_max_object_name_len: 256
          osd_max_object_namespace_len: 64
          osd_max_write_size: 1024
          osd_op_threads: 8
    
          osd_recovery_op_priority: 1
          osd_recovery_max_active: 1
          osd_recovery_max_single_start: 1
          osd_recovery_max_chunk: 1048576
          osd_recovery_threads: 1
          osd_max_backfills: 4
          osd_scrub_begin_hour: 23
          osd_scrub_end_hour: 7

    [root@ceph1 ceph-ansible-3.1.7]# vim group_vars/osds.yml

    devices:
      - /dev/sdb
      - /dev/sdc
      - /dev/sdd
    osd_scenario: collocated
    osd_objectstore: bluestore

    [root@ceph1 ceph-ansible-3.1.7]# vim site.yml

    # Defines deployment design and assigns role to server groups
    - hosts:
      - mons
    #  - agents
      - osds
      - mdss
    #  - rgws
    #  - nfss
    #  - rbdmirrors
      - clients
      - mgrs

    2.5 执行部署 

    [root@ceph1 ceph-ansible-3.1.7]# ansible-playbook site.yml -i hosts

    -bash: ansible-playbook: command not found

    [root@ceph1 ceph-ansible-3.1.7]# yum install ansible -y

    [root@ceph1 ceph-ansible-3.1.7]# ansible-playbook site.yml -i hosts

    成功安装

    注:如果是不会过程出错,先清空集群,在进行部署

    cp infrastructure-playbooks/purge-cluster.yml purge-cluster.yml # 必须copy到项目根目录下
    ansible-playbook -i hosts purge-cluster.yml 

    不升级内核,也可以安装成功,但是没有测试集群的其他性能!

    参考链接:https://yq.aliyun.com/articles/624202

  • 相关阅读:
    作业2 求题目中的数
    2013 C#单元测试
    实现项目WC
    带括号多项式版四则运算
    20道简单加减法随机生成程序扩展版体会
    20道简单加减法随机生成程序
    Jeesite 集成微信支付接口
    第一节:JAVA 语言的学习目标
    vector(未完)
    关于phpstorm端口63342的修改经历
  • 原文地址:https://www.cnblogs.com/zyxnhr/p/10543814.html
Copyright © 2020-2023  润新知