MongoDB 4.2.0三节点副本集部署

最近因业务发展,需要部署一套三节点的MongoDB副本集,现将部署过程总结如下.

一. 节点规划

主机名	IP地址	系统版本	数据库版本	安装目录	初始角色
ec1-notificationdb-01	10.191.101.115	CentOS Linux release 7.9.2009 (Core)	MongoDB v4.2.12	/usr/local/mongodb	primary
ec1-notificationdb-02	10.191.101.21				secondary
ec1-notificationdb-03	10.191.101.172				secondary

二. 系统配置(各节点)

1. 主机名解析

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
10.191.101.115    ec1-notificationdb-01 ec1-notificationdb-01.mypna.com
10.191.101.21    ec1-notificationdb-02 ec1-notificationdb-02.mypna.com
10.191.101.172    ec1-notificationdb-03 ec1-notificationdb-03.mypna.com

2. 新建用户和组

# groupadd -r dba
# useradd -r -g dba -G root mongodb
# mkdir /usr/local/mongodb
# chown -R mongodb.dba /usr/local/mongodb
# su - mongodb
$ cat .bash_profile
......
PATH=/usr/local/mongodb/bin:$PATH:$HOME/.local/bin:$HOME/bin
......

3. 内核参数优化/etc/sysctl.conf

vm.overcommit_memory = 1
vm.swappiness = 0
fs.aio-max-nr = 1048576
fs.file-max = 7672460
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
net.core.somaxconn = 4096
kernel.sem = 50100 64128000 50100 1280
kernel.shmall = 5242880
kernel.shmmax = 12884901888

4. 禁用Transparent Huge Pages (THP)

# echo "never" > /sys/kernel/mm/transparent_hugepage/enabled
# echo "never" >  /sys/kernel/mm/transparent_hugepage/defrag
# cat /etc/rc.local
......
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
  echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi

if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
   echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
......

5. 系统资源使用限制

# cat /etc/security/limits.conf
......
mongodb soft nofile 65535
mongodb hard nofile 65535
mongodb soft nproc 65535
mongodb hard nproc 65535
mongodb fsize unlimited unlimited
mongodb cpu unlimited unlimited
mongodb as unlimited unlimited
mongodb memlock unlimited unlimited
......

6. 系统依赖包安装

# yum -y install libcurl openssl

三. 数据库配置

1. 数据库安装

# su - mongodb
$ cd /usr/local/mongodb
$ wget https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-4.2.12.tgz
$ tar -zxf mongodb-linux-x86_64-rhel70-4.2.12.tgz
$ mv mongodb-linux-x86_64-rhel70-4.2.12/* .
$ rm -rf mongodb-linux-x86_64-rhel70-4.2.12*
$ mkdir {data,log,conf}
$ ls -lh
total 324K
drwxr-xr-x 2 mongodb dba  231 Jan 27 06:47 bin
drwxr-xr-x -2 mongodb dba 231 Jan 28 09:01 conf
drwxr-xr-x 7 mongodb dba  278 Jan 28 09:01 data
-rw-r--r-- 1 mongodb dba  30K Jan 19 22:06 LICENSE-Community.txt
drwxr-xr-x 2 mongodb dba   25 Jan 28 01:41 log
-rw-r--r-- 1 mongodb dba  17K Jan 19 22:06 MPL-2
-rw-r--r-- 1 mongodb dba 2.6K Jan 19 22:06 README
-rw-r--r-- 1 mongodb dba  74K Jan 19 22:06 THIRD-PARTY-NOTICES
-rw-r--r-- 1 mongodb dba 180K Jan 19 22:08 THIRD-PARTY-NOTICES.gotools

2. 新建MongodDB key文件

仅需在任意一个节点配置,这里以ec1-notificationdb-01为例

$ openssl rand -base64 756 > /usr/local/mongodb/keyFile
$ chmod 400 keyFile

将该文件复制到其他两节点

$ rsync -az --progress keyFile mongodb@ec1-notificationdb-02:/usr/local/mongodb/
$ rsync -az --progress keyFile mongodb@ec1-notificationdb-03:/usr/local/mongodb/

3. 新建MongoDB配置文件

$ cat /usr/local/mongodb/conf/mongodb.conf 

systemLog:
  path: "/usr/local/mongodb/log/mongodb.log"
  logAppend: true
  logRotate: rename
  destination: file
  timeStampFormat: iso8601-local    

processManagement:
  fork: true
  pidFilePath: "/usr/local/mongodb/mongodb_27017.pid"

net:
  port: 27017
  bindIpAll: true
  maxIncomingConnections: 10000
  wireObjectCheck: true
  ipv6: true
  tls:
    mode: disabled

security:
  keyFile: "/usr/local/mongodb/keyFile"
  authorization: enabled

storage:
  dbPath: "/usr/local/mongodb/data"
  journal:
    enabled: true
    commitIntervalMs: 100
  directoryPerDB: true
  syncPeriodSecs: 60
  engine: wiredTiger
  wiredTiger:
    engineConfig:
      cacheSizeGB: 1

# operationProfiling:
#  mode: slowOp
#  slowOpThresholdMs: 100
#  slowOpSampleRate: 1.0

replication:
   oplogSizeMB: 2048
   replSetName: repl
   enableMajorityReadConcern: true

4. 使用systemctl管理MongoDB服务

# cat /usr/lib/systemd/system/mongodb.service 
[Unit]
Description=MongoDB document database.
Documentation=https://docs.mongodb.com/manual/
After=syslog.target
After=network.target
Wants=network-online.target

[Service]
User=mongodb
Group=dba
Type=forking
Restart=always

OOMScoreAdjust=-1000
Environment="OPTIONS=-f /usr/local/mongodb/conf/mongodb.conf"
PIDFile=/usr/local/mongodb/mongodb_27017.pid
ExecStart=/usr/local/mongodb/bin/mongod $OPTIONS
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/usr/local/mongodb/bin/mongod --shutdown $OPTIONS
KillMode=mixed
KillSignal=SIGINT
TimeoutSec=0
# file size
LimitFSIZE=infinity
# cpu time
LimitCPU=infinity
# virtual memory size
LimitAS=infinity
# open files
LimitNOFILE=64000
# processes/threads
LimitNPROC=64000
# locked memory
LimitMEMLOCK=infinity
# total threads (user+kernel)
TasksMax=infinity
TasksAccounting=false 

[Install]
WantedBy=multi-user.target

# systemctl enable mongodb
# systemctl start mongodb
# systemctl status mongodb

四. 副本集配置

1. 副本集初始化

 在任意一节点,通过socket进入MongoDB,这里以ec1-notificationdb-01为例,实际上在哪台机器上进行集群初始化,哪台机器会最优先成为主节点.

$ mongo --norc
MongoDB shell version v4.2.12
connecting to: mongodb://127.0.0.1:27017/?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("4fa5aa80-3529-4e08-aa43-b0a9863be7e2") }
MongoDB server version: 4.2.12
> config = {
...     _id : "repl",
...      members : [
...          {_id : 0, host : "ec1-notificationdb-01.mypna.com:27017"},
...          {_id : 1, host : "ec1-notificationdb-02.mypna.com:27017"},
...          {_id : 2, host : "ec1-notificationdb-03.mypna.com:27017"}
...      ]
... }
{
    "_id" : "repl",
    "members" : [
        {
            "_id" : 0,
            "host" : "ec1-notificationdb-01.mypna.com:27017"
        },
        {
            "_id" : 1,
            "host" : "ec1-notificationdb-02.mypna.com:27017"
        },
        {
            "_id" : 2,
            "host" : "ec1-notificationdb-03.mypna.com:27017"
        }
    ]
}

> rs.initiate(config)
{ "ok" : 1 }

MongoDB副本集会进行primary选举,一分钟后再次查看当前节点的状态,执行集群初始化的节点的提示符会变成primary

repl:PRIMARY>

如果此时查看集群配置,会提示未被授权

repl:PRIMARY> rs.config()
2021-01-28T09:28:00.598+0000 E  QUERY    [js] uncaught exception: Error: Could not retrieve replica set config: {
    "operationTime" : Timestamp(1611826071, 1),
    "ok" : 0,
    "errmsg" : "not authorized on admin to execute command { replSetGetConfig: 1.0, lsid: { id: UUID("4fa5aa80-3529-4e08-aa43-b0a9863be7e2") }, $clusterTime: { clusterTime: Timestamp(1611826071, 1), signature: { hash: BinData(0, 148351074C034AAA363CD65968507A0ECF84D863), keyId: 6922739574590406659 } }, $db: "admin" }",
    "code" : 13,
    "codeName" : "Unauthorized",
    "$clusterTime" : {
        "clusterTime" : Timestamp(1611826071, 1),
        "signature" : {
            "hash" : BinData(0,"FINRB0wDSqo2PNZZaFB6Ds+E2GM="),
            "keyId" : NumberLong("6922739574590406659")
        }
    }
} :
rs.conf@src/mongo/shell/utils.js:1547:11
@(shell):1:1

2. 在primary节点新建超级用户

repl:PRIMARY> use admin
switched to db admin
repl:PRIMARY> db.createUser( { user : "dbadmin", pwd : "88888888", roles : [ { role : "root", db : "admin" } ] } )
Successfully added user: {
    "user" : "dbadmin",
    "roles" : [
        {
            "role" : "root",
            "db" : "admin"
        }
    ]
}

使用超级用户重新登录

$ mongo --username dbadmin --password 88888888 --authenticationDatabase admin
repl:PRIMARY> show dbs
admin   0.000GB
config  0.000GB
local   0.000GB
repl:PRIMARY> rs.config()
{
    "_id" : "repl",
    "version" : 1,
    "protocolVersion" : NumberLong(1),
    "writeConcernMajorityJournalDefault" : true,
    "members" : [
        {
            "_id" : 0,
            "host" : "ec1-notificationdb-01.mypna.com:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {
                
            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        },
        {
            "_id" : 1,
            "host" : "ec1-notificationdb-02.mypna.com:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {
                
            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        },
        {
            "_id" : 2,
            "host" : "ec1-notificationdb-03.mypna.com:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {
                
            },
            "slaveDelay" : NumberLong(0),
            "votes" : 1
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatIntervalMillis" : 2000,
        "heartbeatTimeoutSecs" : 10,
        "electionTimeoutMillis" : 10000,
        "catchUpTimeoutMillis" : -1,
        "catchUpTakeoverDelayMillis" : 30000,
        "getLastErrorModes" : {
            
        },
        "getLastErrorDefaults" : {
            "w" : 1,
            "wtimeout" : 0
        },
        "replicaSetId" : ObjectId("601282eb1deee0966902cd12")
    }
}

 3. MongoDB shell定制化

在各节点的mongodb家目录下回自动生一个隐藏文件.mongorc.js,默认该文件为空白,我们可以根据实际需要定义该shell提示符

$ cat .mongorc.js 
prompt = function() {
    if (typeof db == 'undefined') 
        return '(nodb)> ';

    try {
        db.runCommand( {getLastError:1} );
    }
    catch (e) {
        print(e);
    }

    var host = db.serverStatus().host;
    var user = db.runCommand({connectionStatus : 1}).authInfo.authenticatedUsers[0] 
    if (user) {
        return user.user + "@" + db + ":" + host + " " + defaultPrompt();
    } else {
        return "(anonymous)@" + db + ":" + host + " " + defaultPrompt();
    return db + " " + defaultPrompt();
    }
}

重新登录各节点,提示符如下所示

dbadmin@test:ec1-notificationdb-01 repl:PRIMARY>

dbadmin@test:ec1-notificationdb-02 repl:SECONDARY>

dbadmin@test:ec1-notificationdb-03 repl:SECONDARY>