【对象存储】minio集群部署

minio 集群搭建

1. 创建三台虚拟机，本教程采用操作系统均为centos7版本：

操作系统	主机	节点
centos07	192.168.99.8	minio-1
centos07	192.168.99.9	minio-2
centos07	192.168.99.10	minio-2

所有节点配置主机名解析：

cat >> /etc/hosts << EOF
192.168.99.8 minio-1
192.168.99.9 minio-2
192.168.99.10 minio-3
EOF

在minio-1 节点：

docker run -d --name minio01 \
  --restart=always  --net=host \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=admin123" \
  -v /etc/localtime:/etc/localtime \
  -v /data01/minio:/data1/minio \
  -v /data02/minio:/data2/minio \
  -v /data03/minio:/data3/minio \
  -v /data04/minio:/data4/minio \
  -v /data05/minio:/data5/minio \
  -v /data06/minio:/data6/minio \
  -v /data07/minio:/data7/minio \
  -v /data08/minio:/data8/minio  \
   minio/minio:RELEASE.2021-11-24T23-19-33Z.fips server \
  --address "192.168.99.8:49000" \
  --console-address ":49001" \
  http://minio-{1...3}/data{1...8}/minio

在minio2节点执行：

docker run -d --name minio02 \
  --restart=always  --net=host \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=admin123" \
  -v /etc/localtime:/etc/localtime \
  -v /data01/minio:/data1/minio \
  -v /data02/minio:/data2/minio \
  -v /data03/minio:/data3/minio \
  -v /data04/minio:/data4/minio \
  -v /data05/minio:/data5/minio \
  -v /data06/minio:/data6/minio \
  -v /data07/minio:/data7/minio \
  -v /data08/minio:/data8/minio  \
  minio/minio:RELEASE.2021-11-24T23-19-33Z.fips server \
  --address "192.168.99.9:49000" \
  --console-address ":49001" \
  http://minio-{1...3}/data{1...8}/minio

在minio3节点执行：

docker run -d --name minio03 \
  --restart=always  --net=host \
  -e "MINIO_ROOT_USER=admin" \
  -e "MINIO_ROOT_PASSWORD=admin123" \
  -v /etc/localtime:/etc/localtime \
  -v /data01/minio:/data1/minio \
  -v /data02/minio:/data2/minio \
  -v /data03/minio:/data3/minio \
  -v /data04/minio:/data4/minio \
  -v /data05/minio:/data5/minio \
  -v /data06/minio:/data6/minio \
  -v /data07/minio:/data7/minio \
  -v /data08/minio:/data8/minio  \
   minio/minio:RELEASE.2021-11-24T23-19-33Z.fips server \
  --address "192.168.99.10:49000" \
  --console-address ":49001" \
  http://minio-{1...3}/data{1...8}/minio

客户端查看minio状态，3个节点及每个节点上的Drivers全部在线：

$ mc config host add minio http://192.168.99.8:49000 admin admin123
$ mc admin info minio
●  minio-1:49000
   Uptime: 14 minutes
   Version: 2021-11-24T23:19:33Z
   Network: 3/3 OK
   Drives: 4/4 OK

●  minio-2:49000
   Uptime: 12 minutes
   Version: 2021-11-24T23:19:33Z
   Network: 3/3 OK
   Drives: 4/4 OK

●  minio-3:49000
   Uptime: 8 minutes
   Version: 2021-11-24T23:19:33Z
   Network: 3/3 OK
   Drives: 4/4 OK

53 MiB Used, 1 Bucket, 4 Objects
12 drives online, 0 drives offline

nginx 实现负载均衡

1. 创建目录和文件

mkdir -p /cq-aiplatform/soft/nginx/conf
mkdir -p /cq-aiplatform/soft/nginx/html

创建配置文件，配置映射，要不然每次修改都要进入到容器内：

vim /cq-aiplatform/soft/nginx/conf/nginx.conf

复制以下内容到配置文件中：

events {
    worker_connections  1024;
}

http {
upstream minio {
    server 192.168.99.8:49000;
    server 192.168.99.9:49000;
    server 192.168.99.10:49000;
}

upstream console {
    ip_hash;
    server 192.168.99.8:49001;
    server 192.168.99.9:49001;
    server 192.168.99.10:49001;
}

server {
    listen       6333;
    server_name  192.168.99.11;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        proxy_connect_timeout 300;
        # Default is HTTP/1, keepalive is only enabled in HTTP/1.1
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        chunked_transfer_encoding off;

        proxy_pass http://minio;
    }
}

server {
    listen       6334;
    server_name  192.168.99.11;

    # To allow special characters in headers
    ignore_invalid_headers off;
    # Allow any size file to be uploaded.
    # Set to a value such as 1000m; to restrict file size to a specific value
    client_max_body_size 0;
    # To disable buffering
    proxy_buffering off;

    location / {
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-NginX-Proxy true;

        # This is necessary to pass the correct IP to be hashed
        real_ip_header X-Real-IP;

        proxy_connect_timeout 300;
            
        # To support websocket
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
            
        chunked_transfer_encoding off;

        proxy_pass http://console;
    }
}
}

启动docker：

docker run -d  --name nginx_minio  \ 
 -p 6333:6333 \
 -p 6334:6334  \
 -v /cq-aiplatform/soft/nginx/conf/nginx.conf:/etc/nginx/nginx.conf  \
 -v /cq-aiplatform/soft/nginx/logs:/var/log/nginx \
 -v /cq-aiplatform/soft/nginx/html:/usr/share/nginx/html \ 
 nginx

参考资料

1. Docker部署分布式MinIO存储集群

2. docker安装minio+nginx负载均衡配置

3. Enterprise-Grade Cloud Storage with NGINX Plus and Minio