k8s中安装rabbitmq集群

官方文档地址：https://www.rabbitmq.com/kubernetes/operator/quickstart-operator.html

要求

1.k8s版本要1.18及其以上
2.能够在master节点上使用kubectl命令来管理整个k8s集群
3.有默认的一个StorageClass存储类，因为默认情况下 RabbitMQ Operator 创建的 RabbitMQ 集群会为每个实例使用 StorageClass 分配一个 10G 的 PVC。但是官方提供的yaml文件中并没有设置使用StorageClass存储类的配置，所以需要一个默认的StorageClass存储类。

若没有这个默认的StorageClass存储类，则创建RabbitMQ 集群时创建的pod状态是Pending，具体原因是：pod has unbound immediate PersistentVolumeClaims
查看默认StorageClass储存类，注意default

# kubectl get sc -o wide
NAME                    PROVISIONER                     RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
nfs-164                 nfs-nfs-164                     Delete          WaitForFirstConsumer   false                  37d
rook-ceph-block         rook-ceph.rbd.csi.ceph.com      Delete          Immediate              true                   24h
rook-cephfs (default)   rook-ceph.cephfs.csi.ceph.com   Delete          Immediate              true                   23h
storage                 nfs-storage                     Delete          WaitForFirstConsumer   false                  39d

安装步骤

• 安装RabbitMQ集群操作员
• 使用RabbitMQ集群操作员部署RabbitMQ集群
• 查看RabbitMQ日志
• 访问RabbitMQ管理UI
• 设置应用访问RabbitMQ集群

安装

两种方式安装比较

第一种方式是采用插件的方式，详见地址：https://cloud.tencent.com/developer/article/1782766
需要先安装插件管理工具krew，然后才能使用插件的方式进行安装。在安装的过程中会从GitHub上下载东西，访问GitHub慢的不建议采用这种方式

第二种方式是通过yaml文件的方式进行安装，这里采用这种方式进行

使用yaml文件的方式进行安装cluster-operator

yaml文件下载地址：https://github.com/rabbitmq/cluster-operator/releases/download/v1.7.0/cluster-operator.yml

# kubectl create -f cluster-operator.yaml       
namespace/rabbitmq-system created
customresourcedefinition.apiextensions.k8s.io/rabbitmqclusters.rabbitmq.com created
serviceaccount/rabbitmq-cluster-operator created
role.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-role created
clusterrole.rbac.authorization.k8s.io/rabbitmq-cluster-operator-role created
rolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/rabbitmq-cluster-operator-rolebinding created
deployment.apps/rabbitmq-cluster-operator created

会创建一个新的名称空间：rabbitmq-system

# kubectl get all -n rabbitmq-system
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rabbitmq-cluster-operator-7c65454ff9-f52b2   1/1     Running   0          32m

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rabbitmq-cluster-operator   1/1     1            1           32m

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rabbitmq-cluster-operator-7c65454ff9   1         1         1       32m

通过上面的显示可以发现有个一脉相承的如下关系：

资源类型：deployment =》 replicaset =》 pod
具体名称 ：rabbitmq-cluster-operator =》rabbitmq-cluster-operator-7c65454ff9 =》rabbitmq-cluster-operator-7c65454ff9-td2pm

新的自定义资源rabbitmqclusters.rabbitmq.com。自定义资源允许我们定义用于创建RabbitMQ集群的API。以及一些rbac角色。这些是操作员创建、更新和删除RabbitMQ集群所必需的。

# kubectl get customresourcedefinitions.apiextensions.k8s.io 
NAME                                             CREATED AT
...
rabbitmqclusters.rabbitmq.com                    2021-06-04T02:21:42Z
...

使用cluster-operator创建RabbitMQ集群

简单的yaml文件下载地址：https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/hello-world
yaml文件名：rabbitmq.yaml
这是最简单的RabbitmqCluster定义。唯一显式指定的属性是集群的名称。其他一切都将根据集群运营商的默认值进行配置。

examples文件夹还有许多其他引用，比如用TLS、mTLS创建RabbitMQ集群，用生产默认值设置集群，添加社区插件等等。下载地址：https://github.com/rabbitmq/cluster-operator/tree/main/docs/examples/

# cat rabbitmq.yaml
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: my-rabbit-cluster
  namespace: rabbitmq-system

注意：默认没有写名称空间则会部署到defalut中，这里修改成rabbitmq-system

# kubectl create -f rabbitmq.yaml 
rabbitmqcluster.rabbitmq.com/my-rabbit-cluster created

这将在rabbitmq-system命名空间中创建一个名为my-rabbit-cluster的RabbitMQ集群。可以在创建RabbitMQ集群时看到它：

# kubectl get pod -n rabbitmq-system
NAME                                         READY   STATUS    RESTARTS   AGE
my-rabbit-cluster-server-0                   1/1     Running   0          80s

还可以看到创建的rabbitmqclusters.rabbitmq.com自定义资源的实例。

# kubectl get rabbitmqclusters.rabbitmq.com -n rabbitmq-system
NAME                AGE
my-rabbit-cluster   2m33s

查看RabbitMQ日志

# kubectl logs my-rabbit-cluster-server-0 -n rabbitmq-system
 Starting RabbitMQ 3.8.16 on Erlang 24.0.2
 Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
 Licensed under the MPL 2.0. Website: https://rabbitmq.com

  ##  ##      RabbitMQ 3.8.16
  ##  ##
  ##########  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
  ######  ##
  ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: <stdout>

  Config file(s): /etc/rabbitmq/rabbitmq.conf
                  /etc/rabbitmq/conf.d/10-operatorDefaults.conf
                  /etc/rabbitmq/conf.d/11-default_user.conf
                  /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf

  Starting broker...2021-06-04 03:05:59.646 [info] <0.273.0> 
 node           : rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.rabbitmq-system
 home dir       : /var/lib/rabbitmq
 config file(s) : /etc/rabbitmq/rabbitmq.conf
                : /etc/rabbitmq/conf.d/10-operatorDefaults.conf
                : /etc/rabbitmq/conf.d/11-default_user.conf
                : /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf
 cookie hash    : kH25nUaPr1ngafL6UipoTQ==
 log(s)         : <stdout>
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.rabbitmq-system

访问RabbitMQ管理UI

获取用户名和密码

username="$(kubectl get secret my-rabbit-cluster-default-user -n rabbitmq-system -o jsonpath='{.data.username}' | base64 --decode)"
echo "username: $username"  

password="$(kubectl get secret my-rabbit-cluster-default-user -n rabbitmq-system -o jsonpath='{.data.password}' | base64 --decode)"
echo "password: $password"

默认会创建一个类型是ClusterIP的svc

# kubectl get svc -n rabbitmq-system
NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                        AGE
my-rabbit-cluster         ClusterIP   10.3.255.186   <none>        5672/TCP,15672/TCP,15692/TCP   8m18s
my-rabbit-cluster-nodes   ClusterIP   None           <none>        4369/TCP,25672/TCP             8m18s

办法一是修改ClusterIP类型为NodePort，使用宿主机IP和nodeport端口来访问

# kubectl edit svc my-rabbit-cluster -n rabbitmq-system

注:这种方式一开始修改后是能够访问使用的，但是删除集群后重新创建并这样设置，就无法修改成NodePort了，很是奇怪。

办法二是使用kube-proxy进行代理来实现访问

# kubectl port-forward svc/my-rabbit-cluster 15672      
Forwarding from 127.0.0.1:15672 -> 15672
Forwarding from [::1]:15672 -> 15672

现在我们可以通过localhost:15672 在打开浏览器并查看管理UI。或者，可以运行curl命令来验证访问：

curl -u$username:$password localhost:15672/api/overview

[root@develop-master-1 ~]# username="$(kubectl get secret my-rabbit-cluster-default-user -n rabbitmq-system -o jsonpath='{.data.username}' | base64 --decode)" && echo "username: $username"
username: YNbrfwcmEqnuJD6BZthuWZP9bTJQuYmW
[root@develop-master-1 ~]# password="$(kubectl get secret my-rabbit-cluster-default-user -n rabbitmq-system -o jsonpath='{.data.password}' | base64 --decode)" && echo "password: $password"
password: 88DzY3WQ51Q2Q9FfWJKrTaQpLUgSfeWP
[root@develop-master-1 ~]# curl -uYNbrfwcmEqnuJD6BZthuWZP9bTJQuYmW:88DzY3WQ51Q2Q9FfWJKrTaQpLUgSfeWP localhost:15672/api/overview
{"management_version":"3.8.16","rates_mode":"basic","sample_retention_policies":{"global":[600,3600,28800,86400],"basic":[600,3600],"detailed":[600]},"exchange_types":[{"name":"direct","description":"AMQP direct exchange, as per the AMQP specification","enabled":true},{"name":"fanout","description":"AMQP fanout exchange, as per the AMQP specification","enabled":true},{"name":"headers","description":"AMQP headers exchange, as per the AMQP specification","enabled":true},{"name":"topic","description":"AMQP topic exchange, as per the AMQP specification","enabled":true}],"product_version":"3.8.16","product_name":"RabbitMQ","rabbitmq_version":"3.8.16","cluster_name":"my-rabbit-cluster","erlang_version":"24.0.2","erlang_full_version":"Erlang/OTP 24 [erts-12.0.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1]","disable_stats":false,"enable_queue_totals":false,"message_stats":{},"churn_rates":{"channel_closed":0,"channel_closed_details":{"rate":0.0},"channel_created":0,"channel_created_details":{"rate":0.0},"connection_closed":60,"connection_closed_details":{"rate":0.0},"connection_created":0,"connection_created_details":{"rate":0.0},"queue_created":0,"queue_created_details":{"rate":0.0},"queue_declared":0,"queue_declared_details":{"rate":0.0},"queue_deleted":0,"queue_deleted_details":{"rate":0.0}},"queue_totals":{},"object_totals":{"channels":0,"connections":0,"consumers":0,"exchanges":7,"queues":0},"statistics_db_event_queue":0,"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","listeners":[{"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","protocol":"amqp","ip_address":"::","port":5672,"socket_opts":{"backlog":128,"nodelay":true,"linger":[true,0],"exit_on_close":false}},{"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","protocol":"clustering","ip_address":"::","port":25672,"socket_opts":[]},{"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","protocol":"http","ip_address":"::","port":15672,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15672}},{"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","protocol":"http/prometheus","ip_address":"::","port":15692,"socket_opts":{"cowboy_opts":{"sendfile":false},"port":15692,"protocol":"http/prometheus"}}],"contexts":[{"ssl_opts":[],"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","description":"RabbitMQ Management","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15672"},{"ssl_opts":[],"node":"rabbit@my-rabbit-cluster-server-0.my-rabbit-cluster-nodes.default","description":"RabbitMQ Prometheus","path":"/","cowboy_opts":"[{sendfile,false}]","port":"15692","protocol":"'http/prometheus'"}]}

如下界面是通过使用kuboard代理出来的

设置应用访问RabbitMQ集群

# username="$(kubectl get secret my-rabbit-cluster-default-user -n rabbitmq-system -o jsonpath='{.data.username}' | base64 --decode)"
# password="$(kubectl get secret my-rabbit-cluster-default-user -n rabbitmq-system -o jsonpath='{.data.password}' | base64 --decode)"
# service="$(kubectl get service my-rabbit-cluster -n rabbitmq-system -o jsonpath='{.spec.clusterIP}')"
# kubectl run perf-test --image=pivotalrabbitmq/perf-test -- --uri amqp://$username:$password@$service

pod/perf-test created

# kubectl logs --follow perf-test
id: test-041921-512, time: 1.000s, sent: 4720 msg/s, received: 2109 msg/s, min/median/75th/95th/99th consumer latency: 3075/22253/33855/75681/89195 µs
id: test-041921-512, time: 2.000s, sent: 16538 msg/s, received: 13698 msg/s, min/median/75th/95th/99th consumer latency: 91432/157833/203565/344737/356844 µs
id: test-041921-512, time: 3.001s, sent: 18142 msg/s, received: 17832 msg/s, min/median/75th/95th/99th consumer latency: 268145/343634/378988/437575/450074 µs
id: test-041921-512, time: 4.001s, sent: 20991 msg/s, received: 17456 msg/s, min/median/75th/95th/99th consumer latency: 264207/345566/373946/407531/425332 µs
id: test-041921-512, time: 5.015s, sent: 24398 msg/s, received: 13231 msg/s, min/median/75th/95th/99th consumer latency: 358486/701769/755296/803202/832797 µs
id: test-041921-512, time: 6.017s, sent: 14616 msg/s, received: 12665 msg/s, min/median/75th/95th/99th consumer latency: 834423/1083012/1225606/1335011/1374533 µs
id: test-041921-512, time: 7.019s, sent: 16229 msg/s, received: 14603 msg/s, min/median/75th/95th/99th consumer latency: 1335783/1502060/1614123/1700792/1742414 µs
可以看出，perf-test每秒能够产生和消耗大约12000条消息。

使用Prometheus & Grafana监控rabbitmq集群

官方文档地址：https://www.rabbitmq.com/prometheus.html

问题

1.注意到上述rabbitmq集群中只有一个pod，my-rabbit-cluster-server-0
当扩充增加到2个甚至更多的时候，需要k8s节点资源是否足够。
我这边在扩充到2个时会报错：0/3 nodes are available: 3 Insufficient cpu.

原因是k8s节点资源资源不足。
创建rabbitmq集群时使用的是默认配置，默认配置中资源要求如下：

Limits:
  cpu:     2
  memory:  2Gi
Requests:
  cpu:      1
  memory:   2Gi

自动生成第二个pod时查看的详情如下：

Name:           my-rabbit-cluster-server-1
Namespace:      rabbitmq-system
Priority:       0
Node:           <none>
Labels:         app.kubernetes.io/component=rabbitmq
                app.kubernetes.io/name=my-rabbit-cluster
                app.kubernetes.io/part-of=rabbitmq
                controller-revision-hash=my-rabbit-cluster-server-b7d484587
                statefulset.kubernetes.io/pod-name=my-rabbit-cluster-server-1
Annotations:    prometheus.io/port: 15692
                prometheus.io/scrape: true
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  StatefulSet/my-rabbit-cluster-server
Init Containers:
  setup-container:
    Image:      rabbitmq:3.8.16-management
    Port:       <none>
    Host Port:  <none>
    Command:
      sh
      -c
      cp /tmp/erlang-cookie-secret/.erlang.cookie /var/lib/rabbitmq/.erlang.cookie && chown 999:999 /var/lib/rabbitmq/.erlang.cookie && chmod 600 /var/lib/rabbitmq/.erlang.cookie ; cp /tmp/rabbitmq-plugins/enabled_plugins /operator/enabled_plugins && chown 999:999 /operator/enabled_plugins ; chown 999:999 /var/lib/rabbitmq/mnesia/ ; echo '[default]' > /var/lib/rabbitmq/.rabbitmqadmin.conf && sed -e 's/default_user/username/' -e 's/default_pass/password/' /tmp/default_user.conf >> /var/lib/rabbitmq/.rabbitmqadmin.conf && chown 999:999 /var/lib/rabbitmq/.rabbitmqadmin.conf && chmod 600 /var/lib/rabbitmq/.rabbitmqadmin.conf
    Limits:
      cpu:     100m
      memory:  500Mi
    Requests:
      cpu:        100m
      memory:     500Mi
    Environment:  <none>
    Mounts:
      /operator from rabbitmq-plugins (rw)
      /tmp/default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
      /tmp/erlang-cookie-secret/ from erlang-cookie-secret (rw)
      /tmp/rabbitmq-plugins/ from plugins-conf (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from my-rabbit-cluster-server-token-vgvp4 (ro)
Containers:
  rabbitmq:
    Image:       rabbitmq:3.8.16-management
    Ports:       4369/TCP, 5672/TCP, 15672/TCP, 15692/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP, 0/TCP
    Limits:
      cpu:     2
      memory:  2Gi
    Requests:
      cpu:      1
      memory:   2Gi
    Readiness:  tcp-socket :amqp delay=10s timeout=5s period=10s #success=1 #failure=3
    Environment:
      MY_POD_NAME:                    my-rabbit-cluster-server-1 (v1:metadata.name)
      MY_POD_NAMESPACE:               default (v1:metadata.namespace)
      RABBITMQ_ENABLED_PLUGINS_FILE:  /operator/enabled_plugins
      K8S_SERVICE_NAME:               my-rabbit-cluster-nodes
      RABBITMQ_USE_LONGNAME:          true
      RABBITMQ_NODENAME:              rabbit@$(MY_POD_NAME).$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
      K8S_HOSTNAME_SUFFIX:            .$(K8S_SERVICE_NAME).$(MY_POD_NAMESPACE)
    Mounts:
      /etc/pod-info/ from pod-info (rw)
      /etc/rabbitmq/conf.d/10-operatorDefaults.conf from rabbitmq-confd (rw,path="operatorDefaults.conf")
      /etc/rabbitmq/conf.d/11-default_user.conf from rabbitmq-confd (rw,path="default_user.conf")
      /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf from rabbitmq-confd (rw,path="userDefinedConfiguration.conf")
      /operator from rabbitmq-plugins (rw)
      /var/lib/rabbitmq/ from rabbitmq-erlang-cookie (rw)
      /var/lib/rabbitmq/mnesia/ from persistence (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from my-rabbit-cluster-server-token-vgvp4 (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  persistence:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  persistence-my-rabbit-cluster-server-1
    ReadOnly:   false
  plugins-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      my-rabbit-cluster-plugins-conf
    Optional:  false
  rabbitmq-confd:
    Type:                Projected (a volume that contains injected data from multiple sources)
    SecretName:          my-rabbit-cluster-default-user
    SecretOptionalName:  <nil>
    ConfigMapName:       my-rabbit-cluster-server-conf
    ConfigMapOptional:   <nil>
  rabbitmq-erlang-cookie:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  erlang-cookie-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-rabbit-cluster-erlang-cookie
    Optional:    false
  rabbitmq-plugins:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  pod-info:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels['skipPreStopChecks'] -> skipPreStopChecks
  my-rabbit-cluster-server-token-vgvp4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  my-rabbit-cluster-server-token-vgvp4
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age                From               Message
  ----     ------            ----               ----               -------
  Warning  FailedScheduling  33s (x2 over 33s)  default-scheduler  0/3 nodes are available: 3 Insufficient cpu.