• Prometheus+Grafana监控Kubernetes


    涉及文件下载地址:链接:https://pan.baidu.com/s/18XHK7ex_J0rzTtfW-QA2eA 密码:0qn6
    文件中需要下载的镜像需要自己提前下载好,eg:prom/node-exporter:v0.16.0

    Prometheus官方网址,或者百度自己了解脑补:https://prometheus.io/

    官方文档说明链接
    Prometheus是一个开源的系统监控工具。
    根据配置的任务(job)以http/s周期性的收刮(scrape/pull)
    指定目标(target)上的指标(metric)。目标(target)
    可以以静态方式或者自动发现方式指定。
    Prometheus将收刮(scrape)的指标(metric)保存在本地或者远程存储上。

    Prometheus以pull方式来收集指标。对比push方式,
    pull可以集中配置、针对不同的视角搭建不同的监控系统


    Prometheus Server:核心组件,负责收刮和存储时序数据(time series data),并且提供查询接口;

    Jobs/Exporters:客户端,监控并采集指标,对外暴露HTTP服务(/metrics);
    目前已经有很多的软件原生就支持Prometjeus,提供/metrics,可以直接使用;
    对于像操作系统已经不提供/metrics的应用,可以使用现有的exporters
    或者开发自己的exporters来提供/metrics服务;

    Pushgateway:针对push系统设计,Short-lived jobs定时将指标push到Pushgateway,再由Prometheus Server从Pushgateway上pull;

    Alertmanager:报警组件,根据实现配置的规则(rule)进行响应,例如发送邮件;

    Web UI:Prometheus内置一个简单的Web控制台,可以查询指标,查看配置信息或者Service Discovery等,实际工作中,查看指标或者创建仪表盘通常使用Grafana,Prometheus作为Grafana的数据源;

    数据结构
    Prometheus按照时间序列存储指标,每一个指标都由Notation + Samples组成:
    Notation:通常有指标名称与一组label组成:
    <metric name>{<label name>=<label value>, ...}
    Samples:样品,通常包含一个64位的浮点值和一个毫秒级的时间戳
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    下面是在Mac上安装使用Prometheus+Grafana监控Kubernetes演示
    链接==-==Kubernetes Dashboard 安装,快速,简便运行Dashboard

    环境如下:Docker for Mac 或者 Edge 版本的内置的 Kubernetes 集群


    命令查看环境信息

    shenhl:~ user$ kubectl get nodes -o wide
    NAME STATUS ROLES AGE VERSION EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
    docker-for-desktop Ready master 20d v1.9.6 <none> Docker for Mac 4.9.87-linuxkit-aufs docker://18.5.0
    shenhl:~ user$ kubectl get pods --all-namespaces -o wide
    NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
    default kubernetes-bootcamp-5dbf48f7d4-dtn4f 1/1 Running 0 14d 10.1.0.92 docker-for-desktop
    default mysql-756vx 1/1 Running 0 11d 10.1.0.96 docker-for-desktop
    default myweb-494kv 1/1 Running 0 8d 10.1.0.102 docker-for-desktop
    default myweb-77r2b 1/1 Running 0 8d 10.1.0.101 docker-for-desktop
    default myweb-7p7h8 1/1 Running 0 8d 10.1.0.98 docker-for-desktop
    default myweb-jbfs9 1/1 Running 0 8d 10.1.0.100 docker-for-desktop
    default myweb-v6mnf 1/1 Running 0 8d 10.1.0.99 docker-for-desktop
    docker compose-5d4f4d67b6-ttmpk 1/1 Running 0 20d 10.1.0.94 docker-for-desktop
    docker compose-api-7bb7b5968f-bgvz8 1/1 Running 1 20d 192.168.65.3 docker-for-desktop
    kube-system etcd-docker-for-desktop 1/1 Running 7 20d 192.168.65.3 docker-for-desktop
    kube-system kube-apiserver-docker-for-desktop 1/1 Running 7 20d 192.168.65.3 docker-for-desktop
    kube-system kube-controller-manager-docker-for-desktop 1/1 Running 7 20d 192.168.65.3 docker-for-desktop
    kube-system kube-dns-6f4fd4bdf-bxkgg 3/3 Running 0 20d 10.1.0.95 docker-for-desktop
    kube-system kube-proxy-znhpr 1/1 Running 0 20d 192.168.65.3 docker-for-desktop
    kube-system kube-scheduler-docker-for-desktop 1/1 Running 7 20d 192.168.65.3 docker-for-desktop
    kube-system kubernetes-dashboard-5bd6f767c7-z7zqt 1/1 Running 1 15d 10.1.0.93 docker-for-desktop
    ns-monitor grafana-7bcf9c9f9-2g5bf 1/1 Running 0 8d 10.1.0.103 docker-for-desktop
    ns-monitor node-exporter-vpmm8 1/1 Running 0 7h 192.168.65.3 docker-for-desktop
    ns-monitor prometheus-65565d74b8-ctx7b 1/1 Running 0 8d 10.1.0.97 docker-for-desktop
    shenhl:~ user$
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    Kubernetes的Dashboard

    在kubernetest中创建namespace叫做ns-monitor
    创建文件名:namespace.yaml,内容如下:
    【文件在百度云盘可下载链接:https://pan.baidu.com/s/18XHK7ex_J0rzTtfW-QA2eA 密码:0qn6】

    apiVersion: v1
    kind: Namespace
    metadata:
    name: ns-monitor
    labels:
    name: ns-monitor
    1
    2
    3
    4
    5
    6
    在文件目录处,执行创建命令如下:


    kubectl apply -f namespace.yaml
    1
    在kubernetest中部署node-exporter,Node-exporter用于采集kubernetes集群中各个节点的物理指标,比如:Memory、CPU等。可以直接在每个物理节点是直接安装,这里我们使用DaemonSet部署到每个节点上,使用 hostNetwork: true 和 hostPID: true 使其获得Node的物理指标信息,配置tolerations使其在master节点也启动一个pod。
    创建文件名:node-exporter.yaml文件,内容如下:
    【文件在百度云盘可下载链接:https://pan.baidu.com/s/18XHK7ex_J0rzTtfW-QA2eA 密码:0qn6】

    kind: DaemonSet
    apiVersion: apps/v1beta2
    metadata:
    labels:
    app: node-exporter
    name: node-exporter
    namespace: ns-monitor
    spec:
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: node-exporter
    template:
    metadata:
    labels:
    app: node-exporter
    spec:
    containers:
    - name: node-exporter
    image: prom/node-exporter:v0.16.0
    ports:
    - containerPort: 9100
    protocol: TCP
    name: http
    hostNetwork: true
    hostPID: true
    tolerations:
    - effect: NoSchedule
    operator: Exists

    ---
    kind: Service
    apiVersion: v1
    metadata:
    labels:
    app: node-exporter
    name: node-exporter-service
    namespace: ns-monitor
    spec:
    ports:
    - name: http
    port: 9100
    nodePort: 31672
    protocol: TCP
    type: NodePort
    selector:
    app: node-exporter
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    在文件目录处,执行创建命令如下:

    kubectl apply -f node-exporter.yaml
    1
    查看创建是否成功:


    kubectl get pods -n ns-monitor -o wide
    1
    检验node-exporter是否都成功运行
    http://127.0.0.1:31672/metrics或者http://127.0.0.1:9100/metrics


    在kubernetest中部署Prometheus
    官方参考文档:/etc/prometheus/prometheus.yaml的配置
    创建文件名prometheus.yaml,内容如下:
    【文件在百度云盘可下载链接:https://pan.baidu.com/s/18XHK7ex_J0rzTtfW-QA2eA 密码:0qn6】
    ---
    apiVersion: rbac.authorization.k8s.io/v1beta1
    kind: ClusterRole
    metadata:
    name: prometheus
    rules:
    - apiGroups: [""] # "" indicates the core API group
    resources:
    - nodes
    - nodes/proxy
    - services
    - endpoints
    - pods
    verbs:
    - get
    - watch
    - list
    ......为节省篇幅,此处省略,请在百度云盘下载
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    执行创建命令

    kubectl apply -f prometheus.yaml
    kubectl get pods -n ns-monitor -o wide
    1
    2

    验证prometheus的正确性:http://127.0.0.1:31710/graph 或者 http://127.0.0.1:31710/service-discovery 或者 http://127.0.0.1:31710/targets


    在kubernetest中部署grafana
    创建grafana.yaml文件,内容如下:
    【文件在百度云盘可下载链接:https://pan.baidu.com/s/18XHK7ex_J0rzTtfW-QA2eA 密码:0qn6】
    apiVersion: v1
    kind: PersistentVolume
    metadata:
    name: "grafana-data-pv"
    labels:
    name: grafana-data-pv
    release: stable
    spec:
    capacity:
    storage: 5Gi
    accessModes:
    - ReadWriteOnce
    persistentVolumeReclaimPolicy: Recycle
    nfs:
    path: /nfs/grafana/data
    server: 192.168.65.3
    ---
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
    name: grafana-data-pvc
    namespace: ns-monitor
    spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
    storage: 5Gi
    selector:
    matchLabels:
    name: grafana-data-pv
    release: stable
    ---
    kind: Deployment
    apiVersion: apps/v1beta2
    metadata:
    labels:
    app: grafana
    name: grafana
    namespace: ns-monitor
    spec:
    replicas: 1
    revisionHistoryLimit: 10
    selector:
    matchLabels:
    app: grafana
    template:
    metadata:
    labels:
    app: grafana
    spec:
    containers:
    - name: grafana
    image: grafana/grafana:latest
    env:
    - name: GF_AUTH_BASIC_ENABLED
    value: "true"
    - name: GF_AUTH_ANONYMOUS_ENABLED
    value: "false"
    readinessProbe:
    httpGet:
    path: /login
    port: 3000
    volumeMounts:
    - mountPath: /var/lib/grafana
    name: grafana-data-volume
    ports:
    - containerPort: 3000
    protocol: TCP
    volumes:
    - name: grafana-data-volume
    persistentVolumeClaim:
    claimName: grafana-data-pvc
    ---
    kind: Service
    apiVersion: v1
    metadata:
    labels:
    app: grafana
    name: grafana-service
    namespace: ns-monitor
    spec:
    ports:
    - port: 3000
    targetPort: 3000
    selector:
    app: grafana
    type: NodePort
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    执行创建命令、并查看

    kubectl apply -f grafana.yaml
    kubectl get pods -n ns-monitor -o wide
    1
    2
    验证grafana是否成功运行:http://127.0.0.1:30591/login 默认用户名和密码:admin/admin


    配置grafana:把prometheus配置成数据源

    http://prometheus-service.ns-monitor:9090这个链接的来源:


    然后导入Dashboard

    再把 kubernetes的Dashboard的模板导入进来显示:直接把JSON格式内容复制进来就行
    【文件在百度云盘可下载链接:https://pan.baidu.com/s/18XHK7ex_J0rzTtfW-QA2eA 密码:0qn6】
    {
    "__inputs": [
    {
    "name": "DS_PROMETHEUS",
    "label": "prometheus",
    "description": "",
    "type": "datasource",
    "pluginId": "prometheus",
    "pluginName": "Prometheus"
    }
    ],
    "__requires": [
    {
    "type": "grafana",
    "id": "grafana",
    "name": "Grafana",
    "version": "5.0.4"
    },
    ......为节省篇幅,此处省略内容
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19


    Dashboard中的每一个Panel可以自行编辑、保存和回滚!
    如果instance下拉框显示有问题,点击右上方的设置(settings)~变量(Variables),
    修改$instance变量的Regex值,可以直接清空;
    配置数据源、导入Dashboard、安装插件等这些操作可以配置到grafana.yaml文件中,
    但是配置过程比较麻烦,这里先提供在界面上操作的说明,后期需要再处理。
    1
    2
    3
    4
    5
    6

    参考blog:https://blog.csdn.net/chenleiking/article/details/80009529
    参考官网:https://prometheus.io/
    ---------------------
    作者:common_util
    来源:CSDN
    原文:https://blog.csdn.net/shenhonglei1234/article/details/80503353
    版权声明:本文为博主原创文章,转载请附上博文链接!

  • 相关阅读:
    java.lang.IllegalArgumentException: When allowCredentials is true, allowedOrigins cannot contain the special value "*" since that cannot be set on the "Access-Control-Allow-Origin" response header.
    spring-session-data-redis依赖冲突问题
    centos7启动iptables时报Job for iptables.service failed because the control process exited with error cod
    图片上传后台服务报内存溢出 Out Of Memory Java heap space
    mysql 数据库密码忘记重置 进行远程连接
    打Jar包
    Type interface com.innovationV2.mapper.UserMapper is not known to the MapperRegistry
    关于java基础类型Integer String的clone()
    clion使用clang编译
    token & refresh token 机制总结
  • 原文地址:https://www.cnblogs.com/zhengchunyuan/p/11004635.html
Copyright © 2020-2023  润新知