Grafana Loki is a set of components that can be composed into a fully featured logging stack. Unlike other logging systems, Loki is built around the idea of only indexing labels for logs and leaving the original log message unindexed. This means that Loki is cheaper to operate and can be orders of magnitude more efficient.
以前提起日志系统，大家的第一反应就是ELK相关，并且好像也只能想起它，但是现在LOKI的出现让我们有了其他选择。毋容置疑，ELK是一款强大的日志系统，它基于es，提供了很多丰富又强大功能，但同时带来的也是资源的要求，以及高成本的维护，日志20G，倒排索引160G。对于中小型公司，对日志系统的要求并不多，开发更希望的是给我我要的项目日志，让我能grep就好；运维想要的不要占用太多资源，让我能轻松维护。LOKI满足了开发人员的需求，它的简单及低成本也让运维人员可以接受。
https://grafana.com/docs/loki/latest/overview/ 官方文档
使用helm安装是很轻松的方式，这里不做赘述 官网链接：https://grafana.com/docs/loki/latest/installation/helm/ helm repo add loki https://grafana.github.io/loki/charts helm repo update helm upgrade --install loki --namespace=kube-system loki/loki helm upgrade --install promtail --namespace=kube-system loki/promtail
在搜寻了不少网站后，发现并没有使用yaml文件直接安装的，个人也是调配了一份
loki：
注意存储，需要改成自己的存储或者使用本地空目录
apiVersion: v1 kind: ServiceAccount metadata: name: loki namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: loki namespace: kube-system rules: - apiGroups: - extensions resourceNames: - loki resources: - podsecuritypolicies verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: loki namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: loki subjects: - kind: ServiceAccount name: loki --- apiVersion: v1 kind: ConfigMap metadata: name: loki namespace: kube-system data: loki.yaml: | auth_enabled: false chunk_store_config: max_look_back_period: 0s compactor: shared_store: filesystem working_directory: /data/loki/boltdb-shipper-compactor ingester: chunk_block_size: 262144 chunk_idle_period: 3m chunk_retain_period: 1m lifecycler: ring: kvstore: store: inmemory replication_factor: 1 max_transfer_retries: 0 limits_config: enforce_metric_name: false reject_old_samples: true reject_old_samples_max_age: 168h schema_config: configs: - from: "2020-10-24" index: period: 24h prefix: index_ object_store: filesystem schema: v11 store: boltdb-shipper server: http_listen_port: 3100 storage_config: boltdb_shipper: active_index_directory: /data/loki/boltdb-shipper-active cache_location: /data/loki/boltdb-shipper-cache cache_ttl: 24h shared_store: filesystem filesystem: directory: /data/loki/chunks table_manager: retention_deletes_enabled: false retention_period: 0s --- apiVersion: v1 kind: Service metadata: name: loki namespace: kube-system labels: app: loki release: loki spec: ports: - name: http-metrics port: 3100 protocol: TCP targetPort: http-metrics selector: app: loki release: loki --- apiVersion: v1 kind: Service metadata: name: loki-headless namespace: kube-system labels: app: loki release: loki spec: clusterIP: None publishNotReadyAddresses: true ports: - name: http-metrics port: 3100 protocol: TCP targetPort: http-metrics selector: app: loki release: loki --- apiVersion: apps/v1 kind: StatefulSet metadata: name: loki namespace: kube-system labels: app: loki release: loki spec: replicas: 1 revisionHistoryLimit: 10 selector: matchLabels: app: loki release: loki serviceName: loki-headless template: metadata: labels: app: loki release: loki spec: containers: - args: - -config.file=/etc/loki/loki.yaml image: grafana/loki:2.0.0 imagePullPolicy: IfNotPresent name: loki ports: - containerPort: 3100 name: http-metrics protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 45 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 livenessProbe: failureThreshold: 3 httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 45 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 volumeMounts: - name: config mountPath: /etc/loki - name: storage mountPath: /data securityContext: fsGroup: 10001 runAsGroup: 10001 runAsNonRoot: true runAsUser: 10001 serviceAccount: loki serviceAccountName: loki volumes: - name: config configMap: defaultMode: 420 name: loki volumeClaimTemplates: - metadata: name: storage spec: accessModes: - ReadWriteOnce storageClassName: yizhuang-nfs resources: requests: storage: 100Gi
promtail：
apiVersion: v1 kind: ServiceAccount metadata: name: promtail namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: promtail namespace: kube-system rules: - apiGroups: - extensions resourceNames: - promtail resources: - podsecuritypolicies verbs: - use --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: promtail namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: promtail subjects: - kind: ServiceAccount name: promtail --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: promtail-clusterrole rules: - apiGroups: - "" resources: - nodes - nodes/proxy - services - endpoints - pods verbs: - get - watch - list --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: promtail-clusterrolebinding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: promtail-clusterrole subjects: - kind: ServiceAccount name: promtail namespace: kube-system --- apiVersion: v1 kind: ConfigMap metadata: name: promtail namespace: kube-system data: promtail.yaml: | server: http_listen_port: 80 log_level: "warn" client: url: http://loki.kube-system:3100/loki/api/v1/push batchwait: 5s batchsize: 5242880 backoff_config: max_period: 5m max_retries: 10 min_period: 1s external_labels: {} timeout: 10s positions: filename: /run/promtail/positions.yaml scrape_configs: - job_name: kubernetes-pods-name pipeline_stages: - docker: {} kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: - __meta_kubernetes_pod_label_name target_label: __service__ - source_labels: - __meta_kubernetes_pod_node_name target_label: __host__ - action: drop regex: '' source_labels: - __service__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace replacement: $1 separator: / source_labels: - __meta_kubernetes_namespace - __service__ target_label: job - action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace - action: replace source_labels: - __meta_kubernetes_pod_name target_label: pod - action: replace source_labels: - __meta_kubernetes_pod_container_name target_label: container - replacement: /var/log/pods/*$1/*.log separator: / source_labels: - __meta_kubernetes_pod_uid - __meta_kubernetes_pod_container_name target_label: __path__ - job_name: kubernetes-pods-app pipeline_stages: - docker: {} kubernetes_sd_configs: - role: pod relabel_configs: - action: drop regex: .+ source_labels: - __meta_kubernetes_pod_label_name - source_labels: - __meta_kubernetes_pod_label_app target_label: __service__ - source_labels: - __meta_kubernetes_pod_node_name target_label: __host__ - action: drop regex: '' source_labels: - __service__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace replacement: $1 separator: / source_labels: - __meta_kubernetes_namespace - __service__ target_label: job - action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace - action: replace source_labels: - __meta_kubernetes_pod_name target_label: pod - action: replace source_labels: - __meta_kubernetes_pod_container_name target_label: container - replacement: /var/log/pods/*$1/*.log separator: / source_labels: - __meta_kubernetes_pod_uid - __meta_kubernetes_pod_container_name target_label: __path__ - job_name: kubernetes-pods-direct-controllers pipeline_stages: - docker: {} kubernetes_sd_configs: - role: pod relabel_configs: - action: drop regex: .+ separator: '' source_labels: - __meta_kubernetes_pod_label_name - __meta_kubernetes_pod_label_app - action: drop regex: '[0-9a-z-.]+-[0-9a-f]{8,10}' source_labels: - __meta_kubernetes_pod_controller_name - source_labels: - __meta_kubernetes_pod_controller_name target_label: __service__ - source_labels: - __meta_kubernetes_pod_node_name target_label: __host__ - action: drop regex: '' source_labels: - __service__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace replacement: $1 separator: / source_labels: - __meta_kubernetes_namespace - __service__ target_label: job - action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace - action: replace source_labels: - __meta_kubernetes_pod_name target_label: pod - action: replace source_labels: - __meta_kubernetes_pod_container_name target_label: container - replacement: /var/log/pods/*$1/*.log separator: / source_labels: - __meta_kubernetes_pod_uid - __meta_kubernetes_pod_container_name target_label: __path__ - job_name: kubernetes-pods-indirect-controller pipeline_stages: - docker: {} kubernetes_sd_configs: - role: pod relabel_configs: - action: drop regex: .+ separator: '' source_labels: - __meta_kubernetes_pod_label_name - __meta_kubernetes_pod_label_app - action: keep regex: '[0-9a-z-.]+-[0-9a-f]{8,10}' source_labels: - __meta_kubernetes_pod_controller_name - action: replace regex: '([0-9a-z-.]+)-[0-9a-f]{8,10}' source_labels: - __meta_kubernetes_pod_controller_name target_label: __service__ - source_labels: - __meta_kubernetes_pod_node_name target_label: __host__ - action: drop regex: '' source_labels: - __service__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace replacement: $1 separator: / source_labels: - __meta_kubernetes_namespace - __service__ target_label: job - action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace - action: replace source_labels: - __meta_kubernetes_pod_name target_label: pod - action: replace source_labels: - __meta_kubernetes_pod_container_name target_label: container - replacement: /var/log/pods/*$1/*.log separator: / source_labels: - __meta_kubernetes_pod_uid - __meta_kubernetes_pod_container_name target_label: __path__ - job_name: kubernetes-pods-static pipeline_stages: - docker: {} kubernetes_sd_configs: - role: pod relabel_configs: - action: drop regex: '' source_labels: - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror - action: replace source_labels: - __meta_kubernetes_pod_label_component target_label: __service__ - source_labels: - __meta_kubernetes_pod_node_name target_label: __host__ - action: drop regex: '' source_labels: - __service__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - action: replace replacement: $1 separator: / source_labels: - __meta_kubernetes_namespace - __service__ target_label: job - action: replace source_labels: - __meta_kubernetes_namespace target_label: namespace - action: replace source_labels: - __meta_kubernetes_pod_name target_label: pod - action: replace source_labels: - __meta_kubernetes_pod_container_name target_label: container - replacement: /var/log/pods/*$1/*.log separator: / source_labels: - __meta_kubernetes_pod_annotation_kubernetes_io_config_mirror - __meta_kubernetes_pod_container_name target_label: __path__ target_config: sync_period: 10s --- apiVersion: apps/v1 kind: DaemonSet metadata: name: promtail namespace: kube-system labels: app: promtail release: promtail spec: selector: matchLabels: app: promtail release: promtail template: metadata: labels: app: promtail release: promtail spec: containers: - args: - -config.file=/etc/promtail/promtail.yaml env: - name: HOSTNAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: grafana/promtail:2.0.0 imagePullPolicy: IfNotPresent name: promtail ports: - containerPort: 80 name: http-metrics protocol: TCP readinessProbe: failureThreshold: 5 httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 livenessProbe: failureThreshold: 5 httpGet: path: /ready port: http-metrics scheme: HTTP initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 volumeMounts: - name: config mountPath: /etc/promtail - name: run mountPath: /run/promtail - name: docker mountPath: /var/lib/docker/containers readOnly: true - name: pods mountPath: /var/log/pods readOnly: true serviceAccount: promtail serviceAccountName: promtail tolerations: - effect: NoSchedule key: node-role.kubernetes.io/master operator: Exists volumes: - name: config configMap: defaultMode: 420 name: promtail - name: run hostPath: path: /run/promtail type: "" - name: docker hostPath: path: /var/lib/docker/containers type: "" - name: pods hostPath: path: /var/log/pods type: ""
在文件生效后，我们就可以去grafana里去配置对应的数据源
然后稍等一会，让数据进行一些传输，就可以去进行查询了
loki会将我们k8s里定义的所有label扫出来，当然在promtail的收集配置里，你可以进行更多的自定义标签，收集规则等等
我们进行日志的查询，一般先使用选择表达式，然后再使用过滤表达式
对于选择的内容，我们放在{}里，如{namespace="dev", project_name=~"cdy.+"}
=等于 !=不相等 =~正则表达式匹配 !~不匹配正则表达式
过滤的内容，我们只需要接在选择表达式后面即可，如{namespace="dev", project_name=~"cdy.+"} |~ "close"
|= 行包含字符串 != 行不包含字符串 |~ 行匹配正则表达式 !~ 行与正则表达式不匹配