普罗米修斯和Kubewatch与Kubernetes集群同步 [英] Syncing Prometheus and Kubewatch with Kubernetes Cluster

问题描述

我想将Kubernetes集群中发生的所有事件放在某个Python词典中，可能使用一些API从过去发生的事件中提取数据。我在网上发现，可以将Kube-Watch的所有数据存储在普罗米修斯上，然后访问它。我不知道如何设置它，也不知道如何在python中查看所有过去的Pod事件。访问过去事件的任何替代解决方案也是受欢迎的。谢谢！

推荐答案

我将描述一个不复杂并且我认为满足您所有要求的解决方案。 有Eventrouter这样的工具可以获取Kubernetes事件并将它们推送到用户指定的接收器。但是，正如您所提到的，您只需要Pods事件，所以我建议使用一种稍微不同的方法。

简而言之，您可以从Pod中运行kubectl get events --watch命令，并使用Loki这样的日志聚合系统收集该命令的输出。

下面，我将提供详细的分步说明。

1.从Pod内运行kubectl命令

若要仅显示Pod事件，您可以使用：

$ kubectl get events --watch --field-selector involvedObject.kind=Pod

我们希望从Pod中运行此命令。出于安全原因，我创建了一个单独的events-collectorServiceAccount，分配了view角色，我们的Pod将在此ServiceAccount下运行。
注意：我创建了Deployment而不是单个Pod。

$ cat all-in-one.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: events-collector
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: events-collector-binding
subjects:
  - kind: ServiceAccount
    name: events-collector
    namespace: default
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: events-collector
  name: events-collector
spec:
  selector:
    matchLabels:
      app: events-collector
  template:
    metadata:
      labels:
        app: events-collector
    spec:
      serviceAccountName: events-collector
      containers:
      - image: bitnami/kubectl
        name: test
        command: ["kubectl"]
        args: ["get","events", "--watch", "--field-selector", "involvedObject.kind=Pod"]

应用上述清单后，event-collector已创建并按预期收集Pod事件：

$ kubectl apply -f all-in-one.yml
serviceaccount/events-collector created
clusterrolebinding.rbac.authorization.k8s.io/events-collector-binding created
deployment.apps/events-collector created

$ kubectl get deploy,pod | grep events-collector
deployment.apps/events-collector           1/1     1            1           14s
pod/events-collector-d98d6c5c-xrltj            1/1     Running   0          14s

$ kubectl logs -f events-collector-d98d6c5c-xrltj
LAST SEEN   TYPE     REASON      OBJECT                                         MESSAGE
77s         Normal   Scheduled   pod/app-1-5d9ccdb595-m9d5n                     Successfully assigned default/app-1-5d9ccdb595-m9d5n to gke-cluster-2-default-pool-8505743b-brmx
76s         Normal   Pulling     pod/app-1-5d9ccdb595-m9d5n                     Pulling image "nginx"
71s         Normal   Pulled      pod/app-1-5d9ccdb595-m9d5n                     Successfully pulled image "nginx" in 4.727842954s
70s         Normal   Created     pod/app-1-5d9ccdb595-m9d5n                     Created container nginx
70s         Normal   Started     pod/app-1-5d9ccdb595-m9d5n                     Started container nginx
73s         Normal   Scheduled   pod/app-2-7747dcb588-h8j4q                     Successfully assigned default/app-2-7747dcb588-h8j4q to gke-cluster-2-default-pool-8505743b-p7qt
72s         Normal   Pulling     pod/app-2-7747dcb588-h8j4q                     Pulling image "nginx"
67s         Normal   Pulled      pod/app-2-7747dcb588-h8j4q                     Successfully pulled image "nginx" in 4.476795932s
66s         Normal   Created     pod/app-2-7747dcb588-h8j4q                     Created container nginx
66s         Normal   Started     pod/app-2-7747dcb588-h8j4q                     Started container nginx

2.安装Loki

您可以安装Loki来存储日志和处理查询。洛基就像普罗米修斯，但对于原木：)。安装Loki的最简单方法是使用grafana/loki-stack头盔图：

$ helm repo add grafana https://grafana.github.io/helm-charts
"grafana" has been added to your repositories

$ helm repo update
...
Update Complete. ⎈Happy Helming!⎈

$ helm upgrade --install loki grafana/loki-stack


$ kubectl get pods | grep loki
loki-0                            1/1     Running   0          76s
loki-promtail-hm8kn               1/1     Running   0          76s
loki-promtail-nkv4p               1/1     Running   0          76s
loki-promtail-qfrcr               1/1     Running   0          76s

3.使用LogCLI查询Loki

您可以使用LogCLI工具对Loki服务器运行LogQL查询。有关安装和使用此工具的详细信息，请参阅LogCLI documentation。我将演示如何在Linux上安装它：

$ wget https://github.com/grafana/loki/releases/download/v2.2.1/logcli-linux-amd64.zip

$ unzip logcli-linux-amd64.zip
Archive:  logcli-linux-amd64.zip
  inflating: logcli-linux-amd64
  
$ mv logcli-linux-amd64 logcli

$ sudo cp logcli /bin/

$ whereis logcli
logcli: /bin/logcli

要从Kubernetes集群外部查询Loki服务器，您可能需要使用Ingress资源公开它：

$ cat ingress.yml
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /
  name: loki-ingress
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: loki
              servicePort: 3100
            path: /
            
 
$ kubectl apply -f ingress.yml
ingress.networking.k8s.io/loki-ingress created

$ kubectl get ing
NAME           CLASS    HOSTS   ADDRESS         PORTS   AGE
loki-ingress   <none>   *      <PUBLIC_IP>      80      19s

最后，我创建了一个简单的python脚本，我们可以使用它来查询Loki服务器： 注意：我们需要设置LOKI_ADDR环境变量，如documentation中所述。您需要将<PUBLIC_IP>替换为您的入口IP。

$ cat query_loki.py
#!/usr/bin/env python3

import os

os.environ['LOKI_ADDR'] = "http://<PUBLIC_IP>"

os.system("logcli query '{app="events-collector"}'")


$ ./query_loki.py
...
2021-07-02T10:33:01Z {} 2021-07-02T10:33:01.626763464Z stdout F 0s          Normal    Pulling       pod/backend-app-5d99cf4b-c9km4                               Pulling image "nginx"
2021-07-02T10:33:00Z {} 2021-07-02T10:33:00.836755152Z stdout F 0s          Normal    Scheduled     pod/backend-app-5d99cf4b-c9km4                               Successfully assigned default/backend-app-5d99cf4b-c9km4 to gke-cluster-1-default-pool-328bd2b1-288w
2021-07-02T10:33:00Z {} 2021-07-02T10:33:00.649954267Z stdout F 0s          Normal    Started       pod/web-app-6fcf9bb7b8-jbrr9                                 Started container nginx2021-07-02T10:33:00Z {} 2021-07-02T10:33:00.54819851Z stdout F 0s          Normal    Created       pod/web-app-6fcf9bb7b8-jbrr9                                 Created container nginx
2021-07-02T10:32:59Z {} 2021-07-02T10:32:59.414571562Z stdout F 0s          Normal    Pulled        pod/web-app-6fcf9bb7b8-jbrr9                                 Successfully pulled image "nginx" in 4.228468876s
...

这篇关于普罗米修斯和Kubewatch与Kubernetes集群同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！