prometheus-operator

1. Prometheus Operator介紹

介紹文章：http://t.zoukankan.com/twobrother-p-11164391.html

2016年年末，CoreOs引入了Operator 模式，並行布了Prometheus Operator 作為Operator模式的工作範例。Prometheus Operator自動建立和管理Prometheus監控範例。

Prometheus Operator的任務是使得在Kubernetes執行Prometheus僅可能容易，同時保留可設定性以及使Kubernetes設定原生。

Prometheus Operator使我們的生活更容易——部署和維護。

2. 它如何工作

為了理解這個問題，我們首先需要了解Prometheus Operator得工作原理。

Prometheus Operator架構圖.

我們成功部署 Prometheus Operator後可以看到一個新的CRDs(Custom Resource Defination):

Prometheus，定義一個期望的Prometheus deployment。
ServiceMonitor，宣告式指定應該如何監控服務組；Operator根據定義自動建立Prometheusscrape設定。
Alertmanager，定義期望的Alertmanager deployment。

當服務新版本更新時，將會常見一個新Pod。Prometheus監控k8s API，因此當它檢測到這種變化時，它將為這個新服務(pod)建立一組新的設定。

3. ServiceMonitor

Prometheus Operator使用一個CRD，叫做 ServiceMonitor 將設定抽象到目標。
下面是個ServiceMonitor的範例：

apiVersion: monitoring.coreos.com/v1alpha1
kind: ServiceMonitor
metadata:
  name: frontend
  labels:
    tier: frontend
spec:
  selector:
    matchLabels:
      tier: frontend
  endpoints:
  - port: web            # 指定exporter埠，這裡指定的是endpoint的名稱
    interval: 10s        # 重新整理間隔時間

這僅僅是定義一組服務應該如何被監控。現在我們需要定義一個包含了該ServiceMonitor的Prometheus範例到其設定：

apiVersion: monitoring.coreos.com/v1alpha1
kind: Prometheus
metadata:
  name: prometheus-frontend
  labels:
    prometheus: frontend
spec:
  version: v1.3.0
  #定義應包括標籤為「tier=frontend」的所有ServiceMonitor 到伺服器的設定中
  serviceMonitors:
  - selector:
      matchLabels:
        tier: frontend

現在Prometheus將會監控每個帶有tier: frontend label的服務。

4. helm安裝

先決條件：

部署了Helm

準備好動手操作：

 helm repo add coreos https://s3-eu-west-1.amazonaws.com/coreos-charts/stable/
 helm install coreos/prometheus-operator --name prometheus-operator --namespace monitoring

到目前為止，我們已經在我們的叢集中安裝了Prometheus Operator的TPR。
現在我們來部署Prometheus，Alertmanager和Grafana。

TIP: 當我使用一個龐大的Helm Charts時，我更傾向於建立一個獨立的value.yaml檔案將包含我所有自定義的變更。這麼做使我和同事為後期的變化和修改更容易。

helm install coreos/kube-prometheus --name kube-prometheus   \
       -f my_changes/prometheus.yaml                           \
       -f my_changes/grafana.yaml                              \
       -f my_changes/alertmanager.yaml

檢查一切是否執行正常

 kubectl -n monitoring get po
NAME                                                   READY     STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-0                         2/2       Running   0          1h
kube-prometheus-exporter-kube-state-68dbb4f7c9-tr6rp   2/2       Running   0          1h
kube-prometheus-exporter-node-bqcj4                    1/1       Running   0          1h
kube-prometheus-exporter-node-jmcq2                    1/1       Running   0          1h
kube-prometheus-exporter-node-qnzsn                    1/1       Running   0          1h
kube-prometheus-exporter-node-v4wn8                    1/1       Running   0          1h
kube-prometheus-exporter-node-x5226                    1/1       Running   0          1h
kube-prometheus-exporter-node-z996c                    1/1       Running   0          1h
kube-prometheus-grafana-54c96ffc77-tjl6g               2/2       Running   0          1h
prometheus-kube-prometheus-0                           2/2       Running   0          1h
prometheus-operator-1591343780-5vb5q                   1/1       Running   0          1h

存取下Prometheus UI看一下Targets頁面：

 kubectl -n monitoring port-forward prometheus-kube-prometheus-0 9090
Forwarding from 127.0.0.1:9090 -> 9090

瀏覽器展示如下：

5. yaml檔案安裝

此安裝方法本人親測有效，用到的yaml檔案都打包好了。解壓之後直接kubectl apply即可用。會自動監控當前叢集的所有node節點和pod。只需更改yaml檔案中需要用到的映象。我這裡都推到了公司公網harbor倉庫。部分映象已經打成tar包。直接docker load -i即可用。
kube-state.tar.gz
webhook-dingtalk.tar.gz
prometheus-adapter.tar.gz

5.1 安裝

#軟體包整合了node Exporter alertmanager grafana prometheus ingress 所有服務的設定，只需解壓到K8S master中。
[root@lecode-k8s-master monitor]# ll
total 1820
-rw-r--r-- 1 root root     875 Mar 11  2022 alertmanager-alertmanager.yaml
-rw-r--r-- 1 root root     515 Mar 11  2022 alertmanager-podDisruptionBudget.yaml
-rw-r--r-- 1 root root    4337 Mar 11  2022 alertmanager-prometheusRule.yaml
-rw-r--r-- 1 root root    1483 Mar 14  2022 alertmanager-secret.yaml
-rw-r--r-- 1 root root     301 Mar 11  2022 alertmanager-serviceAccount.yaml
-rw-r--r-- 1 root root     540 Mar 11  2022 alertmanager-serviceMonitor.yaml
-rw-r--r-- 1 root root     614 Mar 11  2022 alertmanager-service.yaml
drwxr-x--- 2 root root    4096 Oct 25 13:49 backsvc #這裡是grafana的service設定。nodeport模式。用於外部存取。選擇使用
-rw-r--r-- 1 root root     278 Mar 11  2022 blackbox-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     287 Mar 11  2022 blackbox-exporter-clusterRole.yaml
-rw-r--r-- 1 root root    1392 Mar 11  2022 blackbox-exporter-configuration.yaml
-rw-r--r-- 1 root root    3081 Mar 11  2022 blackbox-exporter-deployment.yaml
-rw-r--r-- 1 root root      96 Mar 11  2022 blackbox-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root     680 Mar 11  2022 blackbox-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root     540 Mar 11  2022 blackbox-exporter-service.yaml
-rw-r--r-- 1 root root    2521 Oct 25 13:36 dingtalk-dep.yaml
-rw-r--r-- 1 root root     721 Mar 11  2022 grafana-dashboardDatasources.yaml
-rw-r--r-- 1 root root 1448347 Mar 11  2022 grafana-dashboardDefinitions.yaml
-rw-r--r-- 1 root root     625 Mar 11  2022 grafana-dashboardSources.yaml
-rw-r--r-- 1 root root    8098 Mar 11  2022 grafana-deployment.yaml
-rw-r--r-- 1 root root      86 Mar 11  2022 grafana-serviceAccount.yaml
-rw-r--r-- 1 root root     398 Mar 11  2022 grafana-serviceMonitor.yaml
-rw-r--r-- 1 root root     468 Mar 30  2022 grafana-service.yaml
drwxr-xr-x 2 root root    4096 Oct 25 13:32 ingress #這裡ingress資源也是可以直接用，可以把Prometheus和grafana服務暴露在外部。
-rw-r--r-- 1 root root    2639 Mar 14  2022 kube-prometheus-prometheusRule.yaml
-rw-r--r-- 1 root root    3380 Mar 14  2022 kube-prometheus-prometheusRule.yamlbak
-rw-r--r-- 1 root root   63531 Mar 11  2022 kubernetes-prometheusRule.yaml
-rw-r--r-- 1 root root    6912 Mar 11  2022 kubernetes-serviceMonitorApiserver.yaml
-rw-r--r-- 1 root root     425 Mar 11  2022 kubernetes-serviceMonitorCoreDNS.yaml
-rw-r--r-- 1 root root    6431 Mar 11  2022 kubernetes-serviceMonitorKubeControllerManager.yaml
-rw-r--r-- 1 root root    7629 Mar 11  2022 kubernetes-serviceMonitorKubelet.yaml
-rw-r--r-- 1 root root     530 Mar 11  2022 kubernetes-serviceMonitorKubeScheduler.yaml
-rw-r--r-- 1 root root     464 Mar 11  2022 kube-state-metrics-clusterRoleBinding.yaml
-rw-r--r-- 1 root root    1712 Mar 11  2022 kube-state-metrics-clusterRole.yaml
-rw-r--r-- 1 root root    2934 Oct 25 13:40 kube-state-metrics-deployment.yaml
-rw-r--r-- 1 root root    3082 Mar 11  2022 kube-state-metrics-prometheusRule.yaml
-rw-r--r-- 1 root root     280 Mar 11  2022 kube-state-metrics-serviceAccount.yaml
-rw-r--r-- 1 root root    1011 Mar 11  2022 kube-state-metrics-serviceMonitor.yaml
-rw-r--r-- 1 root root     580 Mar 11  2022 kube-state-metrics-service.yaml
-rw-r--r-- 1 root root     444 Mar 11  2022 node-exporter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     461 Mar 11  2022 node-exporter-clusterRole.yaml
-rw-r--r-- 1 root root    3047 Mar 11  2022 node-exporter-daemonset.yaml
-rw-r--r-- 1 root root   14356 Apr 11  2022 node-exporter-prometheusRule.yaml
-rw-r--r-- 1 root root     270 Mar 11  2022 node-exporter-serviceAccount.yaml
-rw-r--r-- 1 root root     850 Mar 11  2022 node-exporter-serviceMonitor.yaml
-rw-r--r-- 1 root root     492 Mar 11  2022 node-exporter-service.yaml
-rw-r--r-- 1 root root     482 Mar 11  2022 prometheus-adapter-apiService.yaml
-rw-r--r-- 1 root root     576 Mar 11  2022 prometheus-adapter-clusterRoleAggregatedMetricsReader.yaml
-rw-r--r-- 1 root root     494 Mar 11  2022 prometheus-adapter-clusterRoleBindingDelegator.yaml
-rw-r--r-- 1 root root     471 Mar 11  2022 prometheus-adapter-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     378 Mar 11  2022 prometheus-adapter-clusterRoleServerResources.yaml
-rw-r--r-- 1 root root     409 Mar 11  2022 prometheus-adapter-clusterRole.yaml
-rw-r--r-- 1 root root    2204 Mar 11  2022 prometheus-adapter-configMap.yaml
-rw-r--r-- 1 root root    2530 Oct 25 13:39 prometheus-adapter-deployment.yaml
-rw-r--r-- 1 root root     506 Mar 11  2022 prometheus-adapter-podDisruptionBudget.yaml
-rw-r--r-- 1 root root     515 Mar 11  2022 prometheus-adapter-roleBindingAuthReader.yaml
-rw-r--r-- 1 root root     287 Mar 11  2022 prometheus-adapter-serviceAccount.yaml
-rw-r--r-- 1 root root     677 Mar 11  2022 prometheus-adapter-serviceMonitor.yaml
-rw-r--r-- 1 root root     501 Mar 11  2022 prometheus-adapter-service.yaml
-rw-r--r-- 1 root root     447 Mar 11  2022 prometheus-clusterRoleBinding.yaml
-rw-r--r-- 1 root root     394 Mar 11  2022 prometheus-clusterRole.yaml
-rw-r--r-- 1 root root    5000 Mar 11  2022 prometheus-operator-prometheusRule.yaml
-rw-r--r-- 1 root root     715 Mar 11  2022 prometheus-operator-serviceMonitor.yaml
-rw-r--r-- 1 root root     499 Mar 11  2022 prometheus-podDisruptionBudget.yaml
-rw-r--r-- 1 root root   14021 Mar 11  2022 prometheus-prometheusRule.yaml
-rw-r--r-- 1 root root    1184 Mar 11  2022 prometheus-prometheus.yaml
-rw-r--r-- 1 root root     471 Mar 11  2022 prometheus-roleBindingConfig.yaml
-rw-r--r-- 1 root root    1547 Mar 11  2022 prometheus-roleBindingSpecificNamespaces.yaml
-rw-r--r-- 1 root root     366 Mar 11  2022 prometheus-roleConfig.yaml
-rw-r--r-- 1 root root    2047 Mar 11  2022 prometheus-roleSpecificNamespaces.yaml
-rw-r--r-- 1 root root     271 Mar 11  2022 prometheus-serviceAccount.yaml
-rw-r--r-- 1 root root     531 Mar 11  2022 prometheus-serviceMonitor.yaml
-rw-r--r-- 1 root root     558 Mar 11  2022 prometheus-service.yaml
drw-r--r-- 2 root root    4096 Oct 24 12:31 setup


#先apply setup目錄中的yaml檔案。然後apply一級目錄下的yaml檔案。backsvc中的grafana的service資源清單。根據情況調整為nodeport或ClusterIP。K8S叢集會自動在每臺K8S節點部署node-exporter並收集資料。登入grafana後初始賬號密碼為admin admin。新增dashboard即可監控K8S叢集
[root@lecode-k8s-master monitor]# cd setup/
[root@lecode-k8s-master setup]#   kubectl apply -f .
[root@lecode-k8s-master setup]# cd ..
[root@lecode-k8s-master monitor]# kubectl apply -f .
[root@lecode-k8s-master monitor]# kubectl get po -n monitoring 
NAME                                   READY   STATUS    RESTARTS   AGE
alertmanager-main-0                    2/2     Running   0          74m
alertmanager-main-1                    2/2     Running   0          74m
alertmanager-main-2                    2/2     Running   0          74m
blackbox-exporter-6798fb5bb4-d9m7m     3/3     Running   0          74m
grafana-64668d8465-x7x9z               1/1     Running   0          74m
kube-state-metrics-569d89897b-hlqxj    3/3     Running   0          57m
node-exporter-6vqxg                    2/2     Running   0          74m
node-exporter-7dxh6                    2/2     Running   0          74m
node-exporter-9j5xk                    2/2     Running   0          74m
node-exporter-ftrmn                    2/2     Running   0          74m
node-exporter-qszkn                    2/2     Running   0          74m
node-exporter-wjkgj                    2/2     Running   0          74m
prometheus-adapter-5dd78c75c6-h2jf7    1/1     Running   0          58m
prometheus-adapter-5dd78c75c6-qpwzv    1/1     Running   0          58m
prometheus-k8s-0                       2/2     Running   0          74m
prometheus-k8s-1                       2/2     Running   0          74m
prometheus-operator-75d9b475d9-mmzgs   2/2     Running   0          80m
webhook-dingtalk-6ffc94b49-z9z6l       1/1     Running   0          61m
[root@lecode-k8s-master backsvc]# kubectl get svc -n monitoring 
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
alertmanager-main       NodePort    10.98.35.93     <none>        9093:30093/TCP               72m
alertmanager-operated   ClusterIP   None            <none>        9093/TCP,9094/TCP,9094/UDP   72m
blackbox-exporter       ClusterIP   10.109.10.110   <none>        9115/TCP,19115/TCP           72m
grafana                 NodePort    10.110.48.214   <none>        3000:30300/TCP               72m
kube-state-metrics      ClusterIP   None            <none>        8443/TCP,9443/TCP            72m
node-exporter           ClusterIP   None            <none>        9100/TCP                     72m
prometheus-adapter      ClusterIP   10.97.23.176    <none>        443/TCP                      72m
prometheus-k8s          ClusterIP   10.100.92.254   <none>        9090/TCP                     72m
prometheus-operated     ClusterIP   None            <none>        9090/TCP                     72m
prometheus-operator     ClusterIP   None            <none>        8443/TCP                     78m
webhook-dingtalk        ClusterIP   10.100.131.63   <none>        80/TCP                       72m

5.2 存取服務

暴露服務三種方法：用service資源的nodeport模式，或者用k8s的ingress暴露服務或者本地nginx代理。原生的nginx代理模式

這裡我grafana用的是nodeport模式。Prometheus用的是nginx代理。附上nginx組態檔
[root@lecode-k8s-master setup]# cat /usr/local/nginx/conf/4-layer-conf.d/lecode-prometheus-operator.conf 
#代理prometheus內建Dashboard UI
upstream prometheus-dashboard {
    server 10.100.92.254:9090; #這裡ip為prometheus-k8s svc資源的ip
}

server {
    listen  9090;
    proxy_pass prometheus-dashboard;
}

#代理grafana
upstream grafana {
    server 10.1.82.89:3000; #這裡ip為grafana svc資源的ip
}

server {
    listen  3000;
    proxy_pass grafana;
}

存取Prometheus targets

5.3 接入grafana

存取grafana（預設密碼是admin admin）

去grafana官網下載對應dashboard 地址：https://grafana.com/grafana/dashboards/

6. 監控叢集外的服務

6.1 exporter安裝

在對應服務的本地安裝對應的exporter用於收集資料（這裡以mysql為例）

#下載對應服務的exporter  
#外掛下載地址：https://www.modb.pro/db/216588
#外掛下載地址：https://prometheus.io/download/
#下載完成後解壓mysqld_exporter-0.13.0.linux-amd64.tar.gz

#設定mysql-exporter
在root路徑下，建立.my.cnf檔案。內容如下:
[root@lecode-test-001 ~]# cat /root/.my.cnf 
[client]
user=mysql_monitor
password=Mysql@123



#建立mysql 使用者並授權

CREATE USER ‘mysql_monitor’@‘localhost’ IDENTIFIED BY ‘Mysql@123’ WITH MAX_USER_CONNECTIONS 3;
GRANT PROCESS, REPLICATION CLIENT, SELECT ON . TO ‘mysql_monitor’@‘localhost’;
FLUSH PRIVILEGES;
EXIT

#啟動mysqld_exporter
[root@lecode-test-001 mysql-exporter]# nohup mysqld_exporter &
#找到對應的埠
[root@lecode-test-001 mysql-exporter]# tail -f nohup.out 
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:277 msg="Starting msqyld_exporter" version="(version=0.13.0, branch=HEAD, revision=ad2847c7fa67b9debafccd5a08bacb12fc9031f1)"
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:278 msg="Build context" (gogo1.16.4,userroot@e2043849cb1f,date20210531-07:30:16)=(MISSING)
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=global_variables
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=slave_status
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmp
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.innodb_cmpmem
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:293 msg="Scraper enabled" scraper=info_schema.query_response_time
level=info ts=2022-10-25T09:26:54.464Z caller=mysqld_exporter.go:303 msg="Listening on address" address=:9104 #這是exporter的埠
level=info ts=2022-10-25T09:26:54.464Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
#檢查埠
[root@lecode-test-001 mysql-exporter]# ss -lntup |grep 9104
tcp    LISTEN     0      128      :::9104                 :::*                   users:(("mysqld_exporter",pid=26115,fd=3))

6.2 K8S設定

建立endpoint資源關聯對應服務主機的exporter埠。繫結service資源，通過ServiceMonitor資源新增Prometheus targets，

1）官方格式

 kubectl -n monitoring get prometheus kube-prometheus -o yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    app: prometheus
    chart: prometheus-0.0.14
    heritage: Tiller
    prometheus: kube-prometheus
    release: kube-prometheus
  name: kube-prometheus
  namespace: monitoring
spec:
  ...
  baseImage: quay.io/prometheus/prometheus
  serviceMonitorSelector:
    matchLabels:
      prometheus: kube-prometheus 

#接下來就是按照格式建立對應的ServiceMonitor資源

通過ep資源把外部服務關聯到K8S內部，繫結對應的svc資源。在由serviceMonitor繫結對應的svc資源把資料提交給Prometheus，serviceMonitor通過標籤選擇器關聯service，而service只需要通過埠關聯ep，這裡的標籤和埠一定要注意一致.

2) 建立資源

[root@lecode-k8s-master monitor]# cat mysql.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: mysql-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17  # ip為安裝應用伺服器的ip
  ports:
  - name: mysql
    port: 9104 #export的埠
    protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: mysql
    app.kubernetes.io/name: mysql
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: mysql-test
  namespace: monitoring
spec:
  clusterIP: None
  clusterIPs:
  - None
  ports:
  - name: mysql
    port: 9104
    protocol: TCP
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: mysql
    app.kubernetes.io/name: mysql
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: mysql-test
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: mysql
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: mysql
      app.kubernetes.io/name: mysql
      app.kubernetes.io/part-of: kube-prometheus


#建立        
[root@lecode-k8s-master monitor]# kubectl apply -f mysql.yaml 
endpoints/mysql-test created
service/mysql-test created
servicemonitor.monitoring.coreos.com/mysql-test created
        
#檢查        
[root@lecode-k8s-master monitor]# kubectl get -f  mysql.yaml 
NAME                   ENDPOINTS           AGE
endpoints/mysql-test   192.168.1.17:9104   10m

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/mysql-test   ClusterIP   None         <none>        9104/TCP   10m

NAME                                              AGE
servicemonitor.monitoring.coreos.com/mysql-test   10m

6.3 接入grafana

7 監控叢集外的服務-redis

7.1 exporter安裝

#部署redis-exporter
#外掛下載地址：https://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf redis_exporter-v1.3.2.linux-amd64.tar.gz 
[root@lecode-test-001 ~]# ll
drwxr-xr-x  2 root root     4096 Nov  6  2019 redis_exporter-v1.3.2.linux-amd64
-rw-r--r--  1 root root  3376155 Oct 27 10:26 redis_exporter-v1.3.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv redis_exporter-v1.3.2.linux-amd64 redis_exporter
[root@lecode-test-001 ~]# cd redis_exporter
[root@lecode-test-001 redis_exporter]# ll
total 8488
-rw-r--r-- 1 root root    1063 Nov  6  2019 LICENSE
-rw-r--r-- 1 root root   10284 Nov  6  2019 README.md
-rwxr-xr-x 1 root root 8675328 Nov  6  2019 redis_exporter
[root@lecode-test-001 redis_exporter]# nohup ./redis_exporter -redis.addr  192.168.1.17:6379  -redis.password  'Redislecodetest@shuli123' & 
[1] 4564
[root@lecode-test-001 redis_exporter]# nohup: ignoring input and appending output to ânohup.outâ

[root@lecode-test-001 redis_exporter]# tail -f nohup.out 
time="2022-10-27T10:26:48+08:00" level=info msg="Redis Metrics Exporter v1.3.2    build date: 2019-11-06-02:25:20    sha1: 175a69f33e8267e0a0ba47caab488db5e83a592e    Go: go1.13.4    GOOS: linux    GOARCH: amd64"
time="2022-10-27T10:26:48+08:00" level=info msg="Providing metrics at :9121/metrics"
#埠為9121

7.2 K8S設定

#建立redis-serviceMonitor資源
[root@lecode-k8s-master monitor]# cat redis.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: redis-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17 # ip為安裝應用伺服器的ip
  ports:
  - name: redis
    port: 9121 #exporter埠
    protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: redis-test
  namespace: monitoring
spec:
  clusterIP: None
  clusterIPs:
  - None
  ports:
  - name: redis
    port: 9121
    protocol: TCP
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: redis-test
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: redis
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: redis
      app.kubernetes.io/name: redis
      app.kubernetes.io/part-of: kube-prometheus


#建立資源
[root@lecode-k8s-master monitor]# kubectl apply -f redis.yaml 
endpoints/redis-test created
service/redis-test created
servicemonitor.monitoring.coreos.com/redis-test created
[root@lecode-k8s-master monitor]#  kubectl get ep,svc,serviceMonitor -n monitoring  |grep redis
endpoints/redis-test              192.168.1.17:9121                                                  6m2s
service/redis-test              ClusterIP   None            <none>        9121/TCP                     6m2s
servicemonitor.monitoring.coreos.com/redis-test                6m2s

7.3 Prometheus檢查

7.4 接入grafana

dashboard ID： 11835

8. 監控叢集外的服務-kafka

8.1 安裝exporter

#外掛下載地址：https://www.modb.pro/db/216588
[root@lecode-test-001 ~]# tar xf kafka_exporter-1.4.2.linux-amd64.tar.gz 
[root@lecode-test-001 ~]# ll
drwxrwxr-x  2 2000 2000     4096 Sep 16  2021 kafka_exporter-1.4.2.linux-amd64
-rw-r--r--  1 root root  8499720 Oct 27 15:30 kafka_exporter-1.4.2.linux-amd64.tar.gz
[root@lecode-test-001 ~]# mv kafka_exporter-1.4.2.linux-amd64 kafka_exporter
[root@lecode-test-001 ~]# cd kafka_exporter
[root@lecode-test-001 kafka_exporter]# ll
total 17676
-rwxr-xr-x 1 2000 2000 18086208 Sep 16  2021 kafka_exporter
-rw-rw-r-- 1 2000 2000    11357 Sep 16  2021 LICENSE
[root@lecode-test-001 kafka_exporter]# nohup ./kafka_exporter --kafka.server=192.168.1.17:9092 &
[1] 20777
[root@lecode-test-001 kafka_exporter]# nohup: ignoring input and appending output to ‘nohup.out’
[root@lecode-test-001 kafka_exporter]# tail -f nohup.out 
I1027 15:32:38.904075   20777 kafka_exporter.go:769] Starting kafka_exporter (version=1.4.2, branch=HEAD, revision=0d5d4ac4ba63948748cc2c53b35ed95c310cd6f2)
I1027 15:32:38.905515   20777 kafka_exporter.go:929] Listening on HTTP :9308
#exporter埠為9308

8.2 K8S設定

[root@lecode-k8s-master monitor]# cat kafka.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: kafka-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17  # ip為安裝應用伺服器的ip
  ports:
  - name: kafka
    port: 9308 #export的埠
    protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: kafka
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: kafka-test
  namespace: monitoring
spec:
  clusterIP: None
  clusterIPs:
  - None
  ports:
  - name: kafka
    port: 9308
    protocol: TCP
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: kafka
    app.kubernetes.io/name: kafka
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: kafka-test
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: kafka
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: kafka
      app.kubernetes.io/name: kafka
      app.kubernetes.io/part-of: kube-prometheus
      
#建立
[root@lecode-k8s-master monitor]# kubectl apply -f kafka.yaml 
endpoints/kafka-test created
service/kafka-test created
servicemonitor.monitoring.coreos.com/kafka-test created
[root@lecode-k8s-master monitor]# kubectl get -f kafka.yaml 
NAME                   ENDPOINTS           AGE
endpoints/kafka-test   192.168.1.17:9308   8m49s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/kafka-test   ClusterIP   None         <none>        9308/TCP   8m49s

NAME                                              AGE
servicemonitor.monitoring.coreos.com/kafka-test   8m48s

8.3 Prometheus檢查

8.4 接入grafana

dashboard ID：7589

9. 監控叢集外的服務-zookeeper

9.1 安裝exporter

exporter下載地址：https://github.com/carlpett/zookeeper_exporter/releases/download/v1.0.2/zookeeper_exporter

[root@lecode-test-001 zookeeper_exporter]# nohup ./zookeeper_exporter -zookeeper 192.168.1.17:2181 -bind-addr :9143 &
[2] 8310
[root@lecode-test-001 zookeeper_exporter]# nohup: ignoring input and appending output to ‘nohup.out’
[root@lecode-test-001 zookeeper_exporter]# tail -f nohup.out 
time="2022-10-27T15:58:27+08:00" level=info msg="zookeeper_exporter, version v1.0.2 (branch: HEAD, revision: d6e929223f6b3bf5ff25dd0340e8194cbd4d04fc)\n  build user:       @bd731f434d23\n  build date:       2018-05-01T20:40:14+0000\n  go version:       go1.10.1"
time="2022-10-27T15:58:27+08:00" level=info msg="Starting zookeeper_exporter"
time="2022-10-27T15:58:27+08:00" level=info msg="Starting metric http endpoint on :9143"
#exporter埠為9143

9.2 K8S設定

[root@lecode-k8s-master monitor]# cat zookeeper.yaml 
apiVersion: v1
kind: Endpoints
metadata:
  name: zookeeper-test
  namespace: monitoring
subsets:
- addresses:
  - ip: 192.168.1.17
  ports:
  - name: zookeeper
    port: 9143
    protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: zookeeper
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: zookeeper-test
  namespace: monitoring
spec:
  clusterIP: None
  clusterIPs:
  - None
  ports:
  - name: zookeeper
    port: 9143
    protocol: TCP
  sessionAffinity: None
  type: ClusterIP

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app.kubernetes.io/component: zookeeper
    app.kubernetes.io/name: zookeeper
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 0.49.0
  name: zookeeper-test
  namespace: monitoring
spec:
  endpoints:
  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    interval: 30s
    port: zookeeper
    tlsConfig:
      insecureSkipVerify: true
  selector:
    matchLabels:
      app.kubernetes.io/component: zookeeper
      app.kubernetes.io/name: zookeeper
      app.kubernetes.io/part-of: kube-prometheus


#建立
[root@lecode-k8s-master monitor]# kubectl apply -f zookeeper.yaml 
endpoints/zookeeper-test created
service/zookeeper-test created
servicemonitor.monitoring.coreos.com/zookeeper-test created
[root@lecode-k8s-master monitor]# kubectl get -f  zookeeper.yaml 
NAME                       ENDPOINTS           AGE
endpoints/zookeeper-test   192.168.1.17:9143   9m55s

NAME                     TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
service/zookeeper-test   ClusterIP   None         <none>        9143/TCP   9m55s

NAME                                                  AGE
servicemonitor.monitoring.coreos.com/zookeeper-test   9m55s

9.3 Prometheus檢查

9.4 接入grafana

dashboard ID：15026

K8S之prometheus-operator監控