ArgoCD 可观测性最佳实践

ArgoCD 简介

ArgoCD 是一个基于 GitOps 的持续交付工具，专为 Kubernetes 环境设计，以声明式的方式部署和管理应用程序。

ArgoCD 通过 Prometheus 协议暴露指标，通过这些指标可用于监控 ArgoCD 服务状态和应用状态。ArgoCD 主要暴露三类指标：

Application Controller Metrics：ArgoCD 应用相关指标，例如应用数、ArgoCD 状态等。
API Server Metrics：ArgoCD API 请求指标，例如请求数、响应码等。
Repo Server Metrics：Repo Server 相关指标，例如 Git 请求数、Git 响应时间等。

本次介绍的是通过 KubernetesPrometheus 的自动发现，采集 ArgoCD 的指标数据。

观测云

观测云是一款专为IT工程师打造的全链路可观测产品，它集成了基础设施监控、应用程序性能监控和日志管理，为整个技术栈提供实时可观察性。这款产品能够帮助工程师全面了解端到端的用户体验追踪，了解应用内函数的每一次调用，以及全面监控云时代的基础设施。此外，观测云还具备快速发现系统安全风险的能力，为数字化时代提供安全保障。

部署 DataKit

登录观测云控制台，点击「集成」 -「DataKit」 - 「Kubernetes」，下载 datakit.yaml，拷贝第 3 步中的 token。

编辑 datakit.yaml ，把 token 粘贴到 ENV_DATAWAY 环境变量值中“token=”后面。

kubectl apply -f datakit.yaml
kubectl get pod -n datakit

采集器配置

ArgoCD 指标采集

datakit.yaml 修改 KubernetesPrometheus 采集配置。

mountPath 挂载把 KubernetesPrometheus 的配置文件挂载到容器内。

          - mountPath: /usr/local/datakit/conf.d/kubernetesprometheus/kubernetesprometheus.conf
            name: datakit-conf
            subPath: kubernetesprometheus.conf
            readOnly: true

添加 datakit.yaml 的 configmap 文件下添加 kubernetesprometheus.conf。

    kubernetesprometheus.conf: |-
      [inputs.kubernetesprometheus]
        [[inputs.kubernetesprometheus.instances]]
          role       = "service"
          namespaces = ["argocd"]
          selector   = "app.kubernetes.io/name=argocd-server-metrics"

          scrape     = "true"
          scheme     = "http"
          port       = "__kubernetes_service_port_metrics_targetport"
          path       = "/metrics"
          params     = ""

          [inputs.kubernetesprometheus.instances.custom]
            measurement        = "argocd-server"
            job_as_measurement = false
            [inputs.kubernetesprometheus.instances.custom.tags]
              svc_name      = "__kubernetes_service_name"
              pod_name      = "__kubernetes_service_target_name"
              pod_namespace = "__kubernetes_service_target_namespace"
        [[inputs.kubernetesprometheus.instances]]
          role       = "service"
          namespaces = ["argocd"]
          selector   = "aapp.kubernetes.io/name=argocd-metrics"

          scrape     = "true"
          scheme     = "http"
          port       = "__kubernetes_service_port_metrics_targetport"
          path       = "/metrics"
          params     = ""

          [inputs.kubernetesprometheus.instances.custom]
            measurement        = "argocd"
            job_as_measurement = false
            [inputs.kubernetesprometheus.instances.custom.tags]
              svc_name      = "__kubernetes_service_name"
              pod_name      = "__kubernetes_service_target_name"
              pod_namespace = "__kubernetes_service_target_namespace"

        [[inputs.kubernetesprometheus.instances]]
          role       = "service"
          namespaces = ["argocd"]
          selector   = "app.kubernetes.io/name=argocd-notifications-controller-metrics"

          scrape     = "true"
          scheme     = "http"
          port       = "__kubernetes_service_port_metrics_targetport"
          path       = "/metrics"
          params     = ""

          [inputs.kubernetesprometheus.instances.custom]
            measurement        = "argocd-application-controller"
            job_as_measurement = false
            [inputs.kubernetesprometheus.instances.custom.tags]
              svc_name      = "__kubernetes_service_name"
              pod_name      = "__kubernetes_service_target_name"
              pod_namespace = "__kubernetes_service_target_namespace"

      [inputs.kubernetesprometheus.global_tags]
        instance = "__kubernetes_mate_instance"
        host     = "__kubernetes_mate_host"

role ：填需要采集的负载的类型，比如 service，nodes，deployment，statefulset
namespaces：填对应的 ArgoCD 采集指标服务所在的空间
selector：填对应服务的label信息，或者 selector 信息
port： ArgoCD 指标暴露的端口，这里默认不用改
path：指标暴露的路径
measurement：自定义指标集名称

把 datakit.yaml 上传到可以连接到 Kubernetes 集群的主机上，执行如下命令。

kubectl apply -f datakit.yaml
kubectl get pod -n datakit

当看到状态是“Running”后表示安装 DataKit 成功。

ArgoCD 链路采集

修改 datakit.yaml 打开 opentelemetry 链路采集器。

apiVersion: v1
kind: ConfigMap
metadata:
  name: datakit-conf
  namespace: datakit
data:
  #### opentelemetry
  opentelemetry.conf: |-
    [[inputs.opentelemetry]]
      [inputs.opentelemetry.http]
       enable = true
       http_status_ok = 200

      [inputs.opentelemetry.grpc]
       trace_enable = true
       metric_enable = true
       addr = "0.0.0.0:4317"

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: daemonset-datakit
  name: datakit
  namespace: datakit
spec:
  ...
  template:
    ...
    spec:
      ...
      containers:
        ...
        volumeMounts:
        ...
        - mountPath: /usr/local/datakit/conf.d/opentelemetry/opentelemetry.conf
          name: datakit-conf
          subPath: opentelemetry.conf
        ...

开启 datakit 服务 opentelemetry svc 的端口。

apiVersion: v1
kind: Service
metadata:
  name: datakit-service
  namespace: datakit
spec:
  selector:
    app: daemonset-datakit
  ports:
    - name: datakit
      protocol: TCP
      port: 9529
      targetPort: 9529
    - name: opentelemetry
      protocol: TCP
      port: 4317
      targetPort: 4317

修改 Argo CD 的 ConfigMap argocd-cmd-params-cm，加入 otlp.address 地址，即可获取 Argo CD 的链路。

argocd-cmd-params-cm.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/name: argocd-cmd-params-cm
    app.kubernetes.io/part-of: argocd
  name: argocd-cmd-params-cm
  namespace: argocd
data:
  otlp.address: datakit-service.datakit:4317

执行命令：kubectl apply -f argocd-cmd-params-cm.yaml 然后重启 ArgoCD 服务即可。

ArgoCD 日志采集

DataKit 默认会采集容器的标准输出日志，无需额外配置采集。

关键指标

`argocd-server`

Metric	Description
process_start_time_seconds	The start time of the process since unix epoch in seconds in the API Server
go_goroutines	The number of goroutines that currently exist in the API Server
grpc_server_handled_total	The total number of RPCs completed on the server regardless of success or failure
grpc_server_started_total	The total number of RPCs started on the server
go_memstats_alloc_bytes	The number of heap bytes allocated and still in use in the API Server

`argocd-application-controller`

Metric	Description
argocd_app_info	Information about Applications. It contains tags such as syncstatus and healthstatus that reflect the application state in Argo CD. The metric value is constant
go_memstats_heap_alloc_bytes	The number of heap bytes allocated and still in use in the Application Controller
process_cpu_seconds_total	The total user and system CPU time spent in seconds in the Application Controller
argocd_app_reconcile_count	Reconcile count
argocd_app_reconcile_bucket	Count of Application Reconciliation by Duration Bounds
workqueue_depth	Depth of the Workqueue
argocd_kubectl_exec_total	Count of Kubectl Executions
argocd_app_k8s_request_total	Count of Kubernetes Requests Executed
argocd_kubectl_exec_pending	Count of Pending Kubectl Executions

`argocd-repo-server`

Metric	Description
argocd_git_request_total	Count of Git Ls-Remote Requests
argocd_git_request_duration_seconds_bucket	Git Ls-Remote Requests Performance

场景视图

监控器配置

ArgoCD 异常检测

window("M::argocd-server:(avg(argocd_info)) BY version", '5m')