Deploy Grafana on Kubernetes with Helm

Install the Grafana Helm chart, keep its database on a PVC, provision a Prometheus data source and dashboards, and expose the service through TLS Ingress.

Best practices
Illustration of Prometheus metrics and dashboard visualization

A Grafana dashboard can disappear after a Pod replacement even when Prometheus still has every sample. The two applications store different things. Prometheus holds the time series. Grafana keeps dashboards created in the UI, users, and other settings in its own database. A persistent volume keeps that database around. Provisioning files let a fresh instance recreate the data sources and dashboards you put in version control.

This guide deploys Grafana on Kubernetes with Helm, connects it to Prometheus, and shows what to check when the dashboards load but the panels are empty. The examples use the community chart version 14.4.0, which packages Grafana 13.3.0 and targets Kubernetes 1.25 or later in the 1.x series. The chart examples were rendered locally with Helm.

Install Grafana and open it locally

For a disposable instance, install the Grafana community chart with defaults. If anyone will build dashboards here, install with the persistent configuration in the next section from the start.

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update grafana-community

helm upgrade --install grafana grafana-community/grafana \
  --version 14.4.0 --namespace monitoring --create-namespace

Read the generated initial administrator password and forward a local port to the Grafana Service. The chart's default Service listens on port 80 and forwards to port 3000 in the container.

kubectl -n monitoring get secret grafana \
  -o jsonpath='{.data.admin-password}' | base64 -d; printf '\n'
kubectl -n monitoring port-forward svc/grafana 3000:80

Open http://localhost:3000 and sign in as admin. The Secret holds the bootstrap credential. An administrator password previously set in Grafana can differ from that value.

Keep the database on a PVC

For one replica using Grafana's default SQLite database, mount a persistent volume at its data directory. Make sure the cluster can provision the requested storage. This example uses a Recreate Deployment strategy so the old Pod is stopped before the new one comes up. Expect a short interruption during an upgrade.

Provision a Secret named grafana-bootstrap in monitoring with admin-user and admin-password keys. Save the following as values.yaml. The Secret reference keeps the password out of the values file.

replicas: 1
deploymentStrategy:
  type: Recreate
persistence:
  enabled: true
  size: 10Gi
admin:
  existingSecret: grafana-bootstrap
  userKey: admin-user
  passwordKey: admin-password
datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
      - name: Prometheus
        uid: prometheus
        type: prometheus
        access: proxy
        url: http://prometheus-server.monitoring.svc:80
        isDefault: true
dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: default
        orgId: 1
        type: file
        options:
          path: /var/lib/grafana/dashboards/default
dashboards:
  default:
    node-exporter:
      gnetId: 1860
      revision: 46
      curlOptions: "-sSLf"

The URL in this example matches the standalone Prometheus chart installed as release prometheus in monitoring. Grafana's server reaches that address. Check kubectl -n monitoring get svc for the actual Service and port if you used a different name, namespace, or chart.

Give the dashboard something to query

The chart downloads Node Exporter Full revision 46 during Pod initialization. The provider points at the directory where the chart puts the JSON. The cluster needs network access to the download URL. If it has no external egress, remove the gnetId download entry and mount a reviewed copy of the JSON from a ConfigMap for the file provider.

Apply the configuration and check the claim:

helm upgrade --install grafana grafana-community/grafana \
  --version 14.4.0 --namespace monitoring --create-namespace \
  -f values.yaml
kubectl -n monitoring get pvc,svc

The PVC should become Bound. In Grafana Explore, select Prometheus and query up. A result for an unrelated target confirms basic connectivity. Query node_uname_info to check that Prometheus is receiving node exporter metrics, then choose the Prometheus data source in the dashboard. If a panel is still blank, check its time range and filters before changing the chart.

If you also send Kubernetes metrics to Guance, enable annotation-based discovery in the DataKit KubernetesPrometheus collector and annotate the Pod or Service endpoints you want to scrape. Prometheus and DataKit select their targets independently. Grafana continues to query the Prometheus Service.

Put an Ingress in front of the Service

Save the snippet as ingress.yaml. Add -f ingress.yaml after -f values.yaml when running the Helm command. Replace private-web with your controller's IngressClass. Point DNS at the controller and supply a TLS Secret for that host in the same namespace. Set root_url to the browser-facing HTTPS URL.

ingress:
  enabled: true
  ingressClassName: private-web
  hosts: [grafana.example.com]
  tls:
    - secretName: grafana-tls
      hosts: [grafana.example.com]
grafana.ini:
  server:
    root_url: https://grafana.example.com/

Apply the access controls your Grafana instance needs before sharing the address. The data a user can see depends on Grafana and data-source permissions. Port forwarding remains useful for local troubleshooting.

If kube-prometheus-stack already owns Grafana

Check grafana.enabled in the stack values. When it is enabled, change the grafana: block on the existing release. The dependency version is set by the stack, so check the bundled chart's values.

grafana:
  enabled: true
  deploymentStrategy:
    type: Recreate
  persistence:
    enabled: true
  admin:
    existingSecret: grafana-bootstrap
    userKey: admin-user
    passwordKey: admin-password

This is still a single SQLite-backed replica. Multiple Grafana replicas need a shared PostgreSQL or MySQL database, and Grafana Alerting has a separate HA configuration.

Questions that come up after the first deploy

Why did my dashboards disappear when Kubernetes replaced the Pod?

Without persistence, the chart uses temporary Pod storage for Grafana's data directory. Dashboards saved only in that database can disappear. Provisioned dashboards can be recreated if their source files remain available. Before switching an existing installation to a PVC, export or back up the data you need. The new volume starts without the old volume's data.

How do I reset the admin password?

If you can sign in, change it in Grafana. For recovery, use grafana cli admin reset-admin-password with the running installation's configuration and data directory so it writes to the correct database. Changing a Kubernetes Secret updates the bootstrap input. It does not update an existing Grafana user.

Can I keep dashboards in Git?

Yes. Store the JSON in Git and provision the files with ConfigMaps, or use a GitOps controller to create the labeled ConfigMaps watched by the chart's dashboard sidecar. sidecar.dashboards.enabled enables that watch. The provisioned file remains the source of truth, so an update can replace edits saved in the UI. See Grafana dashboard provisioning for its update and deletion behavior.

Further reading · Visualize Prometheus metrics in Grafana

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting