Deploy Grafana on Kubernetes with Helm
Install the Grafana Helm chart, keep its database on a PVC, provision a Prometheus data source and dashboards, and expose the service through TLS Ingress.
A Grafana dashboard can disappear after a Pod replacement even when Prometheus still has every sample. The two applications store different things. Prometheus holds the time series. Grafana keeps dashboards created in the UI, users, and other settings in its own database. A persistent volume keeps that database around. Provisioning files let a fresh instance recreate the data sources and dashboards you put in version control.
This guide deploys Grafana on Kubernetes with Helm, connects it to Prometheus, and shows what to check when the dashboards load but the panels are empty. The examples use the community chart version 14.4.0, which packages Grafana 13.3.0 and targets Kubernetes 1.25 or later in the 1.x series. The chart examples were rendered locally with Helm.
Install Grafana and open it locally
For a disposable instance, install the Grafana community chart with defaults. If anyone will build dashboards here, install with the persistent configuration in the next section from the start.
helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update grafana-community
helm upgrade --install grafana grafana-community/grafana \
--version 14.4.0 --namespace monitoring --create-namespace
Read the generated initial administrator password and forward a local port to the Grafana Service. The chart's default Service listens on port 80 and forwards to port 3000 in the container.
kubectl -n monitoring get secret grafana \
-o jsonpath='{.data.admin-password}' | base64 -d; printf '\n'
kubectl -n monitoring port-forward svc/grafana 3000:80
Open http://localhost:3000 and sign in as admin. The Secret holds the bootstrap credential. An administrator password previously set in Grafana can differ from that value.
Keep the database on a PVC
For one replica using Grafana's default SQLite database, mount a persistent volume at its data directory. Make sure the cluster can provision the requested storage. This example uses a Recreate Deployment strategy so the old Pod is stopped before the new one comes up. Expect a short interruption during an upgrade.
Provision a Secret named grafana-bootstrap in monitoring with admin-user and admin-password keys. Save the following as values.yaml. The Secret reference keeps the password out of the values file.
replicas: 1
deploymentStrategy:
type: Recreate
persistence:
enabled: true
size: 10Gi
admin:
existingSecret: grafana-bootstrap
userKey: admin-user
passwordKey: admin-password
datasources:
datasources.yaml:
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus-server.monitoring.svc:80
isDefault: true
dashboardProviders:
dashboardproviders.yaml:
apiVersion: 1
providers:
- name: default
orgId: 1
type: file
options:
path: /var/lib/grafana/dashboards/default
dashboards:
default:
node-exporter:
gnetId: 1860
revision: 46
curlOptions: "-sSLf"
The URL in this example matches the standalone Prometheus chart installed as release prometheus in monitoring. Grafana's server reaches that address. Check kubectl -n monitoring get svc for the actual Service and port if you used a different name, namespace, or chart.
Give the dashboard something to query
The chart downloads Node Exporter Full revision 46 during Pod initialization. The provider points at the directory where the chart puts the JSON. The cluster needs network access to the download URL. If it has no external egress, remove the gnetId download entry and mount a reviewed copy of the JSON from a ConfigMap for the file provider.
Apply the configuration and check the claim:
helm upgrade --install grafana grafana-community/grafana \
--version 14.4.0 --namespace monitoring --create-namespace \
-f values.yaml
kubectl -n monitoring get pvc,svc
The PVC should become Bound. In Grafana Explore, select Prometheus and query up. A result for an unrelated target confirms basic connectivity. Query node_uname_info to check that Prometheus is receiving node exporter metrics, then choose the Prometheus data source in the dashboard. If a panel is still blank, check its time range and filters before changing the chart.
If you also send Kubernetes metrics to Guance, enable annotation-based discovery in the DataKit KubernetesPrometheus collector and annotate the Pod or Service endpoints you want to scrape. Prometheus and DataKit select their targets independently. Grafana continues to query the Prometheus Service.
Put an Ingress in front of the Service
Save the snippet as ingress.yaml. Add -f ingress.yaml after -f values.yaml when running the Helm command. Replace private-web with your controller's IngressClass. Point DNS at the controller and supply a TLS Secret for that host in the same namespace. Set root_url to the browser-facing HTTPS URL.
ingress:
enabled: true
ingressClassName: private-web
hosts: [grafana.example.com]
tls:
- secretName: grafana-tls
hosts: [grafana.example.com]
grafana.ini:
server:
root_url: https://grafana.example.com/
Apply the access controls your Grafana instance needs before sharing the address. The data a user can see depends on Grafana and data-source permissions. Port forwarding remains useful for local troubleshooting.
If kube-prometheus-stack already owns Grafana
Check grafana.enabled in the stack values. When it is enabled, change the grafana: block on the existing release. The dependency version is set by the stack, so check the bundled chart's values.
grafana:
enabled: true
deploymentStrategy:
type: Recreate
persistence:
enabled: true
admin:
existingSecret: grafana-bootstrap
userKey: admin-user
passwordKey: admin-password
This is still a single SQLite-backed replica. Multiple Grafana replicas need a shared PostgreSQL or MySQL database, and Grafana Alerting has a separate HA configuration.
Questions that come up after the first deploy
Why did my dashboards disappear when Kubernetes replaced the Pod?
Without persistence, the chart uses temporary Pod storage for Grafana's data directory. Dashboards saved only in that database can disappear. Provisioned dashboards can be recreated if their source files remain available. Before switching an existing installation to a PVC, export or back up the data you need. The new volume starts without the old volume's data.
How do I reset the admin password?
If you can sign in, change it in Grafana. For recovery, use grafana cli admin reset-admin-password with the running installation's configuration and data directory so it writes to the correct database. Changing a Kubernetes Secret updates the bootstrap input. It does not update an existing Grafana user.
Can I keep dashboards in Git?
Yes. Store the JSON in Git and provision the files with ConfigMaps, or use a GitOps controller to create the labeled ConfigMaps watched by the chart's dashboard sidecar. sidecar.dashboards.enabled enables that watch. The provisioned file remains the source of truth, so an update can replace edits saved in the UI. See Grafana dashboard provisioning for its update and deletion behavior.
Further reading · Visualize Prometheus metrics in Grafana
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
