Troubleshoot context deadline exceeded in Prometheus

context deadline exceeded means an operation did not finish before its deadline. First identify where it appears: a failed target scrape, a Prometheus query, or the Grafana/proxy request around that query. Those paths have different timeout settings.

Identify the failing operation

A timeout on the Targets page concerns collection from an exporter. Check its response time, connectivity, DNS, TLS, and scrape_timeout; that timeout must not exceed the scrape interval. Increasing the query timeout will not repair a slow scrape.

For a query failure, compare the exact expression, start/end times, step, and endpoint used by Grafana with a direct Prometheus query. A query that is fast for the last five minutes may be expensive over a month or at a much finer step.

Common causes include:

• Large time ranges, broad selectors, high-cardinality series, and nested subqueries.
• Competing ingestion and query work, high query concurrency, or CPU throttling.
• Memory pressure and frequent garbage collection.
• A slow network path, reverse proxy deadline, or remote storage dependency.

Reduce the work before extending the deadline

Filter to the necessary jobs, services, and instances. Shorten the requested range or use a coarser graph step where appropriate. Avoid unnecessarily broad regular expressions; a regex is not inherently slow in every case, but selecting nearly every series is expensive. Recording rules can precompute repeated aggregations, although they add evaluation work and do not backfill old results automatically.

Check Prometheus CPU, memory, query concurrency, and ingestion load alongside the failed request. Add resources if measured resource pressure is the bottleneck.

Align timeouts across the request path

The documented Prometheus query timeout default is two minutes. For example, this starts a process with a three-minute ceiling:

prometheus --config.file=prometheus.yml --query.timeout=3m

Apply the setting through the existing service, container, or Operator configuration; do not start a second process against the same storage directory. Grafana's Prometheus query timeout, its HTTP request timeout, and any intermediary proxy can impose shorter limits. Confirm the settings available in your Grafana version. A longer deadline allows more work to remain in flight and can worsen overload.

For sustained scale problems, evaluate a distributed metrics system such as Thanos, Cortex, or Mimir. Adding one is an architectural change with storage, query, and operational costs, not an automatic fix for an inefficient expression.

Common questions

Can an occasional timeout be ignored? Correlate it with resource spikes and user impact. A single interruption may be transient; repeated panel or alert failures need investigation.

Why is Grafana slow when the Prometheus browser is fast? Compare the full request first. Different ranges, steps, concurrent panel queries, data source endpoints, and proxy deadlines can explain the difference. Reducing panel query volume may help once that cause is confirmed.

References

• Prometheus command-line options
• Scrape and HTTP client configuration
• Grafana Prometheus data source configuration