Monitor a Go HTTP service with Prometheus

Expose Go runtime metrics and use promhttp to record request counts, in-flight work, and latency by route. Includes a scrape configuration and PromQL.

Best practices
Illustration of Prometheus monitoring and dashboards

The order endpoint returns a 200 for /orders/123 and a 404 for /orders/missing. Those are two outcomes of one route. Grouping by the raw URL would split the traffic across order IDs and make it hard to answer a simple question. Did requests to this handler get slower or start failing more often?

This guide instruments a Go HTTP handler with client_golang and exposes the metrics at /metrics. Start with Go runtime metrics, then add a request counter, an in-flight gauge, and a latency histogram with a stable route label.

Expose the default Go metrics first

Use Go 1.23 or newer for the pinned client version in this example. In an empty directory, create a module and install the client.

go mod init example.com/go-metrics
go get github.com/prometheus/client_golang/prometheus/promhttp@v1.23.2

Save the following as main.go and run go run ..

package main

import (
	"log"
	"net/http"

	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	mux := http.NewServeMux()
	mux.Handle("/metrics", promhttp.Handler())
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", mux))
}

Open http://127.0.0.1:8080/metrics and look for go_goroutines and the go_memstats_* family. The default registry already includes Go and process collectors. The available process_* metrics vary by operating system. A registry created with prometheus.NewRegistry() starts empty. Register the collectors you want and expose that registry with promhttp.HandlerFor. Prometheus Go instrumentation guide

Pick three signals with different jobs

Metric What it measures How to use it
http_requests_total Cumulative calls that returned from the instrumented handler, grouped by route, method, and status code Compare request rates and outcomes for the same route
http_active_requests Calls currently inside the instrumented handler in this process See whether concurrent work is building up
http_request_duration_seconds Distribution of time inside the handler, in seconds Estimate tail latency or count requests below a bucket boundary

An in-flight gauge can rise while completed request rate stays flat. The gauge alone cannot tell you whether the handler is waiting on a database, doing more work, or seeing a short burst of traffic. Use the latency distribution and request outcomes to put that increase in context.

Attach instrumentation to the matched route

Stop the first server and replace main.go with this version. The wrapper gets its route label from registration, where the template is already known. The order ID stays in the handler.

package main

import (
	"fmt"
	"log"
	"net/http"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

func main() {
	const route = "/orders/{id}"
	requests := prometheus.NewCounterVec(prometheus.CounterOpts{
		Name: "http_requests_total",
		Help: "Completed calls to the instrumented handler.",
	}, []string{"route", "method", "code"})
	active := prometheus.NewGauge(prometheus.GaugeOpts{
		Name: "http_active_requests",
		Help: "Calls currently executing in the instrumented handler.",
	})
	duration := prometheus.NewHistogramVec(prometheus.HistogramOpts{
		Name:    "http_request_duration_seconds",
		Help:    "Time spent in the instrumented handler in seconds.",
		Buckets: []float64{0.01, 0.05, 0.1, 0.3, 0.5, 1, 2.5, 5},
	}, []string{"route"})
	prometheus.MustRegister(requests, active, duration)

	orders := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		id := r.PathValue("id")
		if id == "missing" {
			http.NotFound(w, r)
			return
		}
		fmt.Fprintln(w, "order", id)
	})

	labels := prometheus.Labels{"route": route}
	instrumented := promhttp.InstrumentHandlerInFlight(active,
		promhttp.InstrumentHandlerCounter(requests.MustCurryWith(labels),
			promhttp.InstrumentHandlerDuration(duration.MustCurryWith(labels), orders),
		),
	)

	mux := http.NewServeMux()
	mux.Handle("GET "+route, instrumented)
	mux.Handle("/metrics", promhttp.Handler())
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", mux))
}

Run go run . again. Visit /orders/123 and /orders/missing, then look at /metrics. Both calls use route="/orders/{id}", with code="200" and code="404" respectively. Another order ID adds to an existing series. A router-level 404 for an unmatched path, a 405 for the wrong method, and the scrape itself never enter this wrapper.

The counter and histogram are updated after the wrapped handler returns. A panic that escapes the handler skips both updates, though the deferred gauge decrement still runs. The measured duration ends with the server handler, before you can infer how long a client took to receive or process the response. Go ServeMux routing · promhttp instrumentation

Scrape the endpoint and query the tail

This target assumes Prometheus shares the app's network namespace. A separate host or container needs an address reachable from the scraper and access controls on the metrics endpoint.

scrape_configs:
  - job_name: go-app
    scrape_interval: 15s
    static_configs:
      - targets: ['127.0.0.1:8080']

After several successful scrapes, use the first query for requests per second by route and status. The second estimates P99 for each route in seconds.

sum by (route, code) (rate(http_requests_total{job="go-app"}[5m]))

histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_request_duration_seconds_bucket{job="go-app"}[5m]))
)

Take rate() on each counter before combining instances. It can adjust for resets visible in the samples, while increments lost between scrapes cannot be recovered. The classic histogram query keeps le during aggregation and assumes matching bucket boundaries across instances. For a 300 ms service objective, the example has a 0.3 second bucket. Its rate, divided by the rate of the total observations over the same scope, estimates the share at or below the objective when traffic is present. Prometheus query functions

The same exporter can feed Guance through the DataKit Prom collector. Set urls to the reachable metrics URL, then check the stored metric names, tags, and units before moving queries or alerts. The RED and USE guide explains how to put the request signals alongside resource pressure.

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting