Monitor Python applications with Prometheus and /metrics
Instrument Flask and FastAPI with the Prometheus Python client, choose route labels and latency buckets, and aggregate metrics from multiple workers.
An order endpoint can slow down while Python's garbage collection metrics look steady. To see where the slowdown is, expose a request counter and latency histogram for the route. Then check that a scrape represents the whole process or worker group you meant to measure.
Expose a counter on a local scrape endpoint
Install prometheus-client in a virtual environment. Save the example as demo.py, run python demo.py, then read http://127.0.0.1:8000/metrics. The client starts an HTTP server in a daemon thread. The example keeps its main thread alive and simulates requests so you can watch the counter change across scrapes.
from prometheus_client import start_http_server, Counter
import random, time
REQUEST_COUNT = Counter('app_requests_total', 'Total requests', ['method', 'endpoint'])
def handle_request():
REQUEST_COUNT.labels(method='GET', endpoint='/api').inc()
if __name__ == '__main__':
start_http_server(8000, addr='127.0.0.1')
while True:
handle_request()
time.sleep(random.random())
The default registry also exposes metrics such as python_gc_objects_collected_total. A successful scrape proves that the endpoint can be reached. An application counter only moves when the code that increments it runs. Python client HTTP exposition
Match the metric type to the question
A Counter accumulates events and resets when the process restarts. A Gauge represents a value such as queue depth, which can rise or fall. A Histogram counts observations against bucket boundaries and exposes both the count and sum. Use seconds for latency. This example times a stand-in for an application operation.
from prometheus_client import Counter, Gauge, Histogram
from time import sleep
errors = Counter('app_errors_total', 'Total errors', ['type'])
queue_depth = Gauge('job_queue_depth', 'Current queue depth')
queue_depth.set(42)
latency = Histogram('request_duration_seconds', 'Request latency',
buckets=[0.01, 0.05, 0.1, 0.5, 1, 5])
with latency.time():
sleep(0.02)
Choose buckets around thresholds you actually use. If your service objective is 300 ms, add 0.3 as a boundary so the cumulative bucket can count requests at or under that threshold. Each combination of label values creates a separate set of series. Prometheus histograms and summaries
Instrument a Flask route and check its labels
Save the following as app.py, install Flask alongside the client, and run flask --app app run locally. Request /orders/1, /orders/2 and a path that does not exist. Then scrape /metrics.
from flask import Flask, Response, g, request
from prometheus_client import Counter, Histogram, generate_latest, CONTENT_TYPE_LATEST
import time
app = Flask(__name__)
REQS = Counter('http_requests_total', 'Requests', ['method', 'endpoint', 'status'])
LAT = Histogram('http_request_duration_seconds', 'Latency', ['endpoint'],
buckets=[0.01, 0.05, 0.1, 0.3, 0.5, 1, 5])
@app.get('/orders/<int:order_id>')
def order(order_id):
return {'order_id': order_id}
@app.before_request
def before():
g.start_time = time.perf_counter()
@app.after_request
def after(resp):
if request.endpoint != 'metrics':
endpoint = request.url_rule.rule if request.url_rule else 'unmatched'
method = request.method if request.method in {'GET', 'HEAD', 'OPTIONS'} else 'OTHER'
REQS.labels(method, endpoint, str(resp.status_code)).inc()
LAT.labels(endpoint).observe(time.perf_counter() - g.start_time)
return resp
@app.route('/metrics')
def metrics():
return Response(generate_latest(), content_type=CONTENT_TYPE_LATEST)
The first two calls share the label /orders/<int:order_id>. Unmatched paths share unmatched, and the scrape endpoint is excluded. Keep user IDs, raw paths, and query strings out of metric labels. The clock stops when Flask has produced the response for the hook. A streaming response may still be sending its body. Flask request lifecycle
Expose FastAPI metrics with the ASGI app
The client can serve metrics inside a FastAPI application. Save the sample as app.py in a separate directory and run fastapi dev app.py with fastapi[standard] and prometheus-client installed. Request /orders/1 first, then scrape /metrics/. A mount serves a path prefix, so /metrics redirects to the slash-terminated URL. Point the scraper at the final path.
from fastapi import FastAPI
from prometheus_client import Counter, make_asgi_app
app = FastAPI()
ORDERS_READ = Counter('orders_read_total', 'Completed order reads')
@app.get('/orders/{order_id}')
def order(order_id: int):
ORDERS_READ.inc()
return {'order_id': order_id}
app.mount('/metrics', make_asgi_app())
The mount only exposes the registry. Here the route increments its own business counter. To capture request duration and status across routes, add instrumentation that identifies the matched route and define when it records errors and finishes timing. Python client FastAPI example
Check what the scrape represents with multiple workers
These examples use a single process. Under Gunicorn, a default in-memory registry only sees the worker handling the scrape. For client multiprocess mode, give each worker group its own writable directory. Set PROMETHEUS_MULTIPROC_DIR before the workers start and empty that directory between complete runs, when the previous group has stopped. Expose a fresh CollectorRegistry with MultiProcessCollector in the scrape handler. Registering the same application metrics directly on that scrape registry can export them twice.
For Gunicorn, call multiprocess.mark_process_dead(worker.pid) from child_exit to remove the exiting worker's live-mode Gauge files. Keep the Counter and Histogram files so a worker exit does not discard earlier observations during the current run. Choose a Gauge aggregation mode that fits the value, such as livesum for per-worker in-flight requests. Process CPU and memory collectors do not become group metrics through this mechanism. Python client multiprocess mode
The same process boundary applies to a Celery prefork worker. A parent-only HTTP exporter does not see child-local counters. For a machine-bound batch job, a Node Exporter textfile snapshot is an option. Queue depth and backlog belong to the broker's metrics. Python client textfile collector
Read tail latency from the buckets
For the Flask histogram, this query estimates P99 by route across instances in the python-api scrape job.
histogram_quantile(
0.99,
sum by (le, endpoint) (
rate(http_request_duration_seconds_bucket{job="python-api"}[5m])
)
)
Apply rate before summing so Prometheus can account for resets in each series. Preserve le to keep the bucket boundaries, and use the same buckets in each instance you aggregate. The result is an estimate in seconds. Read it alongside request volume. The Python client's Summary exposes count and sum without client-side quantiles. Python Summary documentation
Keep the scrape endpoint reachable only by authorized monitoring clients. If Guance is collecting the data, set the Python service's scrape URL in the DataKit Prom input. Generate a successful request and an error, check the expected counter and bucket changes, and confirm that each worker group is collected once. Application metrics and APM traces can then be compared over the same window. Prometheus best practices
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
