Google Cloud Logging: query, route, and manage log costs
Find GKE errors in Logs Explorer, route new logs with the right permissions, and separate Cloud Logging storage, retention, and export costs.
Checkout is returning errors. The GKE pod still writes to stdout, but the order never turns up in Logs Explorer. Start by opening a recent entry from the same workload. The Kubernetes namespace lives under resource.labels.namespace_name; an application field such as order_id lives under jsonPayload. An error in either path can make a query look like an ingestion failure.
Cloud Logging receives logs according to each service’s collection settings. This walkthrough follows one synthetic GKE entry from query to export, then separates the costs of storing it, keeping it, and sending it elsewhere.
Read the stored log entry before writing a filter
This is a synthetic LogEntry after collection. It includes a monitored resource, a numeric business field and a trace reference.
{
"logName": "projects/example-prod/logs/stdout",
"timestamp": "2026-11-10T14:55:22.123Z",
"severity": "ERROR",
"resource": {
"type": "k8s_container",
"labels": {
"project_id": "example-prod",
"location": "asia-southeast1",
"cluster_name": "payments",
"namespace_name": "prod",
"pod_name": "checkout-7f8c",
"container_name": "checkout"
}
},
"jsonPayload": {"message": "payment failed", "order_id": 1024},
"trace": "projects/example-prod/traces/4bf92f3577b34da6a3ce929d0e0e4736"
}
timestamp, resource, and severity are top-level fields. The payload can be structured JSON, plain text, or a protobuf payload. A managed collector can turn single-line JSON on stdout into structured fields. On GKE and Cloud Run, recognized keys such as severity and logging.googleapis.com/trace can move to top-level LogEntry fields. Check the stored representation before writing queries that assume a field remains in jsonPayload. Structured logging · Monitored resource types
On GKE, the managed Agent collects container stdout and stderr when workload logging is enabled. Standard clusters can change which logging components are collected. Workload logging requires system logging, and Autopilot does not support turning cluster logging off. A Compute Engine VM that writes application logs to files has a different collection path. Check that Ops Agent is installed and a receiver covers those paths. GKE logs · Ops Agent
Query GKE errors in Logs Explorer
Select the project or log view that contains the entry and set a time range around the error. Paste this filter into Logs Explorer. The separate lines are joined by implicit ANDs. Replace the namespace and order number with values from your workload.
resource.type="k8s_container"
resource.labels.namespace_name="prod"
severity>=ERROR
jsonPayload.order_id=1024
For a shell workflow, the same Logging filter works with the read-only command below. Set the project ID and use a time range that covers your incident. This command searches the last 24 hours and returns at most 20 entries. It requires a configured Google Cloud CLI and permission to read the relevant logs.
gcloud logging read '
resource.type="k8s_container"
resource.labels.namespace_name="prod"
severity>=ERROR
jsonPayload.order_id=1024
' --project=example-prod --freshness=24h --limit=20 --format=json
If the result is empty, start with resource.type and add the remaining conditions one at a time. Filter on resource.labels.pod_name to follow a specific pod. Use the exact stored trace value to look up logs across the same trace where the field is populated. The histogram shows the distribution of matching entries over the selected time range. Treat the matches as log records. A retry can produce several records for one failed order. Logging query language · gcloud logging read
Route new entries to the right destination
A sink evaluates incoming logs against its inclusion filter and any exclusions. It does not replay historical records when you create it. In an ordinary project, each sink evaluates an entry independently. Adding a Pub/Sub sink can leave the same entry stored in the bucket reached by _Default. To verify a route, use a new matching event after the sink and its permissions are in place.
| Destination | What it changes |
|---|---|
| BigQuery dataset | Writes new matching logs for SQL joins, reporting, and analysis. Budget for dataset storage and queries. |
| Cloud Storage bucket | Writes batches of JSON files. Routing is processed hourly, and a new sink can take several hours to begin delivery. Lifecycle policies can move objects to other storage classes with different access and minimum-duration terms. |
| Pub/Sub topic | Delivers log entries to a subscriber pipeline near real time. The JSON envelope carries the base64-encoded LogEntry in message.data. |
The identity that creates a sink and its writerIdentity have different jobs. When the sink reports a writer identity, give it the documented routing permission and the appropriate destination access. For Pub/Sub, the relevant destination role is Publisher on the target topic. The application that consumes from the subscription has its own permissions. Set up a log sink · Read Pub/Sub log messages
If the subscriber forwards to Guance, decode a real subscriber sample before building the Pipeline rules. Preserve event time, map severity to a supported status, and keep numeric payload fields numeric. If the source trace is a full resource name, extract the ID and check its representation against the trace already ingested in Guance. A field named trace_id alone does not establish that association. Subscriber authentication, delivery retries, and duplicate handling belong to the forwarding path. Guance Log Pipeline · Associate logs with traces
Count the events that the alert is meant to count
A log-based counter can turn matching ERROR entries into a Cloud Monitoring time series. Keep labels bounded, for example by service or error class. Extracting every order ID as a label can create a separate time series for each value and increase metric costs.
User-defined log-based metrics count new matching entries from the time you create the metric in a billing-enabled project. An exclusion can remove entries from a bucket while a project-scoped metric still counts matching entries received by that project. A bucket-scoped metric counts entries stored in its bucket. System-defined metrics count stored entries under Google’s routing rules. Match the metric scope to the view your on-call team can inspect. If the alert concerns checkout failure rate, use a denominator of eligible checkout attempts. A count of error logs cannot supply it. Log-based metrics · Bucket-scoped metrics
Separate storage, retention, and destination charges
Apply a targeted exclusion to the _Default sink when those entries should not reach its destination. Verify the other sinks and any alerts first. The filter runs after the API receives the logs, so the same write calls still count toward the API quota.
Project-level _Default and user-defined log buckets start at 30 days and can be set to 1–3,650 days. Folder and organization _Default retention is fixed at 30 days. _Required stores certain audit logs for 400 days without Cloud Logging storage or retention charges. Google prices ordinary log storage and retention beyond 30 days separately. The Log Router has no extra routing fee. BigQuery, Cloud Storage, and Pub/Sub usage are metered by their own services. Check the current pricing page for networking log exceptions and eligible allowances before estimating savings. Retention limits · Google Cloud Observability pricing
Use Logs Explorer for individual requests and resource-level investigation. For SQL over logs already stored in an eligible bucket, use Observability Analytics. A BigQuery sink makes sense when the destination dataset itself is part of the reporting architecture, for example when joining with payments data. The retention of the source bucket and the retention of the exported dataset remain separate. SQL analysis of stored logs
Related reading
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
