Understand AWS logging: CloudWatch, CloudTrail, and service logs

AWS logging spans several services. CloudWatch Logs stores and queries many operational logs, CloudTrail records supported account activity, individual services produce their own log types, and S3 can retain archives. Start by deciding which events you need and how each source actually delivers them.

Key points

• CloudWatch Logs organizes events into log groups and streams; retention is configured per group.
• CloudTrail supplies audit events. CloudWatch Logs can store those events alongside operational logs, so the services' responsibilities can overlap in a collection pipeline.
• A log group normally retains data indefinitely until you set a policy. Choose retention deliberately.
• Collection, queries, alarms, archival, and policy review are distinct steps. Verify each one rather than assuming a single integration enables them all.

1. Map the log sources

RDS can expose logs for download and, for supported engines and log types, export them to CloudWatch Logs. ALB access logging and VPC Flow Logs are not universally enabled or automatically sent to S3.

2. Understand CloudWatch Logs organization

A log group commonly represents an application or service, for example /aws/lambda/order-api. It defines retention and other group-level settings. A log stream is a sequence of events from a source such as an execution environment or host; naming and partitioning depend on the producer.

Retention supports a defined set of periods, including options from one day through ten years, as well as indefinite retention. Choose a supported value that meets the log's purpose.

CloudWatch Logs Insights can filter and aggregate events. For example:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count() by bin(5m)

This counts messages containing the case-sensitive text ERROR in five-minute bins. If logs have a structured severity field, query that field instead of assuming every message contains the level in its text. Select the relevant groups and time range to control both results and scanned volume.

3. Turn selected events into alerts

• Metric filters turn matching log records into metric values, such as an error count.
• CloudWatch alarms evaluate metrics and can invoke supported actions, including SNS notifications and configured automated workflows.
• Dashboards combine metrics, alarms, and log-query widgets for investigation.

Test a matching record, the metric change, the alarm state, and the final notification. An available query does not by itself create an alert.

4. Archive and forward logs

S3 export tasks copy selected CloudWatch Logs data to S3. They are asynchronous, and recently ingested data can take time to become exportable. Use lifecycle policies only after assessing retention, retrieval, and access requirements; an export task is not a real-time streaming feed.

Subscription filters forward eligible incoming records to configured destinations such as Lambda, Kinesis Data Streams, or Amazon Data Firehose. This is a streaming delivery mechanism, not an automatic backfill of historical records. Permissions, encoding, quotas, retry behavior, and destination failures still need attention.

Amazon Data Firehose, formerly called Kinesis Data Firehose, can buffer and deliver data to S3 or supported HTTP endpoints. Account for buffering delay and failed-record handling.

For Guance, follow the Firehose HTTP endpoint integration for subscriptions, receiving configuration, and transformations. Begin with one group and verify timestamps, message content, and failed delivery before expanding. Include AWS-side ingestion, forwarding, transfer, and destination charges in the cost comparison.

5. Build an operating checklist

1. Inventory required service and audit logs, then verify coverage and exclusions.
2. Use focused Logs Insights queries for the investigation, accounting for ingestion delay.
3. Configure metric filters and alarms for actionable patterns.
4. Set per-group retention and test any archival path.
5. Apply IAM permissions, encryption, and KMS controls appropriate to the data.
6. Review volume, unused sources, failed deliveries, and retained data regularly.

Frequently asked questions

How do CloudTrail and CloudWatch differ?

CloudTrail records supported account actions for audit and investigation. CloudWatch provides operational monitoring services, including metrics, logs, and alarms. A trail can deliver into CloudWatch Logs; choose both the source coverage and the destination features you need.

Why are log costs higher than expected?

Separate ingestion, queries, retention, transfer, and duplicate copies. Retention controls stored history, while source-side filtering affects future ingestion. Compare archive and forwarding alternatives across the whole path rather than only one destination's storage price.

How can I search logs across AWS, Alibaba Cloud, and a data center?

Agree on service names, environment labels, timestamps, and severity mapping, then configure each source's collection path. Validate data formats, permissions, and retries independently. One cloud's configuration cannot be assumed to work for another source.

Should I download RDS logs or export them to CloudWatch?

A download can suit a one-off investigation. Supported CloudWatch exports are useful for ongoing alerts and searches across instances. Check the database engine's log configuration and export support first.

Related reading

• Azure logging
• AWS Lambda logging
• CloudTrail logging
• What is log management? (Chinese) (Chinese)

References: VPC Flow Logs, CloudWatch S3 export, and subscription filters.