What is AIOps? From alert noise to incident response

See how AIOps detects anomalies, groups alerts and supports incident investigation. Learn when to use a baseline and what to check before automating remediation.

Best practices
Illustration of robotic arms working together

The checkout error rate climbs after a release. A minute later the same alert fires again. Then a slow database call and a certificate warning turn up in the same window. The on-call engineer needs to work out which alerts belong together and where to look first.

AIOps, or artificial intelligence for IT operations, applies data analysis and machine learning to operational work. It can detect unusual behavior, reduce duplicate alerts and help narrow an investigation. Some teams also use it to run preapproved procedures once the evidence and operating permissions are in place. The useful result at 2 a.m. is a short list of leads with the evidence attached. IBM’s AIOps overview

How four alerts become two leads

Imagine these four incoming alerts during the checkout incident. The times and names are illustrative.

Time Alert What the on-call engineer does
10:00 Checkout error rate crosses its threshold Opens a checkout investigation.
10:01 The same checkout alert repeats Deduplicates by a stable key that includes the rule, service and environment. Keeps the underlying event history.
10:02 Latency rises in payment-db Checks whether failing checkout traces include a slow call to this dependency. Adds it to the checkout investigation if the traces support the link.
10:02 A certificate is close to expiring in auth Keeps it separate in this example, where the checkout traces give no link to auth.

There are three alert identities after deduplication. With the assumed trace evidence, they point to two separate things to investigate. A shared timestamp alone would have grouped the database signal and the certificate warning together too. Even if a trace links the database to the failed checkout, you still need more evidence to establish the cause. Compare requests on the new release with an unaffected version or instance where you can. Google SRE on symptoms and causes

Which signals need a baseline?

If the user-facing error rate crosses a clearly defined limit, a fixed rule is easy to explain and review. A baseline is useful when normal traffic changes by hour or day, and there is enough history to tell that pattern from a genuine deviation. A missing feed or a change in the mix of requests can also move the observed pattern. Check for both before escalating. Capacity forecasts similarly depend on a representative history and should be treated as estimates.

Observability supplies the evidence for this work. Instrumentation produces metrics, logs and traces. Common service names, environment labels and timestamps let you match them to each other and to release records. The first workflow can span existing tools, as long as the people investigating can query the records and follow the links.

What a Guance monitor can hand to the on-call engineer

Start with the checkout error metric already collected in Guance. Set the monitor’s evaluation window and add a link to logs for the same service and time range to the event content. Link the monitor to an alert policy and configure recipients. If the policy suppresses repeated notifications, the generated events remain available in Event Explorer. In an end-to-end test, check both what reaches the on-call engineer and whether they can open the link. The monitor’s manual detection check runs the rule without sending an alert. Guance detection rules and alert policies

If the team regularly spends time gathering the same evidence, the policy can also pass monitor events to a configured Agent in Obsy Agent Teams. Set up its workspace access and any tools it needs. Start with tools and permissions that only let the Agent read the records it needs. Workspace intake runs a built-in analysis. In the task session, check which service and time range it identified and whether the records support its leads. Follow up there if you need more context. Guance task intake

Decide what stays under human control

Replay a known incident before using this workflow for paging. Count how many notifications would have gone out and check whether separate incidents would still reach the right people. Then read the investigation links as the person on call. A shorter alert stream helps only if it preserves the evidence and keeps unrelated problems visible.

For remediation, specify which known conditions permit a previously tested runbook, what access the runner has, and how to verify recovery or roll back. Keep approval for changes with a material impact on production. AIOps platform selection guide

Get a tailored plan

Contact us

Join the community

Scan with WeChat
to join the community

Try Guance

Start online and pay only for what you use.

Get started

Choose a Guance plan

Code hosting