Case study: Grafana observability native IRM
From Alert Notification to Observability-Native Context. Rethinking the First Step of Incident Response at Grafana.
Product designer
The problem
An alert indicates something is wrong. It does not surface the context. And an AI investigation always starts cold, which makes that start slow.
Every firing alert leaves the same two questions. Is this an incident? What context sits behind the signal?
The context already exists. It lives somewhere else in Grafana. The alert fires on payment-service, and a responder — or an agent — has to go and find it: open the dashboard, open the logs, open the metrics.
The first part of an investigation is context discovery. For an agent, that means identifying the relevant data indirectly. Confidence drops, the investigation takes longer, and it costs more tokens.
Initial alert group. The notification says something fired. The panel, the logs and the metrics are not here.
Hypothesis
Bring the right observability context into alerts and incidents automatically, so neither the person nor the agent has to reconstruct it.
Exploration
Opinionated context
The engine picks the context from the alert type. A CPU alert arrives with the CPU panel and the related logs and metrics already chosen.
Pros. Nothing to set up.
Cons. Few use cases, and it can surface data that is irrelevant or not enough.
Configurable enrichment
The setup happens once. Every later alert of that type arrives with the context already attached.
Pros. Any data, any use case.
Cons. Someone has to configure it, and they can attach too much.
What an enrichment is
A rule that attaches data — logs, traces, a dashboard panel, any query result — to an alert before the alert is delivered. Each rule is scoped to one alert or to many. Configure it once, and every matching alert is enriched on its own.
Enrichments are rules, not a one-off annotation. Scoped once, they run on every matching alert.
How we decided
We did not pick a direction on one score alone. We scored both options against what actually mattered for a responder and for an agent.
| Criteria | Opinionated default | Configurable |
|---|---|---|
| Fast to start | Yes | No |
| Relevant across teams | No | Yes |
| Reusable long-term | No | Yes |
| Signal-to-noise risk | High | Low |
The team chose configurable: control, instead of a default that tries to fit every alert.
Enrichment is owned by another team, so configurable was also the option that matched how ownership was split. We couldn’t shape the configuration UX, but we recommended shipping the most common enrichment types out of the box, so teams get value before writing a single rule.
Solution
When the alert fires, one sequence runs. There is no manual step between the notification and the context.
- Step 1Alert fires
- Step 2Enrichment runs
- Step 3Alert is enriched
- Step 4Agents can read the data
- Step 5People see it in the UI
Configured once, the same context reaches web, mobile and Slack.
When the incident is declared, the context comes with it. No manual steps.
Before and after
Before
- The responder finds the context by hand.
- That search is repeated on every alert.
- The agent has no direct context, so every investigation starts cold.
- Investigation is a desktop task, separate from the notification.
After
- Enrichments attach the observability context on their own.
- The agent can use that context directly. Early tests show fewer steps to a root cause.
- The same data is reachable from desktop, the mobile app and Slack.
- Configure once, then the observability data is part of the alert.
Impact
The shift is from context discovery to a native observability experience.
What we're watching
Alert enrichment is in public preview. Early tests of AI investigation are promising: time efficiency is about 3×. Adoption, and the effect on responders, are still being measured.
