MCP gives an agent access to tools. Skills help define how it should behave. Memory can store information from previous interactions.

But a domain-specific agent also needs to know how the things in its world fit together.

A support agent needs to understand customers, accounts, tickets, products and known issues. An SRE agent deals with services, databases, deployments, incidents, and runbooks. A sales agent has companies, people, opportunities, meetings, and decisions.

We built Open Index to manage this layer.

It’s an open-source framework for representing domain knowledge as structured entities and relationships that an agent can navigate.

How did we actually land here?

Our earlier knowledge setup used embeddings and Markdown files.

It was a good place to start. As the amount of knowledge grew, though, we started seeing problems that retrieval alone wasn't solving.

The first was non-determinism.

We could replay the same or a very similar scenario and see the agent pick completely different documents. That changed the investigation path and sometimes the final result.

Then there was navigation.

A retrieved document might tell the agent something about a service, but the agent still had to figure out what to inspect next.

Which database does this service use? What else depends on that database? Which deployment touched the service recently? Have we seen a similar incident before?

That information existed, but the connections between those pieces weren't always explicit.

We also started seeing context poisoning. As more documents and information were added, the agent had more opportunities to pick up something irrelevant or outdated and follow the wrong path.

Updating the knowledge base became difficult too after a certain amount of knowledge.

When the agent learns something new, you need to know where that information belongs, whether it already exists, whether an older version should be replaced, and whether it conflicts with something else.

A folder of Markdown files gets difficult to manage once both humans and agents are continuously reading and writing to it.

From Markdown to structured context

We started adding indexes and structured data around our existing context.

That fixed some problems, but created a new set of things we had to handle:

• updates and de-duplication

• stale or conflicting information

• relationships between entities

• version history

• tracking which context the agent used

Eventually, this became its own context layer.

We turned that work into Open Index.

How Open Index works

Open Index lets you define the objects that exist in your domain and the relationships between them.

For example, an SRE setup could have:

Service → Database → Deployment → Incident → Runbook

A support setup could have:

Customer → Account → Ticket → Product → Known Issue

You define the models and schemas. You decide how information gets added or updated. You define how entities are connected.

The agent can then use those connections while working through a task.

Say an investigation starts with a service.

Instead of searching the entire knowledge base every time it needs more information, the agent can follow known relationships from that service to its dependencies, deployments, previous incidents or runbooks.

The context itself gives the agent a map of where it can go next.

What Open Index handles

There are three parts we care about.

• Structured memory: Domain knowledge is stored as structured entities rather than only documents or chunks. The structure is defined by the domain. Open Index doesn't require every agent to use the same schema.

• Navigation: Relationships between entities give the agent paths through the context. An agent investigating a service can move through the objects connected to that service instead of repeatedly trying to rediscover those connections through retrieval.

• Relationships: The connections between entities can represent things such as dependencies, ownership, decisions and correlations. That gives the agent information about how two things are related, not only whether their text happens to be similar.

Getting started with Open Index

Open Index is domain agnostic. You can define the entities and relationships based on whatever your agent works with: marketing, sales, legal, support, security, SRE, or something else.

The basic flow is:

Define your entities and schemas → populate and update context → connect entities → let the agent navigate it.

You can run Open Index with a single command and see how the setup works before building it into an agent.

The tool is open source:

github.com/DrDroidLab/open-index

If you're working on one, we'd like to hear what your context setup looks like and what breaks as it grows.

I was there for KubeAutoDay on June 17 and KubeCon + CloudNativeCon India on June 18 and 19, representing DrDroid at the booth. Here's what stood out, where I think things are headed, and what's worth looking into next.

KubeAutoDay: the warm-up nobody expected

KubeAutoDay ran for the first time in India this year, the evening before KubeCon proper started. It felt less like a standalone conference and more like the pre-party: smaller room, more intimate conversations, people comparing notes before the main event got loud. We sponsored it the same way we sponsored KubeCon, but the tone was different. Less pitch, more "okay, what's everyone actually building?"

What we saw: two days of booth conversations

Most of my two days at KubeCon were spent doing two things: getting the booth ready before doors opened, and then standing at it, explaining what DrDroid does to whoever walked up.

That second part is where the real signal was. I wasn't sitting in sessions hearing what speakers think comes next. I was hearing it straight from engineers and platform leads who stopped by, asked what we do, and then told me what they're actually trying to solve right now. A lot of those conversations circled back to the same place: people aren't asking "how do we monitor more," they're asking "how do we get something to act on what we're already monitoring." Automation and AI-driven response came up constantly, not as a buzzword, but as the next thing teams are actively trying to figure out how to adopt.

Where we're going

That's exactly the bet DrDroid is making. We're not trying to give teams another dashboard. We're building toward incident response that doesn't wait for a human to start digging: auto-investigation, service discovery, an MCP server that lets agents reason across incident data. The conversations at the booth this year confirmed it. Almost everyone who stopped by wasn't asking if this direction makes sense. They were asking how soon they could get it.

Good topics to talk about next

A few things came up enough times at the booth that they're worth writing about properly:

• What teams actually mean when they say "AI agents for incident response." The phrase gets thrown around a lot. The expectations behind it vary more than people admit.

• Where automation should stop and a person should take over. Every conversation eventually got here, and nobody has a clean answer yet.

• What adopting OpenTelemetry looks like for a team that's mid-migration, not greenfield. Most people we talked to are somewhere in the middle, not starting from scratch.

• What changes for a platform team once incident response is built in from day one, instead of added after the first bad outage.

Find us there next time

If you stopped by the booth this year, thank you for the conversation. If you didn't, we'll be back at the next one. Come find us, skip the small talk, and tell us about your last incident. That's the conversation we actually want to have.

Type "AI SRE" into a search bar and you'll get four different answers, none of which agree with each other. One vendor means "an AI chatbot bolted onto your alerts." Another means "we replaced your on-call rotation with an agent." A third just means "we added an LLM summary field to our dashboard." Same two words, three unrelated products.

That's not a branding problem. It's a definition problem, and it's worth fixing now, before "AI SRE" hardens into shorthand for whoever raised the biggest round instead of whoever built the thing that actually diagnoses an incident.

A Working Definition: Software That Performs the Work

AI SRE is software that performs the diagnostic and remediation work of a site reliability engineer during an incident: correlating signals across your stack, forming a hypothesis about root cause, and either acting on that hypothesis or handing a human a ready-to-verify answer.

The key word is performs, not assists. A tool that summarizes an alert for you is assisting. A tool that pulls logs, traces, and deploy history, rules out three plausible causes, and tells you "the checkout service started erroring at 14:32, right after the payment-gateway config push, here's the diff" is performing SRE work. Those are different products even when they share a chat interface.

That distinction is the whole ballgame, and most of the confusion in the space comes from vendors blurring it.

Why the Definition Keeps Shifting

Three things are colliding at once, and none of them are fully sorted out.

The label got attached before the behavior did. "AI SRE" started showing up in pitch decks around the same time GPT-4-class models made it plausible to reason over structured telemetry data. Plenty of companies had the label ready before they had the reasoning loop working. So the term picked up baggage from products that never closed the diagnostic loop, they just added a natural-language layer over an existing dashboard.

AIOps never fully died, and its scope overlaps. AIOps promised anomaly detection and noise reduction: fewer alerts, smarter routing, some correlation. AI SRE promises something further downstream, actual root cause reasoning and remediation. But because both terms touch "AI plus operations," people use them interchangeably, and vendors let them, because AIOps has a decade of enterprise trust behind it that a two-year-old category doesn't.

The agent part is genuinely unsettled. Does AI SRE mean a fully autonomous agent that restarts pods and rolls back deploys without asking? Or does it mean a highly capable co-pilot that never takes write actions, only read actions plus a recommendation? Both exist in the market right now under the same name, and the risk tolerance for each is wildly different. A bank's platform team and a five-person startup have opposite answers to "should this thing touch prod," and the category hasn't split into those two lanes yet.

Three things AI SRE gets confused with

It helps to draw the boundary by exclusion.

It's not a chatbot wrapper on top of your existing monitoring stack. If the product's core value is "ask questions about your dashboards in English," that's a UX improvement on observability, not SRE work. SRE work means correlating across tools your dashboard doesn't already connect: the deploy pipeline, the incident channel, the runbook, the on-call schedule, the actual code diff.

It's not anomaly detection with a new name. Flagging that a metric moved 3 standard deviations from baseline is useful and has existed for years. It's not diagnosis. Diagnosis is explaining why it moved and what to do about it.

It's not a summarizer. Taking a wall of log lines and shrinking them into three sentences saves time, but it doesn't reduce mean time to resolution on its own unless the summary points at a cause a human wouldn't have found as fast.

A working test you can apply to any claim

Here's a rough test for whether something qualifies: can it answer "why did this break" without a human first telling it where to look? If a human has to point the tool at the right service, the right time window, and the right dependency before it produces anything useful, it's a search tool with a nice UI. If it can start from an alert and narrow the blast radius on its own, using context it gathered rather than context it was handed, that's closer to the real thing.

None of this means the category is fake. It means it's early, the same way "cloud native" was messy in 2015 before CNCF gave it edges. The underlying need, cutting the time between "something's wrong" and "here's why and here's the fix," is real and it's getting worse as systems get more distributed and more people ship code without ever learning the infrastructure underneath it.

Where DrDroid draws the line

We built DrDroid around the "performs, not assists" side of that test, so it's worth saying plainly where we land on the questions above rather than leaving it implied.

On read versus write: DrDroid's default posture is diagnostic. It correlates across your alerting, logs, traces, deploys, and infra config on its own, forms a root cause hypothesis, and hands it to your on-call engineer to verify, rather than acting unprompted in prod. Teams can wire up remediation actions on top of that once they trust the diagnosis, but trust has to come first. An agent that's confidently wrong and empowered to restart services is worse than no agent at all.

On the AIOps overlap: AIOps tools tell you something changed. DrDroid's job starts one step later, explaining what changed and why it matters, using the same postmortem logic an experienced SRE would apply: what deployed recently, what's upstream and downstream of the failing service, what looks different from last week's baseline.

That's also why "does it need a human to point it at the problem" is the test we hold ourselves to internally, not just a line for a blog post. It's the difference we're trying to build toward, and it's the reason we'd rather argue for a tight definition of this category than a loose one that includes us by default.

Where the Category Is Heading

Expect the term to fracture into sublabels over the next year or two, the same way "observability" split into logs, metrics, and traces once everyone agreed on what the umbrella term meant. My guess: something like "diagnostic AI SRE" (reasoning, read-only, human approves the fix) and "autonomous AI SRE" (write access, acts within guardrails) become the actual distinction people care about, and "AI SRE" alone stops meaning much on its own, the way "cloud" alone doesn't tell you if someone means SaaS, IaaS, or a Raspberry Pi in their closet.

Until then, the honest move for anyone building or buying in this space is to ask the specific question instead of trusting the label: does it read or does it write, does it need a human to point it at the problem or does it find the problem itself, and what happens the first time it's wrong.

The Future:

At one end, a human reads alerts, opens issues, and fixes manually. At the other end is the future, where humans are rarely involved in Day 2 operations; a system finds the problem, works out the cause, and fixes itself, while a human is only tagged in when needed.

The useful question there isn't "are we automated?" It's like: what specifically is stopping us from moving up one level? And is there any merit in moving another layer up?

At DrDroid, we've been iterating over the different layers of automation & autonomy in production monitoring systems.

In this blog, we have discussed a model to define different levels of autonomy in software systems and what each of these levels means.

Level 0: Observability

Observability is the first layer of moving towards autonomous infrastructure. Here's what it means for a software system to be classified as a Level 0 system:

• Data exists: Your stack is instrumented with fair coverage of logs/metrics/traces/events so that unknowns can be identified and resolved (by humans).

• Signals exist: Exhaustive coverage of alerts across infrastructure, application, and CUJs (Critical User Journeys) that help your team identify/detect early on.

• Human driven: A human does everything. An alert tells you something is wrong; you decide what that means, where to look, and what to do about it.

Level 0: Outcome

An engineering team has enough data at most points to be able to detect, diagnose, and fix an issue with the data points. And that is where the first level ownership comes in.

Ownership: Engineers & Teams own everything.

Level 1: Scripted automation

This is where most of the Pre-AI era automations reside. For example,

• Have a known issue where pods need to be restarted.

• Have a user related alert - auto fetch 5 more data points and append them to the alert.

Folks automated the runbook. Beyond the runbooks, engineers handle everything the scripts don't cover, which is still most of the work to do.

L1 outcome:

The biggest value add often of the first layer is

• Fallbacks with quick remediations come into picture.

• Automations by engineers who would otherwise manually do the exact same steps.

Scripted automation is reliable and brittle at the same time. It does exactly what you told it and nothing else. The moment an incident doesn't match a known pattern (which is quite often!), you end up in the same loop of humans doing things. Also, they can get outdated as systems evolve.

Tools: Automation & workflow frameworks like Playbooks and StackStorm, or just plain vanilla bash/python scripts are the common strategies used by teams here.

Level 2: Agentic investigation

This is the first level where the machine comes into play and does real work instead of being an NPC. For example, an alert fires, and an agent pulls the relevant logs, correlates the recent deploy, checks the downstream service, and hands you a first read on what's going on.

But you still have to decide and act. You don't start things from scratch - the agent compresses the first twenty minutes of every incident into something you can read in a few minutes, instead of manual assessment.

L2 Outcome:

When an L2 layer is successfully implemented, here's what teams benefit from it:

• Humans receive an alert along with detailed investigation - Agent identifies what triggered this issue, accurately and consistently.

• While humans might have skipped analysis or serialised analysis when there's a surge of alerts, here the agent can run parallel investigations and help get up to speed faster.

Note: A successful Level 2 autonomy layer in your software infrastructure means that the system can auto-identify the triggers accurately. "Accurately" is important because a system used in production infrastructure needs high reliability for engineers to trust in high stakes environment.

Pre-requisites:

While this investigation agent sounds simple early on, there are nuances that make it tricky to get to the end line:

• Agent with access to tools: As you retrospect over more and more production incidents, it's clear that only the logs and metrics don't tell the full picture. Teams often end up checking deployment configs (in CI/CD tools), code review (in GitHub repository), customer level data (in internal tools/databases), security/configs in cloud and more. For an agent to be able to pinpoint triggers better, it needs access similar to the access that a senior engineer might have.

• Context: Tribal knowledge, company infrastructure, and known patterns are a big giveaway when it comes to production troubleshooting. You need to setup a basic layer of Context for the agent to be able to operate well - it could begin, initially, with some architecture docs and some skills, but eventually as you try to iterate and improve the agent's accuracy so it becomes near accurate in every instance, it needs to be a continuously updating context map/knowledge graph about your infrastructure.

Ownership:

• At this stage, tickets & alerts are still assigned to humans. Dedicated engineers (on-call) are still looking at noisy alerts in Slack. The first responder on a Pager is still an engineer.

• The engineer does not have the full burden of doing the heavy lifting, but validating the AI's insight & recommendations.

• The builder of the AI Agents + service owner has the implicit responsibility to ensure that the Agent has the right context + right harness to avoid hallucinations, poor correlations.

Level 3: Cognitive decisioning

At Level 3, we are turning the tables with regards to ownership and workflows. Humans are not dedicatedly looking at Day-2 ops every now & then.

The agent doesn't just investigate an issue after it's identified as an alert or incident. It triages every signal, identifies if it's actually an issue, clusters it with other existing/ongoing issues and routes it to the right owner (with the root cause proposed).

Level 3 Outcome:

• The ownership of sifting through alerts & signals --> detecting issues and identifying the owners of the actual trigger of the issue are not done by on-call engineer, but by the Agent.

• Agents consume all alerts & signals, run all investigations and cluster them into actionable buckets. Humans review this on demand.

• The agent also has access to known fixes and can apply them without approval. It can also apply new fixes but needs peer (human) review, as is the norm with any process today.

Pre-Reqs:

• Root cause quality: If your agent isn't able to investigate well, it will have exponentially higher risk here because the failure would compound across multiple queries, and poor data can lead to partial assessments.

• Context Map: When an agent needs to be able to discover and navigate a given architecture, skills alone wouldn't help.

• Self-learning: The architecture upgrades every day. The agent's context map needs to be able to continuously update itself - learning from everyday changes, patterns and more.

The effect here is that it has what you can call a pseudo world model - where all the known components are known.

What makes this possible is the context graph, which I'd call a pseudo world model: a representation of services, dependencies, ownership, and normal behavior that lets the agent reason about consequences instead of pattern-matching on symptoms. It knows that restarting this pod drains that queue and that the queue feeds a job someone cares about.

The word "pseudo" is doing a lot of work here. The model is good enough to propose the right action most of the time, which is why the human role collapses to a single thing: approve. The agent does the thinking; you say yes or no. And nothing gets resolved on the agent's say-so alone, because the model is approximate, and approval is where a person catches the cases it gets wrong.

This is where on-call changes shape. The reason is specific, and it's the part most people get backwards.

And after level 3, you even don't need human on on-call

People go on call because someone has to be accountable for acknowledging an incident and seeing it through to resolution. That's the actual job.

Below level 3, that ownership can't be handed off. The machine can investigate, but it can't be trusted to act, so a human has to be reachable to make the call and give the fix.

Here, the shape inverts. The agent triages, routes, and proposes around the clock. A person isn't tagged for every incident anymore. They get pulled in for two cases: the agent got stuck, or it found something critical enough that it wants a human in the loop before making a decision. On-call stops being "wake up for every alert" and becomes "be reachable when the system asks for you."

That's the real reason to climb the ladder.

Level 4: Autonomous resolution

This is the highest level in AIOps. The system detects, decides, fixes, and closes the loop on its own; we can also say it is self-healing. Humans provide oversight, not sign-off, on each action.

The gap between level 3 and level 4 is just one word, and that changes the whole game: the model goes from pseudo to true.

A pseudo world model is a good map. A true world model is one which is accurate enough that the system can act on its own predictions without a person checking each one. The difference is trust, and trust here is earned through accuracy on the consequences that actually matter. At level 3, you approve because the model might be wrong. At level 4, you don't, because it's been right often enough, on the decisions that count, that supervision can replace approval.

That's a research problem, not a ship date. Building a model of a live distributed system that's accurate enough to trust with self-healing is hard for the same reason modeling is hard everywhere in AI; the map drifts from the territory the moment you finish drawing it, and a distributed system rewrites its own territory every deploy.

How We're Building This Future

At DrDroid, we don't think the journey to autonomous operations happens overnight.

The biggest gap today isn't better LLMs; it's reliable context. Production incidents require reasoning across monitoring systems, deployments, cloud infrastructure, code repositories, ownership information, historical incidents, and internal documentation. Without that context, AI agents remain assistants instead of operators.

That's why our focus has been on building AI SRE agents that combine deep infrastructure context with access to operational tooling. Today, those agents help engineering teams investigate incidents automatically, correlate signals across systems, identify likely root causes, and reduce the manual effort required during on-call.

We're also investing heavily in the next stage: creating a continuously evolving infrastructure context graph that enables higher levels of autonomy, including intelligent alert grouping, issue clustering, ownership discovery, and eventually AI-assisted operational decision making.

We don't believe autonomous operations arrive through one breakthrough model. They arrive by solving each layer of the maturity model with enough reliability that engineers naturally trust the next one.

So, where are the teams actually?

Most teams are still at ground level. A growing number are reaching for agentic investigation, and that reach is realistic today. Cognitive decisioning is the current frontier, where the context graph/knowledge graph is the thing separating the teams that talk about it from the teams shipping it, actually.

Autonomous resolution is where organisations wanted to go, but very few products are actually there today.

The maturity model isn't about measuring how "AI-powered" your operations are. It's about identifying the single capability that unlocks the next stage. Progress isn't made by adding another LLM or another dashboard, it's made by removing the biggest problem in your incident response workflow and that we already know.

This paper describes DrDroid's Context Engine: the continuously updated knowledge layer that captures the state and history of a customer's production environment, and the retrieval system that surfaces the right slice of that knowledge during an agent investigation. We cover what the Context Engine stores, how it is built and kept current, how the agent searches it, and the design choices that make this approach measurably better than RAG, embeddings, or static documentation alone.

The problem we are solving

Production engineering is not a single discipline. It is the orchestration of dozens of systems, each with its own data model, vocabulary, and failure mode. A typical mid-sized engineering team operates across:

• A monitoring stack (Grafana, Prometheus, Datadog, Signoz)

• A deployment pipeline (ArgoCD, Jenkins, GitHub Actions)

• A cloud or hybrid infrastructure provider (AWS, GCP, Azure)

• An orchestrator (Kubernetes)

• An alerting and on-call layer (PagerDuty, Opsgenie)

• A code platform (GitHub, GitLab)

• Collaboration and ticketing (Slack, Jira, Confluence)

• Databases, APMs, error trackers, analytics tools

When an alert fires, the engineer who resolves it fastest is not the one who reads the alert text most carefully. It is the one who carries an internal map of how these systems connect for this specific company. They know which dashboard is the real one. They know that payments-svc and checkout-payments are the same service under two names. They remember that a similar alert two weeks ago was caused by a Redis eviction, not a code change.

This map is built over months of on-call rotations. It is the difference between a five-minute resolution and a fifty-minute one. And until now, it has been almost impossible to give to an AI agent.

A foundation model knows what Kubernetes is. It does not know what your Kubernetes cluster looks like. RAG over your documentation helps a little, but production reality is not in your documentation. It is scattered across live tools, recent deploys, evolving log patterns, and engineer's heads.

The Context Engine is our solution to this gap. It is the knowledge layer that makes a general-purpose AI agent behave like a senior engineer who has been on your team for two years.

What the Context Engine is

A short definition first:

The Context Engine is a structured, continuously updated knowledge layer about a customer's production environment, designed to be queried by an AI agent during operational tasks.

It has two parts. The first is the knowledge itself: a catalog of what exists in the production environment, organized into typed records. The second is the retrieval system that sits on top: a hierarchical search engine, which we refer to as Dynamic Memory Retrieval (DMR), that lets the agent pull the right records at the right moment without dumping everything into a context window.

The analogy we find useful is a library. The knowledge layer is the collection of books on the shelves. DMR is the card catalog and the librarian. You need both. A library without organization is just a pile of paper. A search system with nothing to search is just an empty interface.

The Context Engine is a knowledge layer that sits between your production environment and the AI agent. The goal of the layer is simple to state and harder to build: before the agent answers a question about your system, it should already have the same picture in its head that your senior engineer does.

To get there, the engine continuously builds and maintains four kinds of context. I'll walk through each one, what's inside, and how we keep it updated without anyone on your team babysitting it.

1. The Map of Your Stack

When you connect DrDroid to your existing tools, the first thing the engine builds is an inventory. Not a CMDB you fill out by hand. An actual working map pulled directly from your APM, ArgoCD, GitHub, and Kubernetes.

This map covers every service with its repos and owning team. Every Grafana and Datadog dashboard, the panels inside them, the queries those panels run. Every database, VM, K8s cluster, and namespace. Every third-party vendor your application talks to.

How we keep it current: Each source tool gets its own synchronizer. ArgoCD changes get picked up on a webhook. Kubernetes is reconciled on a short interval against the live cluster state. GitHub activity flows in through repo events. We don't trust any single tool to be authoritative on its own, so the engine resolves conflicts using a precedence order per record type. The result is a map that drifts in minutes, not days.

What this changes: When an alert mentions checkout-api, the agent already knows that means the checkout-api repo on GitHub, owned by the Payments team, running in the prod-us-east cluster, monitored by the checkout-deepdive Grafana dashboard, depending on the orders database and Stripe. It doesn't have to ask. It doesn't have to query five tools to figure it out. It already has it.

2. How Your Services Actually Connect

Maps are static. Production isn't. The engine also builds a live correlation layer which has what depends on what, what traffic flows where.

This is where most "AI for ops" tools quietly fall apart, because reconstructing a service graph from a static config is one thing and keeping it accurate while services come and go is another.

3. Your Team's Tribal Knowledge

This is the part senior engineers care about most, because this is the part they're tired of being the only source of.

The engine stores runbooks and SOPs, synced from Confluence or Notion or uploaded directly. It stores every past investigation and RCA the agent has run, including what it queried, what it correlated, and what it concluded. It watches your alert patterns over time so it can tell a routine 7 AM spike from a real anomaly. It knows who's on the Payments team and who's on-call this week.

How we keep it current: Runbooks sync on a fixed schedule and on doc-update events where the source supports them. Investigations are written into the engine as the agent finishes them, with full traceability of which records they touched. Alert pattern fingerprints are recomputed on a rolling window, so a service that used to be quiet and is now noisy doesn't look "normal" forever just because it once was.

What this changes: A new engineer on-call doesn't need someone senior to tell them "the last three times this fired, it was a config change in service X." The agent says that, because the agent has been watching.

4. What Changed in the Last Hour

Most production incidents trace back to something that changed recently. This is the freshest layer of the engine and the one that matters most during a live incident.

It tracks code changes (every PR, commit, and merge across your repos). It tracks infrastructure diffs (new namespace, renamed cluster, database added). It tracks live alerts (what's firing now, what just stopped). And it polls 150+ vendor status pages, so the agent knows if Stripe or AWS is having a bad day before you do.

How we keep it current: This is the part of the engine that's almost entirely event-driven. PRs and deploys come in on webhooks. Alerts flow in from your alerting tool directly. Vendor status pages are polled aggressively, on the order of every few minutes. The "last hour" delta is recomputed continuously, not cached.

What this changes: The agent walks into an investigation with a delta of what's different in the last hour. Most of the time, the answer is sitting somewhere in that delta.

How retrieval actually works

Building the knowledge is one problem. Pulling the right slice of it at the right time is another. This is the part where most teams trying to build something similar quietly run into a wall.

When we started, the obvious approach was retrieval-augmented generation. Take everything we know about a customer's environment, chunk it, embed it, do similarity search. It's the standard pattern for AI applications, and it works well for general-purpose knowledge bases.

It does not work for production infrastructure.

The reason took us some time to articulate clearly. In production engineering, a single character can mean a completely different thing.

• us-east-1 and us-east-2 are different regions

• checkout-prod and checkout-staging are different services

• pod-abc123 and pod-abc124 are different pods

• v1.4.2 and v1.4.3 might behave totally differently

Embedding models, by design, map semantically similar strings to nearby vectors. They will tell you these pairs are 99% similar. They are not. During an incident, confusing them takes a small problem and turns it into a much bigger one.

The other issue is that production names don't carry semantic content. A service name is an arbitrary identifier. A cluster name is a label. There's nothing for the embedding to "understand." There's just the exact string. Keyword search works much better here, because real production questions almost always anchor on a specific named entity, and that name appears verbatim somewhere in the right record.

So our retrieval layer (we call it Dynamic Memory Retrieval) is keyword-first against the structured graph. We index entities for exact and prefix matching first. We layer embeddings in only for the record types where semantic similarity genuinely helps: runbook prose, ticket descriptions, Slack threads, postmortems. Never as the default for infrastructure records.

A few other design choices worth flagging:

Hierarchical results. A service ranks above its log lines. A dashboard ranks above its individual panels. A runbook ranks above a casual Slack mention. The agent gets the parent record first and drills into children only when it needs to.

Multi-pass retrieval. A single search rarely surfaces everything an investigation needs. The agent runs successive queries, each refined by what the previous one returned. A typical end-to-end investigation does between 5 and 50 retrievals against the engine, interleaved with tool calls against the underlying systems.

Short-term and long-term memory, kept separate. Long-term records describe what's always true: the list of services, the structure of the codebase, the dashboards that exist. Short-term records describe what's currently true: the deploy from 40 minutes ago, the alert firing right now, today's open incident. Mixing them confuses the agent's sense of "now" versus "always." So they're stored and queried as distinct stores.

How the engine improves over time

The engine isn't static. Three things keep happening in the background.

It learns from every investigation. Each RCA the agent produces becomes a record in long-term memory. The next time something looks similar, the agent surfaces it: "This looks like the incident on March 14th where the Redis eviction policy was the cause."

It tracks your environment as it changes. New services, deleted dashboards, renamed clusters get picked up automatically. You don't re-onboard, ever. The engine that handled your environment on day 1 is materially different from the one handling it on day 90, in a good way.

It stays auditable. Every recommendation the agent makes is traceable back to which records it pulled and which tool calls it ran. If a senior engineer doesn't trust an answer, they can walk back through the reasoning step by step. This matters more than it sounds like it should, because trust is what determines whether an AI agent actually gets used.

One of our customers described the practical impact like this: "We went from a 90-day onboarding window for new SREs to two weeks." That's the bet, institutional knowledge living in a system instead of in three senior engineers' heads.

Conclusion

If you're building or evaluating an AI agent for production work, the single most important question is not which model it's using. It's what context the agent walks into the room with.

A foundation model with MCP servers and no context engine is going to spend the first ten tool calls of every investigation re-learning your environment, and it's going to get half of it wrong because production names look semantically similar but mean entirely different things.

A foundation model on top of a continuously maintained knowledge layer of your services, dashboards, runbooks, dependencies, and recent changes will start every investigation already pointed in roughly the right direction.

That's the whole architectural argument behind the Context Engine. Models will keep getting better. The work of building, maintaining, and retrieving the right context is the part that compounds, and it's the part most teams underestimate.

To understand more about content layer and knowledge layer, check out the youtube videos from @drdroiddev channel

If you want to see the engine running against your stack, schedule a technical evaluation.

That gap is the whole problem. So recently we found the solution for it. DrDroid now has an MCP server, and it puts the production context we've already built for your stack, alerts, traces, past incidents, service relationships, infrastructure metadata, directly inside Claude Code or any MCP-compatible IDE.

If you're already running DrDroid, here's how it works.

What's the need for it?

You know the problem. Latency triples, the failing trace is open in your editor, and the fix is probably one line. But the line you need depends on what you can't see from the trace alone. The alert that fired four minutes ago. The deploy that went out at 14:32. The incident from three weeks back with the same signature.

DrDroid already knows all of that. It's in the knowledge graph. The problem was that the graph lived on DrDroid's side and your editor lived on yours, so you'd jump out of the IDE to go get it. The MCP server removes the layer now. The context graph you already rely on now answers from inside the editor where you're writing the fix.

How it work

Connect your IDE and you're querying the context graph directly. On the other end:

• Alerts: current and historical, across every monitoring platform you've connected

• Traces: distributed tracing data from your services

• Past incidents: the full investigation history and root cause analysis, not just that something broke but how it was fixed

• Service relationships: dependencies, upstreams, downstreams

• Infrastructure metadata: Kubernetes, databases, connector configuration

Past incidents is the one worth calling out. Most tooling tells you what's broken now. DrDroid tells you this exact problem happened on March 14th and here's what resolved it. That history is the difference between an agent that hands you telemetry and one that hands you the answer.

The Architecture

Lets understand that your IDE is the client. DrDroid is the server and they talk over SSE.

The server pulls your production context into the editor. Nothing mutates through it. The agent reads your incident history. It does not get to restart your pods. For a tool sitting one query away from production, that boundary is the point.

Because it runs against your DrDroid instance, the same access you already have governs what the MCP server can see. It doesn't need any new surface or any separate permission to reason about. What it looks like in practice

You're in Claude Code with a traces open. You ask, in the same session:

What happened with payment-service last night?

The server returns, inline: the alerts that fired in the last 24 hours, the related deployments, the past incident investigations that match, and the service dependency graph. No alt-tab. No reassembly. The answer arrives next to the code you're about to change.

Each capability is a tool the agent invokes on its own during an investigation, or that you call directly.

The CI/CD set, for example:

list_pipelines()              # discover CI/CD pipelines
get_pipeline_runs()           # recent executions
get_run_logs()                # detailed logs from a run
get_deployments()             # deployment status
get_pipeline_health_summary() # reliability trends over time

Conclusion

If you're a using Drdroid, point Claude Code at your Drdroid /mcp endpoint and the knowledge graph you've been building is live in your editor or you can build your own skills. Open a claude code session, ask it about last night's incident, and watch it pull the alerts, the deploys, and the matching past incident without you touching another tab.

The next time something breaks at 2 AM, you will not be looking for the context. It will already be where your cursor is.

“Production systems” is a not a single component.

Kubernetes is a tangibly identifiable component with deterministic ways of operation. So is a database. And so is code. But “Production Systems” is a complex orchestration of different and discrete components that interact with each other to produce desired results for a business or user.

Joining together pieces, some of which are continuously changing, means that mathematically, the risk of failure is non-linearly higher than the risk of the individual components.

To tackle that, it is quintessential to design, iterate and architect systems with resilience, fallbacks, reliability and visibility.

DrDroid helps engineers leverage AI effectively for automating and accelerating operations. To enable high quality results, DrDroid has a stateful knowledge layer (context engine) about different components of production systems for agents.

The need for context engine

Engineers are using DrDroid to debug production alerts and get sharp data backed RCAs. To ensure that the agent is able to run complex investigations, is grounded in data and is context aware, we realised that a lot of reality is hidden in the day-to-day changes in the system. Here a few examples of typical changes, in sequence of most obvious to less obvious:

• Code Changes: Regular changes in a service in accordance with deliverables and goals

• Resource changes: A database table gets it’s memory upto the limit

• Indirect code changes: A third party dependency being used downstream in a service changes their API payload

• Indirect resource changes: A certificate expires on one of the machines where a critical service is running

• User change: A different user triggers an edge case in a product journey that never got used until now, leading to a failure for the user

None of these issues are unheard of, or shockingly new to an engineer. The problem is that the impact could be visible at a very different point in the system compared to where the issue originated.

In fact, for very different reasons, the same outcome/impact/alert might get triggered at different times.

Being able to mitigate and avoid such issues often require more information than just the alert. This debugging is often a combination of system knowledge, deductive reasoning, tribal knowledge, specialised skills and real-time situational context.

Added to that, engineers are expected to do this in high pressure situations or unexpected times as there are often business guarantees associated with these issues.

Components within a context engine

Our context engine is designed to proactively process and maintain an update knowledge layer about a company.

Here are some of the examples of what the context engine comprises of:

1. System Knowledge:

   

   1. Service Catalog: A list of every service, it's related teams and repositories, monitoring context and dependencies. This can auto-generated by fetching data across multiple tools like an APM or ArgoCD or Github repositories or Kubernetes. You can further edit the details if needed.

      

   2. Infrastructure inventory: A list of all the resources, from Databases to VMs to kubernetes deployments and namespaces.

   3. Pre-instrumented metrics knowledge: The context engine is aware of all the metrics that are instrumented, as well as the dashboards and it's panels. This ensures that agent does not have to query prometheus by first principle but can re-use your existing dashboards too!

   4. Codebase context: The context engine continuously maintains knowledge of the capabilities and features within each repo, their structure. It additionally also stores correlations with other repositories (depending on the information available). Read more on source code security.

   5. Third party dependencies: Within DrDroid, you can maintain a list of all 3rd parties that your application depends upon.

      

2. Assistance for deductive reasoning:

   1. Service & infrastructure correlations: By leveraging the context of your inventory across infrastructure & services, and your traces/service maps, correlations between services and infrastructure is auto-processed. This can be built using your existing service maps from APM / traces or using network mappers in your kubernetes infrastructure (available with our reverse proxy service)

https://www.youtube.com/watch?v=Bt-G0ZRaj-8&t=2s

2.  **Pre-compiled log patterns and fingerprints:** Your log patterns are overtime stored within the context engine, ensuring that the agent can re-use the past experience of log analysis to improve the future log search queries.
    
3.  **Pre-mapped data flows and critical user journeys:** The agent maintains context about the product/architecture and can even keep context of critical workflows across repositories. Additionally, these can be edited to add additional insights and knowledge.
    

1. Tribal Knowledge:

   1. Pre-existing runbooks and SOPs: All your existing runbooks and SOPs can be added to the context engine, either via syncing it with your wiki or uploading/creating documents in the account.

   2. Past Incidents & investigations: The context engine is continuously updated as the agent runs alert investigations and RCAs. Insights of correlations and issues identified are stored for easy referencing and searching for the agent in the future.

   3. Alert Patterns: The context engine stores alerts from Slack channels and webhooks, ensuring the agent can see the pattern of a specific alert and differentiate an anomaly from a regular alert.

   4. Communication rosters: The context engine stores information about the teams and the users within the teams. Users can be auto-synced from Slack where as the teams can be simply configured in the platform.

2. Specialised skills

   1. Internal tools specific knowledge: You can store guidelines about how your internal CLI / tools operate. This would enable the agent to use the tools effectively, ensuring your production workflows can be replicated here.

   2. Best practices on trace analysis: Analysing traces can often be taxing for any engineer and require deep focus to identify the anomaly in them, across the time and spans. The context engine has specialised tools for trace analysis, enabling it to get insights on large volume of spans in a short duration with reduced token consumption.

   3. Observability tools usage: Every tool has it's own data structures and entity design. Best practices around using tools like Signoz, NewRelic, AWS, Kubernetes, etc. are pre-baked into the context engine.

3. Real-time situational context

   

   1. Code Changes: The context engine continuously processes PRs, commits and merges across repositories. This ensures that the agent can get a quick glance on recent changes.

   2. Infrastructure Changes: Addition of inventory in the infrastructure, from a database in your cloud to a namespace in your kubernetes cluster, the diff is continuously tracked.

   3. Ongoing issues and alerts: Alerts are processed and saved in real-time for the agent to use in real-time.

   4. Vendor downtime events: The context engine is continuously querying the status pages of the third party vendors identified in the platform. This ensures that the agent can be notified upfront during an investigation if a downstream vendor API is impacted.

Where is the context engine used?

How does the context engine get used in the platform:

1. Investigation Agent: The investigation agent is our flagship agent which can help you investigate any issue in your production system - from alert investigation & service degradation to security review & cost analysis.

2. Alert Classification/Triaging Agent: Just send all your alerts to DrDroid. When you receive a large stream of alerts, this agent can classify your alert to be a noisy alert (and suppress it) or let that alert come through to you. It grounds every recommendation in actual system data. As you get started, it gives benefit of doubt to an alert being critical (so that a critical issue isn't missed) but over time, it learns from already triaged alerts, data queried and user feedback received. Users can also add specific "notes" at alert type level so that agent doesn't have to think from first principles when it gets started.

3. Alert Grouping Agent: If you have 100+ relevant alerts coming in, rarely are they 100 unique issues. More often than not, these might end up becoming 7 or 19 issues. The Grouping agent listens to your alerts and tries to group it into actionable buckets - it often does it by root cause (found by investigation agent) or by the impacted component / where it fits in the topology. This could ensure that during an incident or when you get a surged stream of alerts where most of them are due to the same issue, the one other signal doesn't get missed.

4. Continuous Improvement Agent: The core principles behind engineering operations is to react, learn and improve. Some teams often get too occupied with firefighting and do not get the bandwidth to work on proactive improvements of the system. This agent gives suggestions for improving your alerts, missing observability data for any of your new services, gives cost or security posture recommendations.

5. Meta agent: This agent helps you to improve context on the DrDroid platform based on ongoing activity, which will in-turn, make all the other agents more powerful.

How is the context engine implemented?

Read more about the design of the context engine and it's architecture in this blog.

1. Specialized Debugging Tools & Skills

Production incidents often require analyzing large volumes of logs, traces, and metrics. This can be token-intensive and time-consuming. Most LLMs and agentic frameworks hit context window limits quickly, can't process production-scale data, and lose quality when analyzing large datasets.

How DrDroid Solves This:

Pre-Built Aggregate Analysis Tools

Instead of feeding raw logs into an LLM, DrDroid has tools designed specifically for handling large-volume logs with:

• Built-in log aggregation and pattern detection

• Complex trace analysis across distributed systems

• Large-volume metrics analysis with outlier detection and ML techniques

Specialized Investigation Skills

Our agent has domain-specific skills built from working with hundreds of engineers on real debugging problems:

• How to query and analyze traces in Signoz or Datadog

• How to navigate APM data efficiently

• How to correlate metrics across multiple monitoring tools

Real Impact: The agent can process 100,000+ log lines in seconds and surface the 5 relevant errors—something that would exhaust a generic LLM's context window.

2. Code & Application Awareness

Most LLMs and agents start every investigation from zero, with only the context of the prompt and sometimes markdown files.

How DrDroid Solves This:

Automatic Code Context Generation

Even before your first chat with the agent, DrDroid builds knowledge of:

• What each repository does

• What capabilities, APIs, features, and workflows each repo covers

• Programming languages, frameworks, and file structures

• Connections between multiple repositories (discovered via traces and logs)

Business Workflow Understanding

You can ask DrDroid to build context around critical business and product workflows. The agent understands:

• "The checkout flow involves payment-service, inventory-service, and notification-service"

• "When users report 'payment stuck,' check these three services in this order"

Real Impact: When an alert fires on "payment-service," DrDroid already knows what that service does, which other services depend on it, and where to look for root causes.

3. Infrastructure & Resource Awareness

An LLM with MCP connections doesn't know which apps run in which Kubernetes clusters, which databases are in which cloud providers, or how your infrastructure is organised. It needs to query multiple tools (costing time and token) and explore it before it's able to answer that.

How DrDroid Solves This:

Auto-Discovery of Infrastructure

DrDroid continuously maps:

• Apps hosted in different Kubernetes clusters

• Databases and their cloud providers

• Service dependencies and communication patterns

• Network topology and resource relationships

Service Map & Dependency Graph

The agent can answer questions like:

• "Which services depend on the payments database?"

• "If eu-west-1 goes down, what's affected?"

• "Show me all services running in the production cluster"

Real Impact: During an incident, the agent instantly knows the blast radius and which downstream services might be affected—without you having to explain your architecture.

4. Past Alert & Incident Pattern Recognition

With generic agents, every investigation is independent. It has no memory of past incidents or patterns.

How DrDroid Solves This: Searchable Alert History

The agent has access to:

• All alerts since platform enablement

• Past incidents and their resolutions

• RCAs and postmortems (from Confluence, docs, or previous investigations)

• Understanding patterns in alerts

When similar issues occur, the agent can say:

• "This looks similar to the incident from Jan 15th where the Redis cache was full"

• "Last time this alert fired, the root cause was a config change in service X"

Real Impact: Repeat incidents get resolved faster because the agent learns from past investigations.

5. Continually Learning System

DrDroid improves with every investigation.

Active Learning from Your Environment

The agent continuously creates notes and memory from:

• Recent commits and merges in your applications

• Investigations and conversations with the agent

• Human conversations in Slack channels (optional)

Contextual Memory Storage

Everything is stored with metadata:

• Timestamp

• Related entities (services, databases, clusters)

• Related team and people

• Relevant tags and categories

Real Impact: The agent gets smarter every week. After a month, it knows your environment better than most new engineers.

6. Context Compaction (1M+ Token Conversations)

Typically, agents do the following with the problem of large context windows:

• Summarize the entire conversation (losing critical context) or

• Hit token limits and can't continue

• Slow down dramatically as conversations grow

With production telemetry data, this can often happen.

How DrDroid Solves This:

Intelligent Compression Without Context Loss

• Tool calls are compressed (only IDs and summaries preserved)

• Reasoning and train of thought remain intact (no summarization)

• Agent maintains full context even beyond 1M tokens

• Smart Tool-Level Compaction

Our tools have built-in context management:

• Logging tool has grep/search capability over large volumes

• Agent can "eyeball and search" logs instead of loading everything into context

Real Impact: You can have a 2-hour debugging session with 500+ tool calls, and the agent never loses context or slows down.

7. Multi-Channel Conversations with Shareability

The agent is designed to work from your place of convenience:

• Slack DMs

• Thread replies to alerts

• Web UI

• CLI (coming soon)

• API triggers

• Voice calls (coming soon)

Seamless Sharing

Any investigation can be:

• Shared with teammates for review

• Linked in postmortems

• Referenced in future incidents

Real Impact: When someone gets paged, they can see the auto-investigation that already ran in the Slack thread—no need to DM the agent separately.

8. Automated Investigations

DrDroid can run proactively or via automated triggers enabling proactive visibility for your team:

• Alert fires in PagerDuty/OpsGenie → Investigation starts automatically

• Cron-based health checks → Agent investigates on schedule

• Custom triggers via API or webhooks

Real Impact: Agent can detect issues even without alerts; By the time you open the alert, the agent has already investigated and summarized the likely root cause.

9. Smart Model Switching (85% Cost Savings)

LLMs have been commoditised and the SOTA model is not necessarily required for every investigation. DrDroid smartly chooses between different LLMs based on investigation complexity

• Simple tasks → Faster, cheaper models

• Complex reasoning → State-of-the-art models

Real Impact: Up to 85% token savings compared to always using frontier models, with no degradation in investigation quality.

10. Dedicated File System & Memory

Memory management for a large scale infrastructure requires a structured approach.

DrDroid has a Persistent Knowledge Base - All context, memory, investigations, and alerts are stored and accessible:

• Agent can navigate past investigations like files

• Search across all historical data

• Reference previous findings instantly

Real Impact: "Show me all investigations related to database timeouts in the last 30 days" returns instant results.

11. Coding Sub-Agent for Hotfixes

Coding agent operates very different from a production investigation agent. DrDroid comes pre-packaged with a coding agent connected to the investigation agent.

Real-Time Coding Agent When needed:

• Spins up a coding agent in an ephemeral sandbox

• Reviews the full repository

• Creates hotfix PRs with proper context

Real Impact: During an incident, the agent can say "I found the bug in payment-service line 247—here's a PR to fix it."

12. Remote Machine & Kubernetes Access

Often, data needs to be reviewed on a remote machine or kubernetes cluster for production incidents. These might be inaccessible or sensitive.

How DrDroid Solves This: Direct Infrastructure Access without token access to the agent

• Execute commands on remote machines via SSH (keys are not exposed to the agent)

• Query read-only Kubernetes clusters directly

• Access VMs and clusters within your VPC via reverse proxy

Real Impact: "Check disk space on prod-api-01" → Agent SSHs in, runs the command, and returns results. No manual execution needed.

13. Image Support for Dashboard Analysis

You might want to debug an issue with a screenshot shared by the customer as the starting point.

How DrDroid Solves This: DrDroid agent support image processing from Slack or UI.

• Your product showing an error

• A Grafana dashboard

• A monitoring alert

The agent analyses it and continues the investigation from there.

Real Impact: "Here's what the user is seeing" → Agent understands the UI issue and investigates the backend cause.

14. Granular Access Control & RBAC

Production systems debugging come with sensitive data and access management. DrDroid ensures only the right people have the right access while debugging.

How DrDroid Solves This:

• Read commands: Execute without approval (safe exploration)

• Write commands: Require RBAC approval per your policy

• SSO integration: Syncs with your internal permissions

• Audit logs: Track who did what

Real Impact: Junior engineers can investigate safely, while dangerous operations require senior approval.

15. Third-Party Vendor Status Tracking

Often production incidents can be partly caused due to 3rd party downtimes or issues.

How DrDroid Solves This: Connected to Vendor Statuspages

• Tracks 150+ status pages for your third-party vendors: Stripe, AWS, Datadog, MongoDB Atlas, etc.

• Flags when vendor issues might be causing downstream impact

Real Impact: "Is this our issue or Stripe's?" → Agent checks Stripe's status page and correlates timing.

16. Automated Quality Evaluation

Production Agents need to come with quality guarantees for the team to track and trust.

How DrDroid Solves This: LLM-Based Evals on Every Investigation

Every investigation is automatically evaluated for:

• Accuracy Safety Errors or hallucinations

• Central teams get visibility into investigation quality and improvement opportunities.

Real Impact: Platform team can see "Investigation quality is 94% this month, down from 97% last month—let's review the low-scoring investigations."

17. User Feedback & Team Visibility

Context within DrDroid can continuously improve over time. But for that to improve, tracking and acting upon user feedback is critical.

How DrDroid Solves This: Collaborative Quality Control

• Every investigation can be upvoted/downvoted

• Feedback back to central team helps improve agent context

Real Impact: Central team & managers have visibility on confidence and impact of AI on the engineers.

18. Reasoning Lifecycle & Audit Trail

Production incident investigations cannot be led to be incorrect due to "hallucinations" or "guesses" by an LLM. DrDroid ensures that every reasoning and logic by the LLM is grounded in facts and data.

How DrDroid Solves This: Transparent Investigation Path

The agent tracks:

• What data it queried

• Why each data point was relevant

• What hypothesis it built from each finding

• How it reached its conclusion

Real Impact: You can backtrack through the investigation to validate correctness, spot gaps, or understand the agent's reasoning.

Summary: Why DrDroid is Purpose-Built for Production

What This Means for Your Team:

• Every investigation starts with past context

• You do not have to guide the LLM or explain your architecture every time

• It can handle production-scale logs or metrics

• It supports automation or proactive help

• It works across different channels where your team lives

Ready to See the agent?

DrDroid is a purpose-built investigation agent that understands your infrastructure, learns from your incidents, and gets smarter every day.

Next steps:

• Watch platform demo videos

• Check our MCP Servers & integrations

• Read the documentation

• See customer case studies

Want to see how it works with your stack?

Setup & go live takes 1-2 hours for smaller teams, < 1 week for enterprises. Get started here.

Something is broken right now. You need answers fast.

Day 1 — Operational Tasks & Maintenance

No fire, but you need to keep things running smoothly.

Day 2 — Automation, Optimization & Reliability

You want to stop doing repetitive things and build resilience.

What's Next?

DrDroid helps your team move from reactive firefighting to proactive operations. Whether you're debugging an incident at 3 AM or building automation to prevent the next one, DrDroid becomes your team's operational co-pilot.

Ready to see how DrDroid works with your stack?

• Watch platform demo videos

• Check our integrations

• Read the documentation

• See customer case studies

Want to try it out? Setup takes 1-2 hours and you'll see value from your first investigation. Get started here.

By the end of this DIY guide, you’ll have:

1. A Slack Bot

2. A backend that can

   • Listen to user messages or alerts in a channel and take agentic action based on prompts or steps that you might have in mind.

   • Query your Grafana instance, analyse logs/dashboards and send info about anomaly in reply to an alert/message in your Slack channel

Pre-requisites

1. Python uv (package manager)

2. Ngrok to expose the slackbot server. Setup instructions

3. Grafana (Optional)

Step 0: Clone the repo:

Repository Link - https://github.com/DrDroidLab/slack-ai-bot-builder

Step 1: Building the Slack Bot with an integrated backend

The backend repo setup here, behaves in multiple ways:

• Acts as an MCP Client for any AI calls you might want to make

• Acts as a server to accept webhooks from Slack and manage configurations

Setting up the Ngrok Tunnel

Expose port 5000 of your localhost, using the command:

ngrok http 5000

You will receive a HTTPS URL of the form https://abc123.ngrok.io

Which is pointing to port 5000 of your system.

Note: We have not setup a server running on port 5000 yet, but that is fine since ngrok is independent of that, and exposes the port regardless.

Creating the Slack Application

1. Go to Slack API Apps – https://api.slack.com/apps

2. Click on Create App, and select the option ‘From a manifest’

3. Copy the manifest json from the repository and replace the placeholder ‘<hostname>’ with the HTTPS URL you got from ngrok. Include the “https://” part as well.

4. Install the app in your workspace.

5. Copy the credentials for the slack application into credentials.yaml

   1. app_id, app_name and signing secret can be found in the basic information tab.

   2. Bot-auth-token can be found in the Oauth & Permissions tab.

   3. openai_key from openai - if you plan to use AI based workflows.

6. Create a channel called #drdroid-slack-bot-tester in your slack workspace & add the bot to the channel.

Setting up the bot server

Run the following commands to set up your virtual environment and activate it.

uv env venv
source .venv/bin/activate

Install dependencies using:

uv sync

Now we can finally run the bot server using:

uv run python app.py

The server is now running on port 5000, and exposed to the outside world via your ngrok tunnel.

Testing the bot

Add the bot to the drdroid-slack-bot-tester workspace that you had previously created.

And just type in a ‘hi’. The bot should send you a sample response.

Step 2: Integrating AI

There is already an example workflow for AI (name: "chatbot") in the workflows.yaml. It is just boilerplate code, you can modify it as required. For now, you can tag your bot in the drdroid-slack-bot-tester, and chat with it.

• Add your OpenAI/LLM key

For example:

Message in Slack: chatbot How to debug Kubernetes CrashLoopBackOff error? @bot
Message in Slack: chatbot I'm getting this alert. What does it mean? @bot

Step 3: Making the bot an Agent by giving AI access to different tools (Grafana for demo)

MCP Servers help abstract out any API and making them accessible to AI.

Setting up Grafana MCP Server

Clone the following repository:

Repository URL - https://github.com/DrDroidLab/grafana-mcp-server

Navigate into the root directory of the repository.

Populate the src/grafana_mcp_server/config.yaml with your grafana credentials.

Install and setup the dependencies using:

uv venv .venv
source .venv/bin/activate
uv sync

Run the MCP server:

uv run -m src.grafana_mcp_server.mcp_server

Your MCP server is now running on port 8000.

Creating an AI Grafana workflow in slack-bot-builder

There is already an example workflow for grafana ai in the workflows.yaml.

We will be running the script scripts/grafana_ai_tool.py in this workflow, it is just boilerplate code, you can modify it as required.
Now you can tag your bot in the drdroid-slack-bot-tester, and ask it to do various things from grafana.
For example:

Message in Slack: Fetch me logs from the currencyservice in grafana ai. 
Message in Slack: Fetch and analyse the Go Microservices dashboard from grafana ai

Next Steps:

Now that you’ve been able to setup a bot, here are a few things you can do:

• Productionise it from your current ngrok setup to a static endpoint

• Integrate with Grafana or open source MCP servers of your favourite tool that you want to leverage for automation

• Add custom prompts and scripts

Stuck anywhere? Ask on our Discord

You log into Grafana, click through three nested menus, find the alert, and bump the threshold from 80% to 85%. Crisis averted. You go back to bed.

Two weeks later, during a postmortem, someone asks: "Who changed the CPU alert threshold? And why?"

Silence. Nobody remembers. There's no history. No context. No way to know if this was a temporary hack or a deliberate tuning decision. Worse, when you refresh your staging environment, the old threshold returns because the change only lived in the production UI.

Sound familiar? You're not alone. This is how most teams manage alerts—and it's fundamentally broken.

Why Managing Alerts in Dashboards Doesn't Scale

We've spent the last decade moving infrastructure to code. Terraform for cloud resources. Helm charts for Kubernetes. Ansible for configuration. Yet somehow, our alert rules—critical infrastructure that wakes up engineers—still live in UI dashboards like it's 2010.

The problems compound quickly:

No version history: When did this alert last change? Who changed it? Why? Your Grafana dashboard shrugs.

No peer review: A junior engineer can accidentally change a critical alert threshold with zero oversight. Try doing that with production code.

No rollback capability: That "quick fix" that made things worse? Good luck remembering the old values.

Environment drift: Production alerts diverge from staging. Dev environments have different rules. Chaos ensues.

No ownership tracking: Who owns this alert? Which team should review changes? The UI doesn't care.

Your infrastructure evolves constantly. Services scale. Traffic patterns shift. Performance characteristics change. But alerts configured through dashboards remain frozen in time, slowly becoming less relevant until they're just noise.

Here's the thing: alert rules are infrastructure-as-code too. They define critical system behavior. They impact your team's quality of life. They deserve the same rigor as any other code.

Enter GitOps for alerts—where alert definitions live in version control, changes happen through pull requests, and every modification is tracked, reviewed, and reversible.

What is GitOps for Alerting?

GitOps for alerting is beautifully simple: store your alert rules as code in Git, manage changes through pull requests, and deploy automatically. Just like any other infrastructure.

Most modern monitoring tools already support this:

• Prometheus: Alert rules in YAML files

• Alertmanager: Routing configuration as code

• Grafana: Alerts exportable as JSON

• Datadog: Monitors manageable via Terraform

• New Relic: Alerts configurable through their API/Terraform

Here's what a typical structure looks like:

/alerts/
  frontend-service.yaml
  database.yaml
  redis.yaml

/teams/
  payments/
    api-alerts.yaml
    database-alerts.yaml
  platform/
    infrastructure-alerts.yaml
    kubernetes-alerts.yaml

A Prometheus alert rule might look like:

groups:
  - name: frontend-service
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
          service: frontend
          team: frontend-team
        annotations:
          summary: "High error rate on {{ $labels.instance }}"
          description: "Error rate is {{ $value }} (threshold 0.05)"
          runbook: "https://wiki.company.com/runbooks/frontend-errors"
          owner: "frontend-oncall@company.com"
The benefits are immediate:

✅ Traceability: Every change is a commit. Git blame tells you who changed what and when.

✅ Peer review: Alert changes go through PR reviews. No more accidental 3 AM threshold adjustments.

✅ Consistency: Deploy the same alerts across all environments. No more production/staging drift.

✅ Rollback capability: Bad change? git revert and you're back to working alerts.

✅ Documentation: PR descriptions explain why changes were made. Context is preserved forever.

But GitOps Alone Isn't Enough

Here's the plot twist: GitOps for alerts solves the how but not the what.

You now have beautiful, version-controlled alert rules. Every change is reviewed and tracked. But you still don't know which rules need updating.

Your Git repo becomes a graveyard of alert rules that might or might not be relevant:

• That CPU alert from 2019 when you ran on smaller instances

• The memory warning tuned for your old Java app (you've since moved to Go)

• The latency threshold set when you had 100 users (you now have 10,000)

You've traded one problem for another. Instead of stale alerts in dashboards, you have stale alerts in Git. They're better organized, sure, but still noisy.

This is where most GitOps alerting stories end. Teams implement the framework but lack the feedback loop to keep it healthy. Alert rules accumulate like sediment. Engineers suffer in silence because "at least it's in Git now."

Using Alert Insights to Drive GitOps Changes

Let real alert data guide your pull requests

The missing piece is data. You need to know which alerts are actually problematic before you can fix them. This is where DrDroid's Alert Insights transforms GitOps from a theoretical improvement into a practical solution.

Alert Insights analyzes your live production alerts and tells you:

• Which alerts fired most frequently last week: Your noisiest offenders, ranked

• Which alerts were ignored: Clear signal of rules that need removal

• Which alerts lack owners or runbooks: Quality issues to address

• Suggested changes: Specific recommendations to mute, tweak, or archive

Now GitOps becomes powerful. You're not guessing which alert rules to update—you have data.

✅ Workflow Example:

Monday: Run Alert Insights

`Top 3 Noisy Alerts:

1. redis_memory_warning - 127 fires, 0 actions taken

2. api_latency_high - 89 fires, acknowledged but not investigated

3. cpu_usage_critical - 45 fires, all during deploy windows`

Tuesday: Create targeted PRs

git checkout -b fix/reduce-redis-memory-noise
# Edit alerts/redis.yaml
# Increase threshold from 70% to 80% based on actual usage patterns
git commit -m "Increase Redis memory threshold to reduce false positives

Alert Insights showed 127 fires with 0 actions last week. Analysis shows Redis memory naturally spikes to 75% during cache warmup."

Wednesday: Review and merge

• Team reviews the PR

• Links to Alert Insights data provide context

• Changes deploy automatically

Thursday: Validate impact

• Alert noise drops immediately

• Next week's Alert Insights confirms improvement

The feedback loop is complete. You're not just organizing alerts better—you're systematically improving them based on real data.

➡️ 🛠️ Want a GitOps-ready alert audit? 👉 Run DrDroid's Alert Insights and get actionable suggestions in minutes.

Recommended Practices for GitOps Alert Management

🔍 Use clear filenames per service/component

Don't create a monolithic alerts.yaml. Break rules into logical groups:

/alerts/
  services/
    payment-api.yaml
    user-service.yaml
  infrastructure/
    kubernetes-nodes.yaml
    database-cluster.yaml
  business/
    checkout-flow.yaml
    user-engagement.yaml

🔄 Add labels/tags to help Alert Insights map alerts to owners

Every alert should include:

yaml

labels: team: payments service: payment-api environment: production severity: P2

This metadata powers Alert Insights' analysis and recommendations.

🧪 Validate rules with test alerts in staging

Before merging, trigger test conditions:

# Simulate high error rate
curl -X POST http://prometheus:9090/api/v1/series \
-d 'match[]=up{job="frontend"}'
🔁 Link PRs to weekly alert review

Common GitOps Pitfalls to Avoid

❌ Bulk silencing alerts without context

"Let's just comment out all the noisy alerts" is tempting but dangerous. Use Alert Insights to understand why alerts are noisy before acting.

❌ Committing rules without reviews

The whole point of GitOps is peer review. Don't bypass it with direct commits, even for "quick fixes."

❌ No tagging = Alert Insights can't map alerts to services

Without proper labels, you lose the ability to analyze alerts by team, service, or severity. Enforce tagging standards.

❌ Alert rules diverging across environments

Use templating to keep staging and production alerts synchronized:

# values-prod.yaml
cpu_threshold: 80
memory_threshold: 85

# values-staging.yaml
cpu_threshold: 90 # Higher tolerance in staging
memory_threshold: 90

Final Take — Let Data Drive Your Alert Rule Changes

GitOps gives you the framework for managing alerts professionally. Version control, peer review, and rollback capabilities bring alerts into the modern era.

But framework without data is just organized chaos. You need to know which alerts to fix, how to fix them, and whether your fixes worked.

Alert Insights provides that missing data layer. It tells you which alert rules are hurting your team, suggests specific improvements, and validates that your changes actually reduced noise.

Together, they create a powerful feedback loop:

1. Alert Insights identifies problematic alerts

2. GitOps enables reviewed, tracked changes

3. Automated deployment ensures consistency

4. Next week's Alert Insights validates improvement

This isn't theoretical. Teams using this approach report 50-70% reduction in alert noise within weeks. On-call engineers sleep better. Real incidents get proper attention. Alert quality becomes a measurable, improvable metric.

Your alerts deserve the same engineering rigor as your code. GitOps provides the foundation. Alert Insights provides the intelligence. Together, they transform alerting from a necessary evil into a competitive advantage.

➡️ ✍️ Want to make smarter, reviewable changes to your alerts? 👉 Run AIOps and let your alerts tell you what to fix.

Your engineers ship features at lightning speed. AI copilots autocomplete entire functions. CI/CD pipelines deploy to production in minutes. Modern development has become a symphony of efficiency, with developers operating at 10x the speed of just five years ago.

But there's one part of your stack that's stuck in 2010: your alerts.

While your team vibecodes their way through complex distributed systems, your alerting engine still screams about every CPU spike, memory blip, and network hiccup like it's the apocalypse. It's like having a Ferrari engine attached to horse-and-buggy wheels. The cognitive dissonance is jarring—and it's killing your team's productivity.

Why Alert Fatigue Is a Real Problem in 2025

Here's the absurd reality: The same engineer who just deployed a sophisticated ML model in production gets woken up at 3 AM because a health check endpoint took 501ms instead of 500ms to respond. The developer who elegantly orchestrated a microservices migration gets paged because a pod restarted—something Kubernetes is literally designed to do automatically.

Modern infrastructure has exploded in complexity. You're running hundreds of microservices, each generating alerts. Kubernetes adds its own layer of notifications. Cloud providers, APMs, and security tools all want their voice heard. The result? An endless stream of "urgent" notifications flooding Slack channels and PagerDuty rotations.

But unlike your codebase—which has intelligent linters, smart IDEs, and AI-powered suggestions—your alerts remain dumb. They can't distinguish between:

• A temporary spike during garbage collection vs. a memory leak

• A planned scaling event vs. an unexpected traffic surge

• A self-healing Kubernetes pod restart vs. a critical service failure

The real problem: You don't know which alerts matter anymore.

Your engineers have adapted the only way they can—by tuning out. When every alert claims to be critical but most are noise, even genuine emergencies get ignored. It's the monitoring equivalent of crying wolf, except the wolf is paging your on-call engineer every 30 minutes.

What you need aren't more dashboards visualizing the chaos. You need intelligent tools that understand context, learn patterns, and show you what's noisy and help you take action. Let's examine three approaches to bringing your alerts into the modern era.

Tool #1 – DrDroid

Best for: Real-time visibility into noisy alerts, across any stack

DrDroid represents the first generation of truly intelligent alerting tools. While your engineers use AI to write code faster, DrDroid uses intelligence to make your alerts smarter.

The platform integrates with your existing stack—Slack, Prometheus, New Relic, OpenTelemetry, and more. But what sets it apart is the Alert Insights feature, which applies actual intelligence to your alert patterns:

• Which alerts are flapping? Just like a smart IDE highlights code smells, DrDroid identifies alerts that repeatedly fire and resolve—clear indicators of misconfiguration.

• Which alerts are being ignored? By analyzing engineer behavior, it spots alerts that get dismissed without action. If developers ignore an alert 100% of the time, why is it still paging them?

• Which alerts lack runbooks or clear owners? Nothing frustrates a vibecoding engineer more than context-switching to an alert with zero information about what to do.

DrDroid doesn't just identify problems—it suggests fixes:

• Automatically mute alerts during deployment windows

• Disable alerts that have never correlated with customer impact

• Add intelligent conditions (like requiring sustained threshold breaches)

• Enrich alerts with missing context, runbooks, and correlation data

The platform's auto-debugging capabilities are particularly impressive. When an alert fires, DrDroid automatically pulls relevant logs, metrics, traces, and even recent code changes. It's like having an AI copilot for incident response.

Consider this scenario: Your payment service alerts on high latency every day at 2 PM. DrDroid notices the pattern, correlates it with a scheduled batch job, and suggests either suppressing the alert during that window or adjusting the threshold. What took hours of manual analysis now happens automatically.

Trade-offs

DrDroid is built for modern, Slack-first teams. If your organization has traditional processes requiring all alerts to flow through legacy ITSM tools, adoption might face resistance. As a newer platform, some enterprise compliance features are still maturing.

➡️ 🧠 Want to know which alerts your team should disable? 👉 Explore DrDroid's Alert Insights — loved by SREs to reduce alert fatigue.

Tool #2 – BigPanda

Best for: Enterprise-scale alert correlation

BigPanda takes a different approach—using machine learning to group related alerts into incidents. When a database issue triggers alerts across 20 services, BigPanda recognizes the pattern and presents them as one incident.

For large enterprises with complex systems, this correlation can help. The platform learns relationships between components and can reduce the number of incidents operators review. It also integrates deeply with enterprise tools like ServiceNow and Dynatrace.

Trade-offs

Here's where the contrast with modern development becomes stark. While your engineers deploy code in minutes, BigPanda requires months of setup. While developers use intuitive tools that work out-of-the-box, BigPanda demands extensive metadata configuration and alert standardization.

More critically, BigPanda doesn't make individual alerts smarter—it just groups dumb alerts better. Those flapping alerts your engineers hate? Still firing, just bundled together. It's like organizing spam into folders instead of fixing your spam filter.

The platform is also expensive, often requiring dedicated administrators and cross-team coordination. For teams used to the speed of modern development, BigPanda's implementation timeline feels like stepping back in time.

Tool #3 – PagerDuty Analytics

Best for: Trend visibility inside the PagerDuty ecosystem

PagerDuty Analytics provides retrospective dashboards showing alert volume, MTTR, and on-call load. For teams already using PagerDuty, it offers visibility into historical patterns and trends.

The analytics can be useful for quarterly reviews and capacity planning. You can see which services generate the most alerts and track improvements over time.

Trade-offs

The limitations mirror the gap between modern development and legacy monitoring. While your engineers get real-time feedback from their tools, PagerDuty Analytics is retrospective only. It tells you that Service X generated 500 alerts last month but not which ones were false positives or what to do about them.

It only analyzes alerts flowing through PagerDuty, missing Slack notifications and other channels. The insights are descriptive, not prescriptive—you see the problem visualized but get no help fixing it. And it requires expensive premium tiers, adding cost without adding intelligence.

Comparison Table

Final Thoughts — Your Alerts Should Be as Smart as Your Code

We've entered an era where engineers can literally describe what they want to build and watch AI generate the code. They deploy with confidence, iterate rapidly, and ship features that would have taken months in mere days.

Yet these same engineers—these 10x vibecoding machines—are still being interrupted by alerts that would have been considered noisy a decade ago.

The disconnect is unsustainable. You can't run a modern engineering organization with stone-age alerting. Your monitoring needs to evolve to match the sophistication of your development practices.

BigPanda and PagerDuty show you the problem in high resolution. Only DrDroid's Alert Insights actually makes your alerts smarter—identifying what's broken, why it's noisy, and exactly how to fix it.

The future of monitoring isn't better dashboards or fancier grouping algorithms. It's intelligent systems that understand context, learn from patterns, and proactively help you maintain signal-to-noise ratio. It's alerts that are as smart as the engineers they're interrupting.

Your team deserves alerting infrastructure that matches their development velocity. Stop letting 2010-era alerts slow down your 2025 engineering team.

➡️ 💡 Ready to reduce alert fatigue the smart way? 👉 Start using Alert Insights to find and fix noisy alerts today — no config needed.

Fed up with alert fatigue, you go on a muting spree. That flaky health check? Silenced. The CPU warning that fires during deploys? Disabled. The memory alert that triggers during garbage collection? Gone.

For a blissful week, your on-call rotation is peaceful. Engineers are sleeping through the night. Slack channels are quiet. Life is good.

Then it happens. A real incident slips through. Customer complaints pour in. Your CEO wants answers. And suddenly, those "noisy" alerts you disabled don't seem so unnecessary anymore.

Here's the uncomfortable truth: anyone can reduce alert noise by turning off alerts. The real challenge—the one that separates good SRE teams from great ones—is reducing noise without sacrificing coverage.

Why Reducing Alert Noise Is Harder Than It Sounds

The naive approach to alert fatigue is seductively simple: just turn off the annoying alerts. But this creates a dangerous blind spot. That CPU alert might be noisy 99% of the time, but what about the 1% when it signals a real problem?

The opposite extreme isn't better. Some teams, burned by missed incidents, keep every alert active "just in case." They end up with hundreds of alerts that cry wolf, training engineers to ignore everything—including real emergencies.

The solution isn't choosing between noise and coverage. It's building a systematic approach that maintains visibility while eliminating false positives. High-performing SRE teams follow a 4-phase framework that transforms chaotic alerting into intelligent monitoring.

This framework isn't theoretical—it's battle-tested by teams managing hundreds of services in production. And with modern tools like Alert Insights, you can measure and validate your improvements with data, not guesswork.

Let's dive into each phase.

Phase 1 – Start with Coverage, Not Silence

The mistake most teams make: start muting

When alert fatigue hits, the instinctive response is to start silencing alerts. It feels productive—each muted alert is one less interruption. But this approach is backwards.

Before you disable a single alert, you need to understand what you're actually trying to monitor. This means mapping your core Service Level Objectives (SLOs) and Service Level Indicators (SLIs) to your alerting strategy.

Most teams rely too heavily on infrastructure alerts—CPU usage, memory consumption, disk space. These are important, but they're indirect signals. A service can have high CPU usage while serving customers perfectly. Conversely, it can have normal resource usage while completely failing its primary function.

Instead, start with user-facing signals:

• Failed user logins (not just authentication service uptime)

• Checkout completion rates (not just payment gateway availability)

• API response times at the 95th percentile (not just average latency)

• Database query failures (not just connection pool metrics)

Map these business-critical indicators first. Only after you have comprehensive coverage of what matters should you start tuning what doesn't.

✅ Principle: Only tune alerts after coverage is solid. It's better to have noisy but comprehensive alerting than quiet but blind monitoring.

Phase 2 – Assign Ownership

Every alert should have an owner, a service, and a runbook

Here's a dirty secret of most alerting systems: nobody owns the alerts. They fire into shared channels where responsibility diffuses across the team. When everyone is responsible, no one is accountable.

This shared ownership model is why alerts never improve. The payment team ignores database alerts because "that's infrastructure's problem." The infrastructure team ignores API latency alerts because "that's the app team's issue." Meanwhile, both alerts keep firing, and your on-call engineer suffers.

The fix is radical but simple: every alert must have a single owner. Not a team, not a rotation—a specific service and the team that owns it. This means:

• No more #alerts-general channels where everything dumps

• No more "infrastructure noise" channels that everyone mutes

• Each team gets their own alert destinations

• Each team is accountable for their signal-to-noise ratio

Implement this with proper tagging:

alert: HighAPILatency service: payment-api team: payments owner: payments-team@company.com escalation: payments-oncall severity: P2

When alerts have clear ownership, magic happens. The payments team suddenly cares about that flapping API alert because it's waking them up, not some random SRE. They'll fix it, tune it, or justify why it needs to stay.

✅ Tip: Alerts without owners almost never get fixed. They become background noise that everyone learns to ignore.

Phase 3 – Enrich, Then Tune

Rich alerts = less cognitive load = faster response

Now that you have coverage and ownership, it's time to make your alerts actually useful. A bare-bones "Service X is down" notification forces engineers to context-switch, investigate, and piece together what's happening. Rich alerts provide everything upfront.

Essential enrichment includes:

• Runbook links: Step-by-step remediation instructions

• Severity levels: Is this customer-impacting or internal-only?

• Business impact: How many users affected? Which features degraded?

• Recent changes: Did a deployment just go out?

• Historical context: Has this happened before? How was it fixed?

But richness isn't verbosity. Don't dump entire log files into alerts. Instead, provide precisely what's needed for rapid decision-making.

Only after enrichment should you start tuning:

Add intelligent conditions: Instead of alerting on every spike, require sustained problems:

• Alert only after 3 consecutive failures

• Require issues to persist for 5 minutes

• Use percentage-based thresholds (5% of requests failing vs. 10 absolute failures)

Adjust thresholds based on reality: That 80% CPU alert made sense with your old infrastructure. But if your auto-scaling kicks in at 70%, you're alerting on normal operations.

Add flapping protection: If an alert fires and resolves repeatedly, it needs damping:

• Require state changes to persist before alerting

• Group rapid-fire alerts into single notifications

• Add cooldown periods between alerts

✅ Insight: Context beats volume every time. One well-enriched alert is worth ten noisy notifications.

Phase 4 – Use Data to Improve Over Time

Enter: Alert Insights by DrDroid

Here's where most frameworks fail: they're static. Teams implement phases 1-3, declare victory, and move on. Six months later, they're back to alert fatigue because systems evolve but alerts don't.

You need a continuous feedback loop—a way to measure what's working and what's still broken. This is where Alert Insights becomes your secret weapon.

After implementing your alert structure, Alert Insights provides ongoing intelligence:

• Which alerts are firing too often? That P1 alert that fires 50 times per week probably needs adjustment

• Which ones are being ignored? If engineers acknowledge but never act on an alert, it's pure noise

• Which lack runbooks or clear owners? Gaps in your enrichment strategy become visible

• What can be safely muted, disabled, or improved? Data-driven recommendations, not guesswork

The workflow becomes systematic:

Every sprint:

1. Review Alert Insights dashboard

2. Identify the top 3 worst offenders

3. Fix ownership, enrichment, or tuning for those alerts

4. Validate improvements in the next sprint

5. Repeat

This creates a virtuous cycle. Your alerts get better every sprint. Your on-call experience improves measurably. And you maintain coverage while reducing noise.

➡️ 🧠 Want a clear report on which alerts are hurting your team? 👉 Run DrDroid Alert Insights — no config required.

Bringing It All Together — Your Team's Framework

Here's your systematic approach to intelligent alerting:

This isn't a one-time project—it's an ongoing practice. Just like you continuously refactor code, you need to continuously refine alerts. The difference is that now you have a framework and the data to guide your decisions.

Final Thought — You Can't Fix What You Don't See

Most teams exist in one of two failure modes. They either suffer in silence with alert fatigue, accepting it as the cost of observability. Or they oversimplify their alerting, creating dangerous blind spots that only become visible during incidents.

Real success looks different: high signal, low noise, and fast resolution. It's alerts that wake you up only when customer impact is imminent. It's notifications that include everything needed to respond. It's a system that improves continuously based on data, not opinions.

This framework gives you the path. Phase by phase, you can transform your alerting from a source of frustration into a competitive advantage. But frameworks only work when you can measure their impact.

Let Alert Insights be your guide. It shows what's working, what's broken, and exactly how to improve. No more guessing which alerts to tune. No more hoping you haven't created blind spots. Just data-driven improvements that make your team's life better.

Your engineers deserve better than alert fatigue. Your customers deserve better than missed incidents. This framework delivers both.

➡️ 🛠️ Tired of guessing which alerts are noisy? 👉 Try Alert Insights and start tuning your alerts based on real data.

Welcome to our complete guide for navigating KubeCon + CloudNativeCon Europe 2025 in London, England, running from 1–4 April 2025. Whether you’re a seasoned cloud native pro or new to the Kubernetes world, this guide has everything you need to maximize your conference experience. And don’t forget – visit the Doctor Droid booth at the event! Show us this blog to score an exclusive 20% discount on your ticket plus a chance to receive special Doctor Droid credits!

Overview

KubeCon + CloudNativeCon is the premier conference for Kubernetes, cloud-native technologies, and open source innovations. Organized by the Cloud Native Computing Foundation (CNCF), this flagship event gathers thousands of developers, engineers, and industry leaders to share ideas, network, and explore the latest trends shaping the future of cloud computing. In the heart of London, expect inspiring keynotes, deep-dive sessions, and engaging co-located events that cover everything from AI/ML integration to edge computing and beyond.

Access Types

KubeCon + CloudNativeCon Europe 2025 offers a single, all-inclusive pass that grants access to:

• All keynote sessions, breakout tracks, and panel discussions

• Hands-on labs and workshops for a practical dive into cloud-native solutions

• Co-located events hosted by CNCF and industry partners

Additionally, there are special discounted options available for students and academic participants. For more details, check out the official KubeCon + CloudNativeCon Europe website for registration options.

Exclusive KubeCon + CloudNativeCon Europe 2025 Discount – Save 20% on Tickets, Courtesy of Doctor Droid!

That’s right – Doctor Droid is proud to sponsor KubeCon + CloudNativeCon Europe 2025, and we’re offering you an exclusive 20% discount on your ticket! Here’s how to claim your savings:

1. Fill out our quick Google Form with your basic information.

2. Receive your discount code directly in your inbox.

3. Register on the official event website and enjoy your 20% savings!

Hurry up – secure your discount today and get ready for an unforgettable experience in London!

Speakers & Tracks at KubeCon + CloudNativeCon Europe 2025

The conference features an impressive line-up of industry thought leaders and technical experts across multiple tracks, including:

• Kubernetes Operations: Best practices in deployment, scaling, and security.

• Cloud Security: Deep dives into safeguarding cloud native environments.

• AI & Machine Learning: Innovations transforming how we manage and operate Kubernetes.

• Edge Computing: Exploring the future of distributed computing in real-world scenarios.

Be sure to check out the detailed schedule on the official event page for a complete list of sessions and speakers.

Planning Your London Experience

Where to Stay

London offers a range of accommodation options to suit every budget:

• Hotels: Browse options on Booking.com, Agoda, or Airbnb for a comfortable stay near the event venue.

• Short-term Rentals: Consider serviced apartments if you prefer a homier experience during your stay.

Getting There

London is well connected by international air travel:

• Heathrow Airport (LHR) – The largest and busiest airport, with easy public transit into central London.

• Gatwick Airport (LGW) – A convenient alternative for many international travellers.

Plan your journey ahead to make the most of your time in this vibrant city!

Visiting London

Culinary Delights

London’s food scene is as diverse as it is delicious. Whether you’re looking for Michelin-starred restaurants or quirky food markets, here are some recommendations:

• Restaurants: Try iconic spots in Soho or trendy eateries in Shoreditch.

• After-Hours: Explore vibrant nightlife in Camden or the West End for live music and cocktails.

• Coffee Shops: Recharge at local favorites like Monmouth Coffee or The Attendant for a caffeine boost.

Weekend Plans

When you’re not immersed in conference sessions, take some time to explore London’s rich history and culture:

• The British Museum: Discover art and antiquities from around the world.

• Tower of London: Step back in time with a visit to this historic fortress.

• Buckingham Palace & Changing of the Guard: A must-see for first-time visitors.

• London Eye: Enjoy panoramic views of the city skyline.

Visit the Doctor Droid Booth

Doctor Droid is the intelligent Slack bot that accelerates incident diagnosis by automatically pinpointing the root cause of production issues. Simply tag the bot in your alert messages, and let it do the heavy lifting!

Stop by our booth at KubeCon + CloudNativeCon Europe 2025 to discover:

• Live Demos: See Doctor Droid in action and learn how it can transform your incident response.

• Puzzles & Giveaways: Test your skills and win exciting Doctor Droid goodies.

• $500 Doctor Droid Credits: Show this blog at our booth and receive $500 in credits to supercharge your troubleshooting capabilities.

For those interested in a one-on-one demo, pre-book a meeting with us here.

Get ready to experience the future of cloud native computing in one of the world’s most exciting cities – London awaits at KubeCon + CloudNativeCon Europe 2025!

Happy conferencing, and see you in London!

It’s a scenario we’ve all witnessed: teams equipped with cutting-edge observability tools still struggling to catch issues before customers notice.

They’ve invested heavily in top-tier APM solutions, container and infrastructure monitoring, and log accessibility. Yet, their on-call engineers remain overwhelmed. Incidents happen more frequently than anyone would like, and the spotlight they find themselves in post-incident is never the kind they want.

For engineering teams, being called out for production issues is a tough pill to swallow. The key lies in post-incident action plans that lead to meaningful, systematic improvements.

While production incidents can’t be entirely eliminated, well-thought-out preventive measures can dramatically improve operational health.

Tools Are the Baseline—Not the Answer

While tools are essential, they primarily address infrastructure or service-level issues. However, most real-world incidents cascade across multiple stacks, often affecting features, products, or customer experiences—areas that are rarely solved by out-of-the-box tools.

To reduce MTTR, teams need processes that improve detection, diagnosis, and resolution speed.

Measures for reducing MTTR drastically:

Here are three processes that I have seen help teams in improving MTTR significantly:

1. Improving Actionability of Alerts (Faster Detection)

   • Trustworthy alerts are a cornerstone of effective incident management. Engineers need a single source of truth to detect issues early—before customers or business stakeholders notice.

   • Poorly configured alerts can destroy this trust, leading teams to rely on escalations from support or business teams instead. Monitoring alert quality is critical. For example, many companies using Doctor Droid track alert quality to ensure non-actionable alerts don’t erode confidence in their systems.

2. Instrumenting Custom Metrics

   • Custom metrics are invaluable for tracking operational health and catching issues tied to features and product breakages. Unlike generic service-level metrics, custom metrics provide leading indicators that can help teams spot potential failures before they escalate.

   • By focusing on metrics relevant to their features and customer experience, teams can gain clarity and react faster.

3. Faster Fixing Through Runbooks and Quick Links

   • Developer experience during on-call is often overlooked. Simple resources like runbooks or quick links for known issues can dramatically reduce the cognitive load on engineers.

   • For example, a link to a pre-built log query can save critical minutes during an incident. These tools empower teams to pinpoint issues faster, enabling quicker resolutions.

Conclusion:

No matter how much you spend on tools, improving MTTR requires engineering investment in processes that enhance detection, diagnosis, and resolution. Custom metrics, actionable alerts, and developer-friendly resources are what truly make the difference.

Engineering teams that focus on these practices find themselves more prepared, more resilient, and better positioned to handle the inevitable challenges of production.

Want to monitor your alerting quality and improve MTTR? Doctor Droid has helped 40+ companies take their incident management to the next level. Get started for free and improve your alerts today!

Welcome to our complete guide for navigating KubeCon + CloudNativeCon North America 2024! Here’s all you need to know to make the most of this event in Salt Lake City, Utah, from November 12-15, 2024. Don’t forget to visit Doctor Droid at Booth Q45—show us this blog for a chance to receive $500 worth of Doctor Droid credits!

Overview

KubeCon + CloudNativeCon is the leading conference for Kubernetes, cloud-native technologies, and open-source solutions. Hosted by the Cloud Native Computing Foundation (CNCF), this event gathers thousands of developers, engineers, and business leaders to exchange knowledge, network, and discover the future of cloud-native ecosystems.

Access Types

KubeCon + CloudNativeCon North America 2024 offers different access passes to cater to all types of attendees:

• Full Access Pass: Includes access to all sessions, keynotes, and co-located events

  

• KubeCon + CloudNativeCon Only Pass: Includes access to all sessions, keynotes, excluding co-located events

  

• Academic Pass: Discounted pass for students looking to dive into cloud-native technologies.

• Individual pass: Perfect for attendees who are paying for the conference by themselves.

Be sure to review the registration options on the official KubeCon + CloudNativeCon North America 2024 website.

Exclusive KubeCon + CloudNativeCon North America 2024 Discount – Save 20% on Tickets, Courtesy of Doctor Droid!

That’s right—Doctor Droid is a proud sponsor of KubeCon + CloudNativeCon North America 2024, and we’re hooking you up with an exclusive 20% discount on your tickets! Here’s how to secure your spot and save big:

1. Fill out this google form with your basic info.

2. Get your 20% discount code delivered straight to your inbox.

3. Register for KubeCon + CloudNativeCon North America 2024 and enjoy the savings!

Don’t sit on this—grab your discount before it’s gone!

Co-Located Events at KubeCon + CloudNativeCon North America 2024

Expand your KubeCon + CloudNativeCon experience by joining co-located events, each tailored to specific interests within the cloud-native realm:

• AppDeveloperCon: Focuses on tools and techniques for building cloud-native applications.

• ArgoCon: Dive into Argo workflows, events, and continuous delivery for Kubernetes.

• BackstageCon: Explore Backstage’s developer portal and best practices for engineering platforms.

• Cilium + eBPF Day: A deep dive into Cilium and eBPF for networking and security.

• Cloud Native & Kubernetes AI Day: Discuss AI and ML workloads on Kubernetes.

• CloudNative StartupFest: Networking and insights for startup founders and innovators.

• Cloud Native University: Educational sessions on the fundamentals of cloud-native technologies.

• Data on Kubernetes Day: Explore data management practices and tools for Kubernetes.

• EnvoyCon: Dedicated to the Envoy proxy community, focusing on networking and observability.

• Istio Day: Learn about Istio and service mesh technologies in cloud-native environments.

• Kubernetes on Edge Day: Explore the role of Kubernetes in edge computing.

• Observability Day: A day centered around observability tools and practices in the cloud.

• OpenFeature Summit: Discussions on feature flagging and experimentation in cloud-native setups.

• OpenTofu Day: Open-source infrastructure management and IaC best practices.

• Platform Engineering Day: Dedicated to platform engineering in cloud-native environments.

• WasmCon: Focuses on WebAssembly and its role in cloud-native development.

KubeCon + CloudNativeCon North America 2024 Unofficial Conference Parties

After a day of learning and networking, unwind at the official conference parties.

• Check out Conference Parties for the latest info on social events.

• Check out events on lu.ma too to keep exploring.

Speakers & Tracks at KubeCon + CloudNativeCon North America 2024

KubeCon + CloudNativeCon features keynotes from industry leaders and breakout sessions across multiple tracks, including:

• Kubernetes Operations: Topics around the deployment, scaling, and security of Kubernetes.

• AI and Machine Learning: How AI and ML are transforming Kubernetes.

• Edge Computing: Use cases and solutions for Kubernetes at the edge.

• Cloud Security: Tools and practices to ensure security in a cloud-native environment.

Planning for KubeCon + CloudNativeCon North America 2024 Utah Logistics

Stays near Salt Lake City

Salt Lake City has several convenient options for accommodation. Whether you prefer hotels or Airbnbs, here are a few recommendations:

• Hotels: As of now, all hotels on booking.com as well as most other websites are sold out right now.

• Airbnb: Airbnb still has a healthy number of properties available in the city, although most of them are not within walking distance of the Convention Center.

• 

Airports nearby

• Salt Lake City International Airport (SLC) is the nearest airport, just 5 miles from downtown.

Visiting Salt Lake City

Restaurants

Salt Lake City offers a diverse culinary scene. Recommended spots include:

• Red Iguana: Known for authentic Mexican cuisine.

• The Copper Onion: A top choice for American fare.

• Takashi: Excellent sushi in the heart of the city.

After Hour Locations

• Beer Bar: Perfect for a laid-back evening with a variety of beers.

• The Bayou: Offers an extensive selection of brews and Cajun-style food.

Coffee Shops

Need a caffeine boost? Check out:

• La Barba Coffee: Known for artisanal coffee.

• Publik Coffee Roasters: A great spot to unwind with quality brews.

Weekend Plans

Consider exploring Utah’s natural beauty over the weekend. Popular options include:

• Bonneville Salt Flats: A unique desert landscape.

  

• Big Cottonwood Canyon: Ideal for scenic drives and hikes.

  

Visit Doctor Droid Booth - Q45

Doctor Droid is a root cause identification slack bot which can assist on-call engineers diagnose incidents and find root cause really fast. All you need to do is reply to your alert message in slack by tagging the bot. If you are interested to get a demo and explore more about Doctor Droid, visit us at Booth Q45 in the venue!

Pre-book a meeting with us using this link.

Stop by Booth Q45 to discover how Doctor Droid’s automated RCA can help you debug & fix your production issues faster! What else is up for grabs at the event?

• Puzzles & Goodies: Test your mental muscle and win some amazing gifts.

• $500 Doctor Droid Credits: Show this blog at our booth to receive $500 in Doctor Droid credits!