1. Specialized Debugging Tools & Skills

Production incidents often require analyzing large volumes of logs, traces, and metrics. This can be token-intensive and time-consuming. Most LLMs and agentic frameworks hit context window limits quickly, can't process production-scale data, and lose quality when analyzing large datasets.

How DrDroid Solves This:

Pre-Built Aggregate Analysis Tools

Instead of feeding raw logs into an LLM, DrDroid has tools designed specifically for handling large-volume logs with:

• Built-in log aggregation and pattern detection

• Complex trace analysis across distributed systems

• Large-volume metrics analysis with outlier detection and ML techniques

Specialized Investigation Skills

Our agent has domain-specific skills built from working with hundreds of engineers on real debugging problems:

• How to query and analyze traces in Signoz or Datadog

• How to navigate APM data efficiently

• How to correlate metrics across multiple monitoring tools

Real Impact: The agent can process 100,000+ log lines in seconds and surface the 5 relevant errors—something that would exhaust a generic LLM's context window.

2. Code & Application Awareness

Most LLMs and agents start every investigation from zero, with only the context of the prompt and sometimes markdown files.

How DrDroid Solves This:

Automatic Code Context Generation

Even before your first chat with the agent, DrDroid builds knowledge of:

• What each repository does

• What capabilities, APIs, features, and workflows each repo covers

• Programming languages, frameworks, and file structures

• Connections between multiple repositories (discovered via traces and logs)

Business Workflow Understanding

You can ask DrDroid to build context around critical business and product workflows. The agent understands:

• "The checkout flow involves payment-service, inventory-service, and notification-service"

• "When users report 'payment stuck,' check these three services in this order"

Real Impact: When an alert fires on "payment-service," DrDroid already knows what that service does, which other services depend on it, and where to look for root causes.

3. Infrastructure & Resource Awareness

An LLM with MCP connections doesn't know which apps run in which Kubernetes clusters, which databases are in which cloud providers, or how your infrastructure is organised. It needs to query multiple tools (costing time and token) and explore it before it's able to answer that.

How DrDroid Solves This:

Auto-Discovery of Infrastructure

DrDroid continuously maps:

• Apps hosted in different Kubernetes clusters

• Databases and their cloud providers

• Service dependencies and communication patterns

• Network topology and resource relationships

Service Map & Dependency Graph

The agent can answer questions like:

• "Which services depend on the payments database?"

• "If eu-west-1 goes down, what's affected?"

• "Show me all services running in the production cluster"

Real Impact: During an incident, the agent instantly knows the blast radius and which downstream services might be affected—without you having to explain your architecture.

4. Past Alert & Incident Pattern Recognition

With generic agents, every investigation is independent. It has no memory of past incidents or patterns.

How DrDroid Solves This: Searchable Alert History

The agent has access to:

• All alerts since platform enablement

• Past incidents and their resolutions

• RCAs and postmortems (from Confluence, docs, or previous investigations)

• Understanding patterns in alerts

When similar issues occur, the agent can say:

• "This looks similar to the incident from Jan 15th where the Redis cache was full"

• "Last time this alert fired, the root cause was a config change in service X"

Real Impact: Repeat incidents get resolved faster because the agent learns from past investigations.

5. Continually Learning System

DrDroid improves with every investigation.

Active Learning from Your Environment

The agent continuously creates notes and memory from:

• Recent commits and merges in your applications

• Investigations and conversations with the agent

• Human conversations in Slack channels (optional)

Contextual Memory Storage

Everything is stored with metadata:

• Timestamp

• Related entities (services, databases, clusters)

• Related team and people

• Relevant tags and categories

Real Impact: The agent gets smarter every week. After a month, it knows your environment better than most new engineers.

6. Context Compaction (1M+ Token Conversations)

Typically, agents do the following with the problem of large context windows:

• Summarize the entire conversation (losing critical context) or

• Hit token limits and can't continue

• Slow down dramatically as conversations grow

With production telemetry data, this can often happen.

How DrDroid Solves This:

Intelligent Compression Without Context Loss

• Tool calls are compressed (only IDs and summaries preserved)

• Reasoning and train of thought remain intact (no summarization)

• Agent maintains full context even beyond 1M tokens

• Smart Tool-Level Compaction

Our tools have built-in context management:

• Logging tool has grep/search capability over large volumes

• Agent can "eyeball and search" logs instead of loading everything into context

Real Impact: You can have a 2-hour debugging session with 500+ tool calls, and the agent never loses context or slows down.

7. Multi-Channel Conversations with Shareability

The agent is designed to work from your place of convenience:

• Slack DMs

• Thread replies to alerts

• Web UI

• CLI (coming soon)

• API triggers

• Voice calls (coming soon)

Seamless Sharing

Any investigation can be:

• Shared with teammates for review

• Linked in postmortems

• Referenced in future incidents

Real Impact: When someone gets paged, they can see the auto-investigation that already ran in the Slack thread—no need to DM the agent separately.

8. Automated Investigations

DrDroid can run proactively or via automated triggers enabling proactive visibility for your team:

• Alert fires in PagerDuty/OpsGenie → Investigation starts automatically

• Cron-based health checks → Agent investigates on schedule

• Custom triggers via API or webhooks

Real Impact: Agent can detect issues even without alerts; By the time you open the alert, the agent has already investigated and summarized the likely root cause.

9. Smart Model Switching (85% Cost Savings)

LLMs have been commoditised and the SOTA model is not necessarily required for every investigation. DrDroid smartly chooses between different LLMs based on investigation complexity

• Simple tasks → Faster, cheaper models

• Complex reasoning → State-of-the-art models

Real Impact: Up to 85% token savings compared to always using frontier models, with no degradation in investigation quality.

10. Dedicated File System & Memory

Memory management for a large scale infrastructure requires a structured approach.

DrDroid has a Persistent Knowledge Base - All context, memory, investigations, and alerts are stored and accessible:

• Agent can navigate past investigations like files

• Search across all historical data

• Reference previous findings instantly

Real Impact: "Show me all investigations related to database timeouts in the last 30 days" returns instant results.

11. Coding Sub-Agent for Hotfixes

Coding agent operates very different from a production investigation agent. DrDroid comes pre-packaged with a coding agent connected to the investigation agent.

Real-Time Coding Agent When needed:

• Spins up a coding agent in an ephemeral sandbox

• Reviews the full repository

• Creates hotfix PRs with proper context

Real Impact: During an incident, the agent can say "I found the bug in payment-service line 247—here's a PR to fix it."

12. Remote Machine & Kubernetes Access

Often, data needs to be reviewed on a remote machine or kubernetes cluster for production incidents. These might be inaccessible or sensitive.

How DrDroid Solves This: Direct Infrastructure Access without token access to the agent

• Execute commands on remote machines via SSH (keys are not exposed to the agent)

• Query read-only Kubernetes clusters directly

• Access VMs and clusters within your VPC via reverse proxy

Real Impact: "Check disk space on prod-api-01" → Agent SSHs in, runs the command, and returns results. No manual execution needed.

13. Image Support for Dashboard Analysis

You might want to debug an issue with a screenshot shared by the customer as the starting point.

How DrDroid Solves This: DrDroid agent support image processing from Slack or UI.

• Your product showing an error

• A Grafana dashboard

• A monitoring alert

The agent analyses it and continues the investigation from there.

Real Impact: "Here's what the user is seeing" → Agent understands the UI issue and investigates the backend cause.

14. Granular Access Control & RBAC

Production systems debugging come with sensitive data and access management. DrDroid ensures only the right people have the right access while debugging.

How DrDroid Solves This:

• Read commands: Execute without approval (safe exploration)

• Write commands: Require RBAC approval per your policy

• SSO integration: Syncs with your internal permissions

• Audit logs: Track who did what

Real Impact: Junior engineers can investigate safely, while dangerous operations require senior approval.

15. Third-Party Vendor Status Tracking

Often production incidents can be partly caused due to 3rd party downtimes or issues.

How DrDroid Solves This: Connected to Vendor Statuspages

• Tracks 150+ status pages for your third-party vendors: Stripe, AWS, Datadog, MongoDB Atlas, etc.

• Flags when vendor issues might be causing downstream impact

Real Impact: "Is this our issue or Stripe's?" → Agent checks Stripe's status page and correlates timing.

16. Automated Quality Evaluation

Production Agents need to come with quality guarantees for the team to track and trust.

How DrDroid Solves This: LLM-Based Evals on Every Investigation

Every investigation is automatically evaluated for:

• Accuracy Safety Errors or hallucinations

• Central teams get visibility into investigation quality and improvement opportunities.

Real Impact: Platform team can see "Investigation quality is 94% this month, down from 97% last month—let's review the low-scoring investigations."

17. User Feedback & Team Visibility

Context within DrDroid can continuously improve over time. But for that to improve, tracking and acting upon user feedback is critical.

How DrDroid Solves This: Collaborative Quality Control

• Every investigation can be upvoted/downvoted

• Feedback back to central team helps improve agent context

Real Impact: Central team & managers have visibility on confidence and impact of AI on the engineers.

18. Reasoning Lifecycle & Audit Trail

Production incident investigations cannot be led to be incorrect due to "hallucinations" or "guesses" by an LLM. DrDroid ensures that every reasoning and logic by the LLM is grounded in facts and data.

How DrDroid Solves This: Transparent Investigation Path

The agent tracks:

• What data it queried

• Why each data point was relevant

• What hypothesis it built from each finding

• How it reached its conclusion

Real Impact: You can backtrack through the investigation to validate correctness, spot gaps, or understand the agent's reasoning.

Summary: Why DrDroid is Purpose-Built for Production

What This Means for Your Team:

• Every investigation starts with past context

• You do not have to guide the LLM or explain your architecture every time

• It can handle production-scale logs or metrics

• It supports automation or proactive help

• It works across different channels where your team lives

Ready to See the agent?

DrDroid is a purpose-built investigation agent that understands your infrastructure, learns from your incidents, and gets smarter every day.

Next steps:

• Watch platform demo videos

• Check our MCP Servers & integrations

• Read the documentation

• See customer case studies

Want to see how it works with your stack?

Setup & go live takes 1-2 hours for smaller teams, < 1 week for enterprises. Get started here.

Something is broken right now. You need answers fast.

Day 1 — Operational Tasks & Maintenance

No fire, but you need to keep things running smoothly.

Day 2 — Automation, Optimization & Reliability

You want to stop doing repetitive things and build resilience.

What's Next?

DrDroid helps your team move from reactive firefighting to proactive operations. Whether you're debugging an incident at 3 AM or building automation to prevent the next one, DrDroid becomes your team's operational co-pilot.

Ready to see how DrDroid works with your stack?

• Watch platform demo videos

• Check our integrations

• Read the documentation

• See customer case studies

Want to try it out? Setup takes 1-2 hours and you'll see value from your first investigation. Get started here.

By the end of this DIY guide, you’ll have:

1. A Slack Bot

2. A backend that can

   • Listen to user messages or alerts in a channel and take agentic action based on prompts or steps that you might have in mind.

   • Query your Grafana instance, analyse logs/dashboards and send info about anomaly in reply to an alert/message in your Slack channel

Pre-requisites

1. Python uv (package manager)

2. Ngrok to expose the slackbot server. Setup instructions

3. Grafana (Optional)

Step 0: Clone the repo:

Repository Link - https://github.com/DrDroidLab/slack-ai-bot-builder

Step 1: Building the Slack Bot with an integrated backend

The backend repo setup here, behaves in multiple ways:

• Acts as an MCP Client for any AI calls you might want to make

• Acts as a server to accept webhooks from Slack and manage configurations

Setting up the Ngrok Tunnel

Expose port 5000 of your localhost, using the command:

ngrok http 5000

You will receive a HTTPS URL of the form https://abc123.ngrok.io

Which is pointing to port 5000 of your system.

Note: We have not setup a server running on port 5000 yet, but that is fine since ngrok is independent of that, and exposes the port regardless.

Creating the Slack Application

1. Go to Slack API Apps – https://api.slack.com/apps

2. Click on Create App, and select the option ‘From a manifest’

3. Copy the manifest json from the repository and replace the placeholder ‘<hostname>’ with the HTTPS URL you got from ngrok. Include the “https://” part as well.

4. Install the app in your workspace.

5. Copy the credentials for the slack application into credentials.yaml

   1. app_id, app_name and signing secret can be found in the basic information tab.

   2. Bot-auth-token can be found in the Oauth & Permissions tab.

   3. openai_key from openai - if you plan to use AI based workflows.

6. Create a channel called #drdroid-slack-bot-tester in your slack workspace & add the bot to the channel.

Setting up the bot server

Run the following commands to set up your virtual environment and activate it.

uv env venv
source .venv/bin/activate

Install dependencies using:

uv sync

Now we can finally run the bot server using:

uv run python app.py

The server is now running on port 5000, and exposed to the outside world via your ngrok tunnel.

Testing the bot

Add the bot to the drdroid-slack-bot-tester workspace that you had previously created.

And just type in a ‘hi’. The bot should send you a sample response.

Step 2: Integrating AI

There is already an example workflow for AI (name: "chatbot") in the workflows.yaml. It is just boilerplate code, you can modify it as required. For now, you can tag your bot in the drdroid-slack-bot-tester, and chat with it.

• Add your OpenAI/LLM key

For example:

Message in Slack: chatbot How to debug Kubernetes CrashLoopBackOff error? @bot
Message in Slack: chatbot I'm getting this alert. What does it mean? @bot

Step 3: Making the bot an Agent by giving AI access to different tools (Grafana for demo)

MCP Servers help abstract out any API and making them accessible to AI.

Setting up Grafana MCP Server

Clone the following repository:

Repository URL - https://github.com/DrDroidLab/grafana-mcp-server

Navigate into the root directory of the repository.

Populate the src/grafana_mcp_server/config.yaml with your grafana credentials.

Install and setup the dependencies using:

uv venv .venv
source .venv/bin/activate
uv sync

Run the MCP server:

uv run -m src.grafana_mcp_server.mcp_server

Your MCP server is now running on port 8000.

Creating an AI Grafana workflow in slack-bot-builder

There is already an example workflow for grafana ai in the workflows.yaml.

We will be running the script scripts/grafana_ai_tool.py in this workflow, it is just boilerplate code, you can modify it as required.
Now you can tag your bot in the drdroid-slack-bot-tester, and ask it to do various things from grafana.
For example:

Message in Slack: Fetch me logs from the currencyservice in grafana ai. 
Message in Slack: Fetch and analyse the Go Microservices dashboard from grafana ai

Next Steps:

Now that you’ve been able to setup a bot, here are a few things you can do:

• Productionise it from your current ngrok setup to a static endpoint

• Integrate with Grafana or open source MCP servers of your favourite tool that you want to leverage for automation

• Add custom prompts and scripts

Stuck anywhere? Ask on our Discord

You log into Grafana, click through three nested menus, find the alert, and bump the threshold from 80% to 85%. Crisis averted. You go back to bed.

Two weeks later, during a postmortem, someone asks: "Who changed the CPU alert threshold? And why?"

Silence. Nobody remembers. There's no history. No context. No way to know if this was a temporary hack or a deliberate tuning decision. Worse, when you refresh your staging environment, the old threshold returns because the change only lived in the production UI.

Sound familiar? You're not alone. This is how most teams manage alerts—and it's fundamentally broken.

Why Managing Alerts in Dashboards Doesn't Scale

We've spent the last decade moving infrastructure to code. Terraform for cloud resources. Helm charts for Kubernetes. Ansible for configuration. Yet somehow, our alert rules—critical infrastructure that wakes up engineers—still live in UI dashboards like it's 2010.

The problems compound quickly:

No version history: When did this alert last change? Who changed it? Why? Your Grafana dashboard shrugs.

No peer review: A junior engineer can accidentally change a critical alert threshold with zero oversight. Try doing that with production code.

No rollback capability: That "quick fix" that made things worse? Good luck remembering the old values.

Environment drift: Production alerts diverge from staging. Dev environments have different rules. Chaos ensues.

No ownership tracking: Who owns this alert? Which team should review changes? The UI doesn't care.

Your infrastructure evolves constantly. Services scale. Traffic patterns shift. Performance characteristics change. But alerts configured through dashboards remain frozen in time, slowly becoming less relevant until they're just noise.

Here's the thing: alert rules are infrastructure-as-code too. They define critical system behavior. They impact your team's quality of life. They deserve the same rigor as any other code.

Enter GitOps for alerts—where alert definitions live in version control, changes happen through pull requests, and every modification is tracked, reviewed, and reversible.

What is GitOps for Alerting?

GitOps for alerting is beautifully simple: store your alert rules as code in Git, manage changes through pull requests, and deploy automatically. Just like any other infrastructure.

Most modern monitoring tools already support this:

• Prometheus: Alert rules in YAML files

• Alertmanager: Routing configuration as code

• Grafana: Alerts exportable as JSON

• Datadog: Monitors manageable via Terraform

• New Relic: Alerts configurable through their API/Terraform

Here's what a typical structure looks like:

/alerts/
  frontend-service.yaml
  database.yaml
  redis.yaml

/teams/
  payments/
    api-alerts.yaml
    database-alerts.yaml
  platform/
    infrastructure-alerts.yaml
    kubernetes-alerts.yaml

A Prometheus alert rule might look like:

groups:
  - name: frontend-service
    rules:
      - alert: HighErrorRate
        expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
        for: 5m
        labels:
          severity: warning
          service: frontend
          team: frontend-team
        annotations:
          summary: "High error rate on {{ $labels.instance }}"
          description: "Error rate is {{ $value }} (threshold 0.05)"
          runbook: "https://wiki.company.com/runbooks/frontend-errors"
          owner: "frontend-oncall@company.com"
The benefits are immediate:

✅ Traceability: Every change is a commit. Git blame tells you who changed what and when.

✅ Peer review: Alert changes go through PR reviews. No more accidental 3 AM threshold adjustments.

✅ Consistency: Deploy the same alerts across all environments. No more production/staging drift.

✅ Rollback capability: Bad change? git revert and you're back to working alerts.

✅ Documentation: PR descriptions explain why changes were made. Context is preserved forever.

But GitOps Alone Isn't Enough

Here's the plot twist: GitOps for alerts solves the how but not the what.

You now have beautiful, version-controlled alert rules. Every change is reviewed and tracked. But you still don't know which rules need updating.

Your Git repo becomes a graveyard of alert rules that might or might not be relevant:

• That CPU alert from 2019 when you ran on smaller instances

• The memory warning tuned for your old Java app (you've since moved to Go)

• The latency threshold set when you had 100 users (you now have 10,000)

You've traded one problem for another. Instead of stale alerts in dashboards, you have stale alerts in Git. They're better organized, sure, but still noisy.

This is where most GitOps alerting stories end. Teams implement the framework but lack the feedback loop to keep it healthy. Alert rules accumulate like sediment. Engineers suffer in silence because "at least it's in Git now."

Using Alert Insights to Drive GitOps Changes

Let real alert data guide your pull requests

The missing piece is data. You need to know which alerts are actually problematic before you can fix them. This is where DrDroid's Alert Insights transforms GitOps from a theoretical improvement into a practical solution.

Alert Insights analyzes your live production alerts and tells you:

• Which alerts fired most frequently last week: Your noisiest offenders, ranked

• Which alerts were ignored: Clear signal of rules that need removal

• Which alerts lack owners or runbooks: Quality issues to address

• Suggested changes: Specific recommendations to mute, tweak, or archive

Now GitOps becomes powerful. You're not guessing which alert rules to update—you have data.

✅ Workflow Example:

Monday: Run Alert Insights

`Top 3 Noisy Alerts:

1. redis_memory_warning - 127 fires, 0 actions taken

2. api_latency_high - 89 fires, acknowledged but not investigated

3. cpu_usage_critical - 45 fires, all during deploy windows`

Tuesday: Create targeted PRs

git checkout -b fix/reduce-redis-memory-noise
# Edit alerts/redis.yaml
# Increase threshold from 70% to 80% based on actual usage patterns
git commit -m "Increase Redis memory threshold to reduce false positives

Alert Insights showed 127 fires with 0 actions last week. Analysis shows Redis memory naturally spikes to 75% during cache warmup."

Wednesday: Review and merge

• Team reviews the PR

• Links to Alert Insights data provide context

• Changes deploy automatically

Thursday: Validate impact

• Alert noise drops immediately

• Next week's Alert Insights confirms improvement

The feedback loop is complete. You're not just organizing alerts better—you're systematically improving them based on real data.

➡️ 🛠️ Want a GitOps-ready alert audit? 👉 Run DrDroid's Alert Insights and get actionable suggestions in minutes.

Recommended Practices for GitOps Alert Management

🔍 Use clear filenames per service/component

Don't create a monolithic alerts.yaml. Break rules into logical groups:

/alerts/
  services/
    payment-api.yaml
    user-service.yaml
  infrastructure/
    kubernetes-nodes.yaml
    database-cluster.yaml
  business/
    checkout-flow.yaml
    user-engagement.yaml

🔄 Add labels/tags to help Alert Insights map alerts to owners

Every alert should include:

yaml

labels: team: payments service: payment-api environment: production severity: P2

This metadata powers Alert Insights' analysis and recommendations.

🧪 Validate rules with test alerts in staging

Before merging, trigger test conditions:

# Simulate high error rate
curl -X POST http://prometheus:9090/api/v1/series \
-d 'match[]=up{job="frontend"}'
🔁 Link PRs to weekly alert review

Common GitOps Pitfalls to Avoid

❌ Bulk silencing alerts without context

"Let's just comment out all the noisy alerts" is tempting but dangerous. Use Alert Insights to understand why alerts are noisy before acting.

❌ Committing rules without reviews

The whole point of GitOps is peer review. Don't bypass it with direct commits, even for "quick fixes."

❌ No tagging = Alert Insights can't map alerts to services

Without proper labels, you lose the ability to analyze alerts by team, service, or severity. Enforce tagging standards.

❌ Alert rules diverging across environments

Use templating to keep staging and production alerts synchronized:

# values-prod.yaml
cpu_threshold: 80
memory_threshold: 85

# values-staging.yaml
cpu_threshold: 90 # Higher tolerance in staging
memory_threshold: 90

Final Take — Let Data Drive Your Alert Rule Changes

GitOps gives you the framework for managing alerts professionally. Version control, peer review, and rollback capabilities bring alerts into the modern era.

But framework without data is just organized chaos. You need to know which alerts to fix, how to fix them, and whether your fixes worked.

Alert Insights provides that missing data layer. It tells you which alert rules are hurting your team, suggests specific improvements, and validates that your changes actually reduced noise.

Together, they create a powerful feedback loop:

1. Alert Insights identifies problematic alerts

2. GitOps enables reviewed, tracked changes

3. Automated deployment ensures consistency

4. Next week's Alert Insights validates improvement

This isn't theoretical. Teams using this approach report 50-70% reduction in alert noise within weeks. On-call engineers sleep better. Real incidents get proper attention. Alert quality becomes a measurable, improvable metric.

Your alerts deserve the same engineering rigor as your code. GitOps provides the foundation. Alert Insights provides the intelligence. Together, they transform alerting from a necessary evil into a competitive advantage.

➡️ ✍️ Want to make smarter, reviewable changes to your alerts? 👉 Run AIOps and let your alerts tell you what to fix.

Your engineers ship features at lightning speed. AI copilots autocomplete entire functions. CI/CD pipelines deploy to production in minutes. Modern development has become a symphony of efficiency, with developers operating at 10x the speed of just five years ago.

But there's one part of your stack that's stuck in 2010: your alerts.

While your team vibecodes their way through complex distributed systems, your alerting engine still screams about every CPU spike, memory blip, and network hiccup like it's the apocalypse. It's like having a Ferrari engine attached to horse-and-buggy wheels. The cognitive dissonance is jarring—and it's killing your team's productivity.

Why Alert Fatigue Is a Real Problem in 2025

Here's the absurd reality: The same engineer who just deployed a sophisticated ML model in production gets woken up at 3 AM because a health check endpoint took 501ms instead of 500ms to respond. The developer who elegantly orchestrated a microservices migration gets paged because a pod restarted—something Kubernetes is literally designed to do automatically.

Modern infrastructure has exploded in complexity. You're running hundreds of microservices, each generating alerts. Kubernetes adds its own layer of notifications. Cloud providers, APMs, and security tools all want their voice heard. The result? An endless stream of "urgent" notifications flooding Slack channels and PagerDuty rotations.

But unlike your codebase—which has intelligent linters, smart IDEs, and AI-powered suggestions—your alerts remain dumb. They can't distinguish between:

• A temporary spike during garbage collection vs. a memory leak

• A planned scaling event vs. an unexpected traffic surge

• A self-healing Kubernetes pod restart vs. a critical service failure

The real problem: You don't know which alerts matter anymore.

Your engineers have adapted the only way they can—by tuning out. When every alert claims to be critical but most are noise, even genuine emergencies get ignored. It's the monitoring equivalent of crying wolf, except the wolf is paging your on-call engineer every 30 minutes.

What you need aren't more dashboards visualizing the chaos. You need intelligent tools that understand context, learn patterns, and show you what's noisy and help you take action. Let's examine three approaches to bringing your alerts into the modern era.

Tool #1 – DrDroid

Best for: Real-time visibility into noisy alerts, across any stack

DrDroid represents the first generation of truly intelligent alerting tools. While your engineers use AI to write code faster, DrDroid uses intelligence to make your alerts smarter.

The platform integrates with your existing stack—Slack, Prometheus, New Relic, OpenTelemetry, and more. But what sets it apart is the Alert Insights feature, which applies actual intelligence to your alert patterns:

• Which alerts are flapping? Just like a smart IDE highlights code smells, DrDroid identifies alerts that repeatedly fire and resolve—clear indicators of misconfiguration.

• Which alerts are being ignored? By analyzing engineer behavior, it spots alerts that get dismissed without action. If developers ignore an alert 100% of the time, why is it still paging them?

• Which alerts lack runbooks or clear owners? Nothing frustrates a vibecoding engineer more than context-switching to an alert with zero information about what to do.

DrDroid doesn't just identify problems—it suggests fixes:

• Automatically mute alerts during deployment windows

• Disable alerts that have never correlated with customer impact

• Add intelligent conditions (like requiring sustained threshold breaches)

• Enrich alerts with missing context, runbooks, and correlation data

The platform's auto-debugging capabilities are particularly impressive. When an alert fires, DrDroid automatically pulls relevant logs, metrics, traces, and even recent code changes. It's like having an AI copilot for incident response.

Consider this scenario: Your payment service alerts on high latency every day at 2 PM. DrDroid notices the pattern, correlates it with a scheduled batch job, and suggests either suppressing the alert during that window or adjusting the threshold. What took hours of manual analysis now happens automatically.

Trade-offs

DrDroid is built for modern, Slack-first teams. If your organization has traditional processes requiring all alerts to flow through legacy ITSM tools, adoption might face resistance. As a newer platform, some enterprise compliance features are still maturing.

➡️ 🧠 Want to know which alerts your team should disable? 👉 Explore DrDroid's Alert Insights — loved by SREs to reduce alert fatigue.

Tool #2 – BigPanda

Best for: Enterprise-scale alert correlation

BigPanda takes a different approach—using machine learning to group related alerts into incidents. When a database issue triggers alerts across 20 services, BigPanda recognizes the pattern and presents them as one incident.

For large enterprises with complex systems, this correlation can help. The platform learns relationships between components and can reduce the number of incidents operators review. It also integrates deeply with enterprise tools like ServiceNow and Dynatrace.

Trade-offs

Here's where the contrast with modern development becomes stark. While your engineers deploy code in minutes, BigPanda requires months of setup. While developers use intuitive tools that work out-of-the-box, BigPanda demands extensive metadata configuration and alert standardization.

More critically, BigPanda doesn't make individual alerts smarter—it just groups dumb alerts better. Those flapping alerts your engineers hate? Still firing, just bundled together. It's like organizing spam into folders instead of fixing your spam filter.

The platform is also expensive, often requiring dedicated administrators and cross-team coordination. For teams used to the speed of modern development, BigPanda's implementation timeline feels like stepping back in time.

Tool #3 – PagerDuty Analytics

Best for: Trend visibility inside the PagerDuty ecosystem

PagerDuty Analytics provides retrospective dashboards showing alert volume, MTTR, and on-call load. For teams already using PagerDuty, it offers visibility into historical patterns and trends.

The analytics can be useful for quarterly reviews and capacity planning. You can see which services generate the most alerts and track improvements over time.

Trade-offs

The limitations mirror the gap between modern development and legacy monitoring. While your engineers get real-time feedback from their tools, PagerDuty Analytics is retrospective only. It tells you that Service X generated 500 alerts last month but not which ones were false positives or what to do about them.

It only analyzes alerts flowing through PagerDuty, missing Slack notifications and other channels. The insights are descriptive, not prescriptive—you see the problem visualized but get no help fixing it. And it requires expensive premium tiers, adding cost without adding intelligence.

Comparison Table

Final Thoughts — Your Alerts Should Be as Smart as Your Code

We've entered an era where engineers can literally describe what they want to build and watch AI generate the code. They deploy with confidence, iterate rapidly, and ship features that would have taken months in mere days.

Yet these same engineers—these 10x vibecoding machines—are still being interrupted by alerts that would have been considered noisy a decade ago.

The disconnect is unsustainable. You can't run a modern engineering organization with stone-age alerting. Your monitoring needs to evolve to match the sophistication of your development practices.

BigPanda and PagerDuty show you the problem in high resolution. Only DrDroid's Alert Insights actually makes your alerts smarter—identifying what's broken, why it's noisy, and exactly how to fix it.

The future of monitoring isn't better dashboards or fancier grouping algorithms. It's intelligent systems that understand context, learn from patterns, and proactively help you maintain signal-to-noise ratio. It's alerts that are as smart as the engineers they're interrupting.

Your team deserves alerting infrastructure that matches their development velocity. Stop letting 2010-era alerts slow down your 2025 engineering team.

➡️ 💡 Ready to reduce alert fatigue the smart way? 👉 Start using Alert Insights to find and fix noisy alerts today — no config needed.

Fed up with alert fatigue, you go on a muting spree. That flaky health check? Silenced. The CPU warning that fires during deploys? Disabled. The memory alert that triggers during garbage collection? Gone.

For a blissful week, your on-call rotation is peaceful. Engineers are sleeping through the night. Slack channels are quiet. Life is good.

Then it happens. A real incident slips through. Customer complaints pour in. Your CEO wants answers. And suddenly, those "noisy" alerts you disabled don't seem so unnecessary anymore.

Here's the uncomfortable truth: anyone can reduce alert noise by turning off alerts. The real challenge—the one that separates good SRE teams from great ones—is reducing noise without sacrificing coverage.

Why Reducing Alert Noise Is Harder Than It Sounds

The naive approach to alert fatigue is seductively simple: just turn off the annoying alerts. But this creates a dangerous blind spot. That CPU alert might be noisy 99% of the time, but what about the 1% when it signals a real problem?

The opposite extreme isn't better. Some teams, burned by missed incidents, keep every alert active "just in case." They end up with hundreds of alerts that cry wolf, training engineers to ignore everything—including real emergencies.

The solution isn't choosing between noise and coverage. It's building a systematic approach that maintains visibility while eliminating false positives. High-performing SRE teams follow a 4-phase framework that transforms chaotic alerting into intelligent monitoring.

This framework isn't theoretical—it's battle-tested by teams managing hundreds of services in production. And with modern tools like Alert Insights, you can measure and validate your improvements with data, not guesswork.

Let's dive into each phase.

Phase 1 – Start with Coverage, Not Silence

The mistake most teams make: start muting

When alert fatigue hits, the instinctive response is to start silencing alerts. It feels productive—each muted alert is one less interruption. But this approach is backwards.

Before you disable a single alert, you need to understand what you're actually trying to monitor. This means mapping your core Service Level Objectives (SLOs) and Service Level Indicators (SLIs) to your alerting strategy.

Most teams rely too heavily on infrastructure alerts—CPU usage, memory consumption, disk space. These are important, but they're indirect signals. A service can have high CPU usage while serving customers perfectly. Conversely, it can have normal resource usage while completely failing its primary function.

Instead, start with user-facing signals:

• Failed user logins (not just authentication service uptime)

• Checkout completion rates (not just payment gateway availability)

• API response times at the 95th percentile (not just average latency)

• Database query failures (not just connection pool metrics)

Map these business-critical indicators first. Only after you have comprehensive coverage of what matters should you start tuning what doesn't.

✅ Principle: Only tune alerts after coverage is solid. It's better to have noisy but comprehensive alerting than quiet but blind monitoring.

Phase 2 – Assign Ownership

Every alert should have an owner, a service, and a runbook

Here's a dirty secret of most alerting systems: nobody owns the alerts. They fire into shared channels where responsibility diffuses across the team. When everyone is responsible, no one is accountable.

This shared ownership model is why alerts never improve. The payment team ignores database alerts because "that's infrastructure's problem." The infrastructure team ignores API latency alerts because "that's the app team's issue." Meanwhile, both alerts keep firing, and your on-call engineer suffers.

The fix is radical but simple: every alert must have a single owner. Not a team, not a rotation—a specific service and the team that owns it. This means:

• No more #alerts-general channels where everything dumps

• No more "infrastructure noise" channels that everyone mutes

• Each team gets their own alert destinations

• Each team is accountable for their signal-to-noise ratio

Implement this with proper tagging:

alert: HighAPILatency service: payment-api team: payments owner: payments-team@company.com escalation: payments-oncall severity: P2

When alerts have clear ownership, magic happens. The payments team suddenly cares about that flapping API alert because it's waking them up, not some random SRE. They'll fix it, tune it, or justify why it needs to stay.

✅ Tip: Alerts without owners almost never get fixed. They become background noise that everyone learns to ignore.

Phase 3 – Enrich, Then Tune

Rich alerts = less cognitive load = faster response

Now that you have coverage and ownership, it's time to make your alerts actually useful. A bare-bones "Service X is down" notification forces engineers to context-switch, investigate, and piece together what's happening. Rich alerts provide everything upfront.

Essential enrichment includes:

• Runbook links: Step-by-step remediation instructions

• Severity levels: Is this customer-impacting or internal-only?

• Business impact: How many users affected? Which features degraded?

• Recent changes: Did a deployment just go out?

• Historical context: Has this happened before? How was it fixed?

But richness isn't verbosity. Don't dump entire log files into alerts. Instead, provide precisely what's needed for rapid decision-making.

Only after enrichment should you start tuning:

Add intelligent conditions: Instead of alerting on every spike, require sustained problems:

• Alert only after 3 consecutive failures

• Require issues to persist for 5 minutes

• Use percentage-based thresholds (5% of requests failing vs. 10 absolute failures)

Adjust thresholds based on reality: That 80% CPU alert made sense with your old infrastructure. But if your auto-scaling kicks in at 70%, you're alerting on normal operations.

Add flapping protection: If an alert fires and resolves repeatedly, it needs damping:

• Require state changes to persist before alerting

• Group rapid-fire alerts into single notifications

• Add cooldown periods between alerts

✅ Insight: Context beats volume every time. One well-enriched alert is worth ten noisy notifications.

Phase 4 – Use Data to Improve Over Time

Enter: Alert Insights by DrDroid

Here's where most frameworks fail: they're static. Teams implement phases 1-3, declare victory, and move on. Six months later, they're back to alert fatigue because systems evolve but alerts don't.

You need a continuous feedback loop—a way to measure what's working and what's still broken. This is where Alert Insights becomes your secret weapon.

After implementing your alert structure, Alert Insights provides ongoing intelligence:

• Which alerts are firing too often? That P1 alert that fires 50 times per week probably needs adjustment

• Which ones are being ignored? If engineers acknowledge but never act on an alert, it's pure noise

• Which lack runbooks or clear owners? Gaps in your enrichment strategy become visible

• What can be safely muted, disabled, or improved? Data-driven recommendations, not guesswork

The workflow becomes systematic:

Every sprint:

1. Review Alert Insights dashboard

2. Identify the top 3 worst offenders

3. Fix ownership, enrichment, or tuning for those alerts

4. Validate improvements in the next sprint

5. Repeat

This creates a virtuous cycle. Your alerts get better every sprint. Your on-call experience improves measurably. And you maintain coverage while reducing noise.

➡️ 🧠 Want a clear report on which alerts are hurting your team? 👉 Run DrDroid Alert Insights — no config required.

Bringing It All Together — Your Team's Framework

Here's your systematic approach to intelligent alerting:

This isn't a one-time project—it's an ongoing practice. Just like you continuously refactor code, you need to continuously refine alerts. The difference is that now you have a framework and the data to guide your decisions.

Final Thought — You Can't Fix What You Don't See

Most teams exist in one of two failure modes. They either suffer in silence with alert fatigue, accepting it as the cost of observability. Or they oversimplify their alerting, creating dangerous blind spots that only become visible during incidents.

Real success looks different: high signal, low noise, and fast resolution. It's alerts that wake you up only when customer impact is imminent. It's notifications that include everything needed to respond. It's a system that improves continuously based on data, not opinions.

This framework gives you the path. Phase by phase, you can transform your alerting from a source of frustration into a competitive advantage. But frameworks only work when you can measure their impact.

Let Alert Insights be your guide. It shows what's working, what's broken, and exactly how to improve. No more guessing which alerts to tune. No more hoping you haven't created blind spots. Just data-driven improvements that make your team's life better.

Your engineers deserve better than alert fatigue. Your customers deserve better than missed incidents. This framework delivers both.

➡️ 🛠️ Tired of guessing which alerts are noisy? 👉 Try Alert Insights and start tuning your alerts based on real data.

Welcome to our complete guide for navigating KubeCon + CloudNativeCon Europe 2025 in London, England, running from 1–4 April 2025. Whether you’re a seasoned cloud native pro or new to the Kubernetes world, this guide has everything you need to maximize your conference experience. And don’t forget – visit the Doctor Droid booth at the event! Show us this blog to score an exclusive 20% discount on your ticket plus a chance to receive special Doctor Droid credits!

Overview

KubeCon + CloudNativeCon is the premier conference for Kubernetes, cloud-native technologies, and open source innovations. Organized by the Cloud Native Computing Foundation (CNCF), this flagship event gathers thousands of developers, engineers, and industry leaders to share ideas, network, and explore the latest trends shaping the future of cloud computing. In the heart of London, expect inspiring keynotes, deep-dive sessions, and engaging co-located events that cover everything from AI/ML integration to edge computing and beyond.

Access Types

KubeCon + CloudNativeCon Europe 2025 offers a single, all-inclusive pass that grants access to:

• All keynote sessions, breakout tracks, and panel discussions

• Hands-on labs and workshops for a practical dive into cloud-native solutions

• Co-located events hosted by CNCF and industry partners

Additionally, there are special discounted options available for students and academic participants. For more details, check out the official KubeCon + CloudNativeCon Europe website for registration options.

Exclusive KubeCon + CloudNativeCon Europe 2025 Discount – Save 20% on Tickets, Courtesy of Doctor Droid!

That’s right – Doctor Droid is proud to sponsor KubeCon + CloudNativeCon Europe 2025, and we’re offering you an exclusive 20% discount on your ticket! Here’s how to claim your savings:

1. Fill out our quick Google Form with your basic information.

2. Receive your discount code directly in your inbox.

3. Register on the official event website and enjoy your 20% savings!

Hurry up – secure your discount today and get ready for an unforgettable experience in London!

Speakers & Tracks at KubeCon + CloudNativeCon Europe 2025

The conference features an impressive line-up of industry thought leaders and technical experts across multiple tracks, including:

• Kubernetes Operations: Best practices in deployment, scaling, and security.

• Cloud Security: Deep dives into safeguarding cloud native environments.

• AI & Machine Learning: Innovations transforming how we manage and operate Kubernetes.

• Edge Computing: Exploring the future of distributed computing in real-world scenarios.

Be sure to check out the detailed schedule on the official event page for a complete list of sessions and speakers.

Planning Your London Experience

Where to Stay

London offers a range of accommodation options to suit every budget:

• Hotels: Browse options on Booking.com, Agoda, or Airbnb for a comfortable stay near the event venue.

• Short-term Rentals: Consider serviced apartments if you prefer a homier experience during your stay.

Getting There

London is well connected by international air travel:

• Heathrow Airport (LHR) – The largest and busiest airport, with easy public transit into central London.

• Gatwick Airport (LGW) – A convenient alternative for many international travellers.

Plan your journey ahead to make the most of your time in this vibrant city!

Visiting London

Culinary Delights

London’s food scene is as diverse as it is delicious. Whether you’re looking for Michelin-starred restaurants or quirky food markets, here are some recommendations:

• Restaurants: Try iconic spots in Soho or trendy eateries in Shoreditch.

• After-Hours: Explore vibrant nightlife in Camden or the West End for live music and cocktails.

• Coffee Shops: Recharge at local favorites like Monmouth Coffee or The Attendant for a caffeine boost.

Weekend Plans

When you’re not immersed in conference sessions, take some time to explore London’s rich history and culture:

• The British Museum: Discover art and antiquities from around the world.

• Tower of London: Step back in time with a visit to this historic fortress.

• Buckingham Palace & Changing of the Guard: A must-see for first-time visitors.

• London Eye: Enjoy panoramic views of the city skyline.

Visit the Doctor Droid Booth

Doctor Droid is the intelligent Slack bot that accelerates incident diagnosis by automatically pinpointing the root cause of production issues. Simply tag the bot in your alert messages, and let it do the heavy lifting!

Stop by our booth at KubeCon + CloudNativeCon Europe 2025 to discover:

• Live Demos: See Doctor Droid in action and learn how it can transform your incident response.

• Puzzles & Giveaways: Test your skills and win exciting Doctor Droid goodies.

• $500 Doctor Droid Credits: Show this blog at our booth and receive $500 in credits to supercharge your troubleshooting capabilities.

For those interested in a one-on-one demo, pre-book a meeting with us here.

Get ready to experience the future of cloud native computing in one of the world’s most exciting cities – London awaits at KubeCon + CloudNativeCon Europe 2025!

Happy conferencing, and see you in London!

It’s a scenario we’ve all witnessed: teams equipped with cutting-edge observability tools still struggling to catch issues before customers notice.

They’ve invested heavily in top-tier APM solutions, container and infrastructure monitoring, and log accessibility. Yet, their on-call engineers remain overwhelmed. Incidents happen more frequently than anyone would like, and the spotlight they find themselves in post-incident is never the kind they want.

For engineering teams, being called out for production issues is a tough pill to swallow. The key lies in post-incident action plans that lead to meaningful, systematic improvements.

While production incidents can’t be entirely eliminated, well-thought-out preventive measures can dramatically improve operational health.

Tools Are the Baseline—Not the Answer

While tools are essential, they primarily address infrastructure or service-level issues. However, most real-world incidents cascade across multiple stacks, often affecting features, products, or customer experiences—areas that are rarely solved by out-of-the-box tools.

To reduce MTTR, teams need processes that improve detection, diagnosis, and resolution speed.

Measures for reducing MTTR drastically:

Here are three processes that I have seen help teams in improving MTTR significantly:

1. Improving Actionability of Alerts (Faster Detection)

   • Trustworthy alerts are a cornerstone of effective incident management. Engineers need a single source of truth to detect issues early—before customers or business stakeholders notice.

   • Poorly configured alerts can destroy this trust, leading teams to rely on escalations from support or business teams instead. Monitoring alert quality is critical. For example, many companies using Doctor Droid track alert quality to ensure non-actionable alerts don’t erode confidence in their systems.

2. Instrumenting Custom Metrics

   • Custom metrics are invaluable for tracking operational health and catching issues tied to features and product breakages. Unlike generic service-level metrics, custom metrics provide leading indicators that can help teams spot potential failures before they escalate.

   • By focusing on metrics relevant to their features and customer experience, teams can gain clarity and react faster.

3. Faster Fixing Through Runbooks and Quick Links

   • Developer experience during on-call is often overlooked. Simple resources like runbooks or quick links for known issues can dramatically reduce the cognitive load on engineers.

   • For example, a link to a pre-built log query can save critical minutes during an incident. These tools empower teams to pinpoint issues faster, enabling quicker resolutions.

Conclusion:

No matter how much you spend on tools, improving MTTR requires engineering investment in processes that enhance detection, diagnosis, and resolution. Custom metrics, actionable alerts, and developer-friendly resources are what truly make the difference.

Engineering teams that focus on these practices find themselves more prepared, more resilient, and better positioned to handle the inevitable challenges of production.

Want to monitor your alerting quality and improve MTTR? Doctor Droid has helped 40+ companies take their incident management to the next level. Get started for free and improve your alerts today!

Welcome to our complete guide for navigating KubeCon + CloudNativeCon North America 2024! Here’s all you need to know to make the most of this event in Salt Lake City, Utah, from November 12-15, 2024. Don’t forget to visit Doctor Droid at Booth Q45—show us this blog for a chance to receive $500 worth of Doctor Droid credits!

Overview

KubeCon + CloudNativeCon is the leading conference for Kubernetes, cloud-native technologies, and open-source solutions. Hosted by the Cloud Native Computing Foundation (CNCF), this event gathers thousands of developers, engineers, and business leaders to exchange knowledge, network, and discover the future of cloud-native ecosystems.

Access Types

KubeCon + CloudNativeCon North America 2024 offers different access passes to cater to all types of attendees:

• Full Access Pass: Includes access to all sessions, keynotes, and co-located events

  

• KubeCon + CloudNativeCon Only Pass: Includes access to all sessions, keynotes, excluding co-located events

  

• Academic Pass: Discounted pass for students looking to dive into cloud-native technologies.

• Individual pass: Perfect for attendees who are paying for the conference by themselves.

Be sure to review the registration options on the official KubeCon + CloudNativeCon North America 2024 website.

Exclusive KubeCon + CloudNativeCon North America 2024 Discount – Save 20% on Tickets, Courtesy of Doctor Droid!

That’s right—Doctor Droid is a proud sponsor of KubeCon + CloudNativeCon North America 2024, and we’re hooking you up with an exclusive 20% discount on your tickets! Here’s how to secure your spot and save big:

1. Fill out this google form with your basic info.

2. Get your 20% discount code delivered straight to your inbox.

3. Register for KubeCon + CloudNativeCon North America 2024 and enjoy the savings!

Don’t sit on this—grab your discount before it’s gone!

Co-Located Events at KubeCon + CloudNativeCon North America 2024

Expand your KubeCon + CloudNativeCon experience by joining co-located events, each tailored to specific interests within the cloud-native realm:

• AppDeveloperCon: Focuses on tools and techniques for building cloud-native applications.

• ArgoCon: Dive into Argo workflows, events, and continuous delivery for Kubernetes.

• BackstageCon: Explore Backstage’s developer portal and best practices for engineering platforms.

• Cilium + eBPF Day: A deep dive into Cilium and eBPF for networking and security.

• Cloud Native & Kubernetes AI Day: Discuss AI and ML workloads on Kubernetes.

• CloudNative StartupFest: Networking and insights for startup founders and innovators.

• Cloud Native University: Educational sessions on the fundamentals of cloud-native technologies.

• Data on Kubernetes Day: Explore data management practices and tools for Kubernetes.

• EnvoyCon: Dedicated to the Envoy proxy community, focusing on networking and observability.

• Istio Day: Learn about Istio and service mesh technologies in cloud-native environments.

• Kubernetes on Edge Day: Explore the role of Kubernetes in edge computing.

• Observability Day: A day centered around observability tools and practices in the cloud.

• OpenFeature Summit: Discussions on feature flagging and experimentation in cloud-native setups.

• OpenTofu Day: Open-source infrastructure management and IaC best practices.

• Platform Engineering Day: Dedicated to platform engineering in cloud-native environments.

• WasmCon: Focuses on WebAssembly and its role in cloud-native development.

KubeCon + CloudNativeCon North America 2024 Unofficial Conference Parties

After a day of learning and networking, unwind at the official conference parties.

• Check out Conference Parties for the latest info on social events.

• Check out events on lu.ma too to keep exploring.

Speakers & Tracks at KubeCon + CloudNativeCon North America 2024

KubeCon + CloudNativeCon features keynotes from industry leaders and breakout sessions across multiple tracks, including:

• Kubernetes Operations: Topics around the deployment, scaling, and security of Kubernetes.

• AI and Machine Learning: How AI and ML are transforming Kubernetes.

• Edge Computing: Use cases and solutions for Kubernetes at the edge.

• Cloud Security: Tools and practices to ensure security in a cloud-native environment.

Planning for KubeCon + CloudNativeCon North America 2024 Utah Logistics

Stays near Salt Lake City

Salt Lake City has several convenient options for accommodation. Whether you prefer hotels or Airbnbs, here are a few recommendations:

• Hotels: As of now, all hotels on booking.com as well as most other websites are sold out right now.

• Airbnb: Airbnb still has a healthy number of properties available in the city, although most of them are not within walking distance of the Convention Center.

• 

Airports nearby

• Salt Lake City International Airport (SLC) is the nearest airport, just 5 miles from downtown.

Visiting Salt Lake City

Restaurants

Salt Lake City offers a diverse culinary scene. Recommended spots include:

• Red Iguana: Known for authentic Mexican cuisine.

• The Copper Onion: A top choice for American fare.

• Takashi: Excellent sushi in the heart of the city.

After Hour Locations

• Beer Bar: Perfect for a laid-back evening with a variety of beers.

• The Bayou: Offers an extensive selection of brews and Cajun-style food.

Coffee Shops

Need a caffeine boost? Check out:

• La Barba Coffee: Known for artisanal coffee.

• Publik Coffee Roasters: A great spot to unwind with quality brews.

Weekend Plans

Consider exploring Utah’s natural beauty over the weekend. Popular options include:

• Bonneville Salt Flats: A unique desert landscape.

  

• Big Cottonwood Canyon: Ideal for scenic drives and hikes.

  

Visit Doctor Droid Booth - Q45

Doctor Droid is a root cause identification slack bot which can assist on-call engineers diagnose incidents and find root cause really fast. All you need to do is reply to your alert message in slack by tagging the bot. If you are interested to get a demo and explore more about Doctor Droid, visit us at Booth Q45 in the venue!

Pre-book a meeting with us using this link.

Stop by Booth Q45 to discover how Doctor Droid’s automated RCA can help you debug & fix your production issues faster! What else is up for grabs at the event?

• Puzzles & Goodies: Test your mental muscle and win some amazing gifts.

• $500 Doctor Droid Credits: Show this blog at our booth to receive $500 in Doctor Droid credits!

Sentry is one of the best tools in the industry right now for error and exception tracking. It has high quality SDKs across the stack and has great integrations as well as a powerful dashboard for users to learn about an exception in the code. You can read more about Sentry here.

Debugging an exception in Sentry

An exception could arise because of n-different reasons. It could be because of code change in the place where the exception came, it can be a code change upstream, it could be bad user input or it could even be just an edge case that hadn’t appeared until now. In case of canary deployments, it could get even trickier as different containers or users could be on different versions of the code.

In production, investigation of a simple looking Sentry issue can span across multiple data sources & contexts:

• Your infrastructure & deployment resources like Kubernetes

• Your code repository to check the code for recent changes or even analysing the flow of data

• Your database/logs to check for user entered data

• Discussion with internal team members regarding expected behaviour

Using Doctor Droid to debug the issue

Initiating an investigation

You can start investigation of an alert directly from the home page which has all the recent alerts.

Once an investigation is created, this is what it looks like.

Here are the key elements of the investigation panel:

• The alerts that it's investigating

• The recommended investigation strategy and preliminary data for your evaluation

• Additional panels related to related investigations or alerts

Investigation Strategy

Now what is it that it's able to fetch till now? Depending on the alert context, the platform recommends different steps.

It identified that the first thing that it should check is the stack trace itself in Sentry. So it goes and fetches the stack trace from Sentry, including the culprit.

Once it's able to fetch that, it goes and looks for the recent code changes for the same stack trace that it identified within the your GitHub repository. It shows you the recent commits, the URLs, and you can go and check if this this is potentially something that was done in the last couple of days.

You can also then go and check if within your Kubernetes infrastructure if there was any recent deployment that could be correlated with it. Given that this is related to the prototype instance, we can see that here there is a couple of releases for prototype in the last one hour. So it could potentially be the reason for this instance, for this alert to actually come up. And now you have all the data here.

You can ask it for more data, chat with it, ask for more data.

And what's what's also good is that it'll give you references to either existing playbooks, dashboards, or any other data point that your system already has. So we have integrations with almost every tool that your monitoring and observability stack would potentially have.

And we also have options for you to self host these integrations so that the data remains within your own plane.

Try it today

If if this is something that looked exciting to you, we have a lot more demos coming up, like how do you auto investigate an API latency alert, or how do you investigate CPU utilisation alerts on your databases.

Visit www.drdroid.io and try it out for yourself for your own stack. We have a free trial that we provide for the tool. So if you have any questions, please reach out to us, and we'll be happy to answer.

For Meta, reducing downtime has been crucial to ensuring millions (or should I say Billions?) of users have a seamless experience. Recently, Meta shared about one of their internal platforms that helped reduce MTTR by ~50% for critical alerts.

This blog explores how Meta accomplished this by leveraging AI, machine learning & runbook automation to transform its incident response processes, making them faster and more efficient.

Let's dive into the key components that enabled this efficiency gain.

Objective

Imagine trying to find a needle in a haystack—blindfolded. That’s what incident management can feel like without the right tools. Meta’s objective was to take off that blindfold and make the process of finding and fixing problems as swift and accurate as possible. Their goal? Cut down the Mean Time to Resolution (MTTR) by half, so that when things go wrong, they can be fixed faster than you can say “downtime.”

By harnessing the power of AI and machine learning, Meta aimed to automate the grunt work of incident management—spotting issues, figuring out what’s broken, and fixing it—all without requiring a superhero on standby. This isn’t just about cool tech; it’s about making sure users experience as little disruption as possible, turning potential disasters into minor hiccups that barely anyone notices.

What Meta built?

Alright, let's peek under the hood of Meta's incident-busting machine. They didn't just slap on a new coat of paint; they rebuilt the entire engine. Here are the three turbocharged components that turned their incident response from a clunky old jalopy into a sleek, AI-powered sports car:

Component 1: Automated Runbooks

Remember those old-school detective novels where the brilliant sleuth solves the case with a magnifying glass and a pipe? Well, Meta created a digital Sherlock Holmes, minus the pipe smoke. They call it Dr. Patternson (Dr. P for short), and it's like having a tireless detective on call 24/7.

Dr. P is an automated runbook system that encodes expert knowledge into executable investigation workflows. It's like giving every on-call engineer a cheat sheet written by the smartest person in the room. With its own SDK, simplified APIs, and ML algorithms, Dr. P can quickly analyze data, correlate events, and generate findings faster than you can say "Elementary, my dear Watson."

But wait, there's more! Dr. P comes with a fully managed platform that deploys these runbooks, monitors for issues, and even triggers investigations automatically when an alert fires. It's like having a whole team of digital detectives working round the clock, leaving no log unturned.

Component 2: Analysis Algorithms Service

If Dr. P is the detective, then the Analysis Algorithms Service is its time machine. This nifty piece of tech allows Meta's engineers to zoom through vast amounts of data at warp speed. Picture this: You've got more data than stars in the sky, and you need to find that one glowing red dot that's causing all the trouble. That's where this service comes in. It's packed with ML algorithms for dimensional analysis, time series analysis, anomaly detection, and more. But the real magic is in its pre-aggregation layer, which shrinks datasets by up to 500 times! It's like compressing the entire library of Congress into a pocket-sized book, without losing a single word.

The result? Insights that used to take hours now pop up in seconds. It's so fast, you might think it's predicting the future. (Spoiler alert: it's not. That's still on the roadmap for 2025.)

Component 3: Event Isolation Assistance

Last but not least, we have the Event Isolation Assistance. Think of it as a super-smart metal detector for that proverbial needle in a haystack.

This system uses ML models to rank thousands of events and pinpoint the root cause of an incident. It's like having a psychic on your team, except this one actually works. By focusing on config-based and code-based isolation, it can filter out 80% of the uninteresting events during an active investigation. That's right, it separates the wheat from the chaff, leaving engineers with a much smaller, much more suspicious pile of events to investigate.

But it doesn't just point fingers. The system provides annotations explaining its reasoning, making it transparent and trustworthy. It's like having a really smart friend who not only tells you the answer but also shows their work.

Guided Investigations:

Sometimes, even the smartest AI needs a human touch. That's where Guided Investigations come in. Think of it as a choose-your-own-adventure book, but for fixing tech problems.

These decision trees provide step-by-step workflows that help investigators narrow down the root cause of an issue. It's like having a seasoned pro whispering in your ear, guiding you through the digital labyrinth. By combining automated workflows with human expertise, these guided investigations can tackle complex issues that might stump a fully automated system.

And the best part? They're right where you need them, integrated with Meta's detection systems. It's like having a tech support genie, ready to pop out whenever an alert goes off. No need to rub a lamp – just click a button!

Current state of Investigations at Meta:

So, where does Meta stand now in their AIOps journey? Let's just say they've gone from digital chaos to zen master status.

Today, Meta's foundational systems are more popular than cat videos (well, almost). Hundreds of teams have adopted them, running over 500,000 analyses per week. That's more check-ups than a hypochondriac gets in a lifetime!

The impact? A cool 50% decrease in MTTR for critical alerts across the company. It's like they've upgraded from a horse-drawn buggy to a supersonic jet when it comes to fixing problems. Take the Ads Manager team, for instance. They've gone from spending days investigating issues to resolving them in minutes. It's like they've traded in their magnifying glass for a high-powered microscope with AI-assisted focusing.

Implementing your Own Dr. Patternson using Doctor Droid:

If you want to implement a solution like Dr. Patternson within your team without investing the time or cost like Microsoft, you might want to explore Doctor Droid.

Doctor Droid is a AI-assisted intelligence platform to help engineering teams reduce investigation time of production issues by 10x. Here's what you can do with Doctor Droid:

(a) Codify your investigation mental models:

Doctor Droid PlayBooks is an Open-Source On-call automation platform. With one click, you can run your investigation steps and have all the diagnosis data across all tools, directly fed in response to your alerts.

(b) Leverage your past knowledge to get intelligent suggestions:

Doctor Droid's AIOps Platform can provide your on-call engineers with intelligent recommendations by leveraging the knowledge that you already have accessible in your system over the past time.

Both A & B combined, you are effectively going to end up with an equivalent of Dr. Patternson.

Try it out today by signing up here!

Conclusion:

Meta's AIOps journey has transformed their incident response from a digital firefight into a well-oiled machine. Let's break down the impressive results:

• 50% reduction in Mean Time to Resolution (MTTR) for critical alerts across the company

• Over 500,000 automated analyses run per week

• 80% of uninteresting events filtered out during active investigations

• Ads Manager team improved investigation time from days to minutes

• Nearly 50% of previously manual investigations now automated

These statistics paint a picture of a dramatically more efficient system, but what does it mean in the real world?

For Meta, it means:

• Fewer service disruptions for millions of users

• Faster resolution when issues do occur

• Engineers spending less time on repetitive tasks and more on innovation

• Improved overall system reliability and user experience

The secret sauce? A combination of:

1. Automated Runbooks (Dr. Patternson)

2. Analysis Algorithms Service

3. Event Isolation Assistance

4. Guided Investigations

This powerful quartet has turned Meta's incident response into a symphony of efficiency, conducting a harmonious blend of AI automation and human expertise.

As we look to the future, Meta's AIOps journey serves as a beacon for the tech industry. It shows us that with the right tools and approach, we can tame the chaos of complex systems and create a more reliable digital world. So the next time you scroll through your feed without a hitch, remember - there's a good chance Meta's AIOps team had a hand in making that seamless experience possible.

Big Tech companies often have scale enough to justify allocating resources to building internal tools. In this blog, we discuss about RCACoPilot -- an automated incident classification and investigation engine built by Microsoft to improve the lives for their on-call engineers.

Less than a year ago, Microsoft published a research paper discussing RCACoPilot. It's a longish paper ~ 16 pages, so I decided to condense it into a shorter blog.

Context

Picture this: You're an on-call engineer at Microsoft. It's 3 AM, and suddenly, alerts start blaring. Something's wrong with the email service (which delivers over 150 billion messages daily) that millions of people rely on. Your job? Figure out what's causing the issue and fix it ASAP. No pressure, right?

This scenario plays out all too often at Microsoft & at most companies. A company's systems are only getting more complex by the day, and on-call engineers were drowning in a sea of alerts, logs, and metrics. This often leads to escalations and time being spent on tickets rather than planned work. They needed a way to streamline the process and a way to quickly make sense of all this information and zero in on the root cause of problems. That's what RCACoPilot tries to solve for them.

Why should you even read this article? The Results

Let's cut to the chase: How well does RCACopilot perform & is it even worth reading about their approach? Their performance is pretty impressive, as it turns out.

Tested on 653 real-world incidents from Microsoft's email service (which handles about 150 billion messages daily), RCACopilot achieved:

• 76.6% accuracy in predicting root cause categories

• A Macro-F1 score of 0.533, showing good performance across various incident types

• Significantly reduced MTTR:

  • Auto-diagnosis ran with an average time of 1-10 minutes depending on the complexity of the incident handlers (more on it below).

  • An average classification time of just 4.2 seconds per incident.

[Quick note on that Macro-F1 score: It's a measure that gives equal importance to each category, regardless of how often it appears. A score of 0.533 tells us that RCACopilot performs well across various incident types, not just the common ones. This is crucial in a system where rare, critical issues are just as important as frequent, minor ones.]

These numbers outperformed all baseline methods, including traditional machine learning approaches and non fine-tuned GPT models.

But the real proof is in the deployment. Parts of RCACopilot have been in use at Microsoft for over four years, across more than 30 teams. On-call engineers report significant time savings in incident management tasks, from diagnosis to mitigation.

What this basically means is that in less than a few minutes, most of the likely investigation steps are run, analysed and the engineers are told a likely incident Root Cause category.

RCACoPilot -- The Architecture

Now, let's peek under the hood of RCACoPilot. Think of it as a super-smart detective for computer problems. Here's a simple breakdown of how it works:

1. Diagnosis Identifier: In the original paper, this is part of what they call the "Diagnostic Information Collection Stage". It is responsible to take in an incident from their alerting tool, parse the incident context and match it to the existing playbooks (called Incident Handlers) basis the mapping logics defined.

2. Diagnosis Data Fetch & Summarisation: The playbook identified in the previous section is custom defined by the on-call engineers (called OCEs). The system now executes all the steps pre-configured in their playbook -- from fetching logs & metrics to running more steps.

3. Incident Predictor: Now on top of the diagnosis data that's fetched, a couple of things are done: converting it into an embedding and comparing the embedding with past embeddings. These embeddings are now leveraged to identify the potential Root Cause for this issue. This corresponds to the "Root Cause Prediction Stage" in the paper.

The actual benefit of RCACoPilot lies in how the entire pipeline works out. When an alert comes in, the Incident Handler kicks into gear, following predefined playbooks to gather relevant data. This could involve querying databases, analyzing log files, or even running diagnostic scripts.

Component 1: Diagnosis Identifier

Incident Parser:

The Incident is the entry point of RCACoPilot. It's connected to the existing alerting systems in the company through triggers or webhooks. It then parses the incident to identify the key entities of interest.

Handler:

A Handler is set of steps pre-defined to be run for a specific type of investigation. This is effectively a programmatic SOP for a certain type of issue. These are NOT AI GENERATED. These are something that every on-call engineer documents -- the investigation strategies for a type of issue within their service. Types of Actions

The Incident Handler uses several types of actions to investigate and respond to incidents:

1. Scope Switching Action: This allows the handler to adjust its focus dynamically. It might start by looking at a single server, then expand to an entire cluster if needed.

2. Query Action: Think of this as the handler's way of asking questions. It can pull data from various sources like databases, log files, or even run scripts to gather system information. The results come back as key-value pairs, giving the handler structured data to work with.

3. Mitigation Action: Sometimes, the handler can take steps to address the problem directly. This could involve restarting a service, clearing disk space, or even calling in specialized teams for complex issues.

This is what an incident handler looks like for too many messages stuck in the delivery queue alert.

Component 2: Diagnosis Data Fetch & Summarisation

In this part, the system is executing a series of commands as per the incident handler definition.

• It has access to different internal / external tools with the relevant information.

• Each step in handler is structured and mapped to a technical task (be it an API call, a log fetch, a metric fetch or any other task).

• The system automatically interacts with every different data tool and then fetches it.

These are some of the data points that the system can fetch from the handlers (playbooks):

• Logs: Application logs, system logs, security logs.

• Metrics: Performance metrics and resource utilization stats.

• Traces: Detailed records of how requests flow through the system.

• Configuration data: Current system settings that might be relevant.

Component 3: Incident Predictor

The ML part of RCACopilot happens in what we call "Incident Predictor". This is where it uses Large Language Models (LLMs) to make sense of all the data collected by the Incident Handler.

Here's how it uses LLMs to analyze incidents:

1. Summarization: First, the LLM takes all the diagnostic information collected and creates a concise summary. This step is crucial because it condenses vast amounts of data into something manageable for both the AI and human engineers.

2. Similarity Matching: Next, RCACopilot creates an embedding to represent incidents as points in a high-dimensional space and then uses a Nearest Neighbour Search Algorithm. ELI5: similar incidents will be close to each other in this space.

   Using this technique, RCACopilot finds past incidents that are most similar to the current one. This is important because similar past incidents can provide valuable clues about the current problem.

3. Chain-of-Thought Prompting: This is where things get really interesting. RCACopilot uses a technique called "Chain-of-thought" prompting. Instead of just asking the LLM "What's the root cause?", it prompts the model to think through the problem step-by-step, much like a human engineer would.

   It does this by showing the LLM examples of how similar past incidents were solved. This is akin to training a junior engineer by walking them through past case studies before asking them to solve a new problem.

4. Root Cause Prediction: Based on this careful analysis, the LLM then predicts the most likely root cause of the incident. But it doesn't stop there.

5. Explanation Generation: Crucially, the LLM also generates an explanation for its prediction. This isn't just a black box spitting out an answer - it's more like a colleague explaining their reasoning. This explanation helps human engineers understand and verify the AI's conclusion.

Feedback loop: While it doesn't learn in real-time, feedback from engineers can be used to periodically retrain and improve the model. This means RCACopilot can get better over time, learning from each incident it analyzes.

By leveraging the power of LLMs in this way, RCACoPilot can quickly analyze complex incidents, drawing insights from vast amounts of data and past experiences. It's like having an AI assistant that has seen every incident your organization has ever faced, can think through problems step-by-step, and can clearly explain its reasoning. This not only speeds up incident resolution but also helps engineers learn and improve their own diagnostic skills.

Implementing your Own RCACoPilot using Doctor Droid:

If you want to implement a solution like RCACoPilot within your team without investing the time or cost like Microsoft, you might want to explore Doctor Droid.

Doctor Droid is a AI-assisted intelligence platform to help engineering teams reduce investigation time of production issues by 10x. Here's what you can do with Doctor Droid:

(a) Codify your investigation mental models:

Doctor Droid PlayBooks is an Open-Source On-call automation platform. With one click, you can run your investigation steps and have all the diagnosis data across all tools, directly fed in response to your alerts.

(b) Leverage your past knowledge to get intelligent suggestions:

Doctor Droid's AIOps Platform can provide your on-call engineers with intelligent recommendations by leveraging the knowledge that you already have accessible in your system over the past time.

Both A & B combined, you are effectively going to end up with an equivalent of RCACoPilot.

Try it out today by signing up here!

Setup Phase

Step 1: Ensure a running instance of Postgres and Redis on the local machine. Installation guides are available for different machine types to set up Postgres and Redis. Alternatively, Docker can be used for the same purpose.

Step 2: Use the DB Docker Compose file located in the Docker folder to set up Postgres and Redis.

Step 3: Verify the setup by checking Docker Desktop to see if both instances are running.

Build Phase

Step 4: Create all the tables and relations between them that the API server will use by running the command

python manage.py migrate

Step 5: Confirm the creation of all tables and relations in the Postgres DB using a tool like Postico.

Run Phase

Step 6: Start the API server on the local machine using the below command

python manage.py runserver

The server will run on port 8000, but the port can be changed according to preference.

Step 7: Go to the web folder and use the following command to bring up the React application

npm start

Step 8: Inside the web folder, use the command shown to start the Nginx server. This ensures the React application can connect with the Django application.

nginx -c $PWD/nginx.local.conf -g "daemon off;"

Step 9: Once both servers are running, open a browser and go to localhost to see the running application.

Testing Changes

Step 10: Test changes by creating a playbook and an HTTP task. For example, remove the method field in the API task and save the changes.

Step 11: Django will load the changes and restart the server. Refresh the application platform and try to create an API task. The method field should be missing.

Step 12: To restore the method field, undo the changes, save, and refresh the page. Try to create the task again.

For further assistance with local setup, refer to the details mentioned in the github project. The contribution page includes a link to the playbook architecture and local setup guide.

A playbook consists of tasks. A task is an instruction that's executed through the portal. Let's create a task.

A task could involve fetching a metric from CloudWatch or running a kubectl command on a server. Let's create a new task to fetch logs from CloudWatch.

Add notes to the playbook. A note is a custom guideline for the user, related to the playbook or a specific step. Add a note indicating that this task fetches logs.

Hover over a step to view the note. It's also possible to add multiple tasks in a step. For instance, fetch a Datadog metric. After adding the task, check the metric.

Add variables to the playbook. For example, add a service as a variable. To use the variable, enter a dollar sign followed by the variable name.

Add a step with conditions. A condition is a rule that determines whether a certain action should be taken. For example, check logs from CloudWatch and check if the row count equals six. Then add a task. Add a metric and run the task.

Save the playbook. Execute the playbook and observe the results with the condition. If the condition isn't met, the step isn't recommended and the next step is executed.

These are all the core components of a playbook.

Consider a scenario where a critical deployment is planned. As a developer, you would want to track a set of logs and metrics continuously after the deployment.

In large-scale deployments, monitoring is crucial. Tight thresholds and noisy services often lead to frequent alerts. If something breaks, the process starts over, monitoring everything again with tight thresholds. This can be a hassle. To simplify this, set up a post-environment monitoring workflow.

Setup a workflow

Step 1: Navigate to the Doctor Droid Playbooks platform and create a new workflow for this purpose.

Step 2: Select a trigger with the API trigger and choose a playbook which includes tasks for querying logs & metrics that you would want to track post the deployment.

Step 3: Execute this playbook and publish its summary in Slack.

Step 4: Select the appropriate channel and set up a cron schedule. For instance, set it to run every minute.

Step 5: Save the cron schedule and save the workflow

Test the workflow

Step 1: Copy the API trigger code and run from terminal. You'll receive a workflow execution ID.

Step 2: Observe the successful workflow execution with the provided ID.

Step 3: Once the workflow executes, the output of the metrics and logs from the playbook configured in it.