Automated Incident Investigation: A Pragmatic Look

It is 3:14 AM. Your phone buzzes on the nightstand. The screen glares in the dark: P1 - Payment Gateway 502 Bad Gateway.

You drag yourself out of bed, open your laptop, and start the ritual. You check Datadog for spikes, dive into AWS CloudWatch for logs, cross-reference the latest GitHub commits, and try to build a mental map of what broke in the last four hours. It is a painful, manual process of correlating telemetry from a dozen different sources.

This week, the industry promised to make this pain go away. AWS announced the general availability of their DevOps Agent, designed for automated incident investigation. Anthropic released Claude Code Routines for unattended dev automation. The pitch is alluring: an autonomous teammate that wakes up at 3 AM, correlates the logs, traces the dependencies, and tells you exactly what went wrong.

But before we hand the keys to the kingdom over to automated systems, we need to take a step back. I have spent my career migrating systems from bare metal to Kubernetes, and if there is one truth in operations, it is this: adding a layer of opaque complexity on top of a broken system does not fix the system. It just hides the cracks until they shatter.

Let's cut through the noise and look at what is actually happening under the hood of these new tools, where they fall short, and how we can pragmatically integrate them without losing control of our infrastructure.

The Reality Check: Complexity Breeds Complexity

We have spent the last decade breaking apart perfectly functional monolithic applications into sprawling microservice architectures. We told ourselves this would make teams faster and deployments safer. Instead, we built distributed systems so complex that no single human being can hold the entire architecture in their head.

Think of your infrastructure like a massive commercial harbor. Ten years ago, we had one giant cargo ship (the monolith). If it stopped moving, you checked the engine. Today, we have ten thousand small speedboats (microservices) constantly tossing packages to each other at sixty miles per hour. When a package drops into the ocean, figuring out which boat threw it and which boat missed it is a logistical nightmare.

To solve this, we added service meshes, distributed tracing, and centralized logging. We built massive dashboards. And now, because the dashboards are too complex to read, we are introducing automated incident investigation tools to read the dashboards for us.

This is the reality check: we are trying to solve architectural complexity with operational magic.

Tools like the AWS DevOps Agent and Claude Code Routines are impressive feats of engineering. They can parse gigabytes of telemetry faster than you can open your browser. But they are not magic. They are correlation engines relying entirely on the quality of the data you feed them.

The Core Problem: The Visibility Gap

The real bottleneck in modern incident response is not a lack of analytical power; it is a fundamental lack of runtime visibility.

As highlighted in a recent Lightrun report, engineering teams are increasingly in the dark regarding how these new automated assistants understand runtime state. Software runs, but sometimes it doesn't, and the gap between static code analysis and dynamic runtime behavior is massive.

Let's use a restaurant kitchen analogy.

Imagine a busy restaurant where customers are suddenly complaining that their soup is cold.

If you ask an automated assistant to investigate, it will read the recipe (your source code), check the ingredient delivery logs (your CI/CD pipeline), and read the waiter's notes (your application logs). Based on this, it might conclude that the recipe is correct and the ingredients arrived on time, so it has no idea why the soup is cold.

What the assistant cannot do is walk into the kitchen, put a thermometer in the pot, and realize the stove's pilot light went out.

This is the runtime visibility gap. Automated incident investigation tools excel at static correlation. They can tell you that a deployment happened at 2:00 AM and memory spiked at 2:05 AM. But if your application does not emit clear, deterministic signals about its internal state during execution, the agent is just guessing fast.

Under the Hood: How Automated Triage Actually Works

Before we rely on a tool's abstraction, we need to understand the plumbing.

When AWS says the DevOps Agent "analyzes incidents by learning application relationships," it sounds like science fiction. It is not. It is applied graph theory and structured querying.

Here is what happens when a CloudWatch alarm triggers:

1. Event Ingestion: The alarm sends a payload via EventBridge to the agent.
2. Context Gathering: The agent parses the resource ID (e.g., an EC2 instance or ECS service) from the payload.
3. Graph Traversal: It queries AWS Resource Explorer or X-Ray to find dependencies. "What database does this service talk to? What load balancer sits in front of it?"
4. Telemetry Fetching: It generates and executes queries against CloudWatch Logs and Metrics for those specific resources within the incident's time window.
5. Summarization: It feeds the raw log errors and metric anomalies into a large language model (Amazon Bedrock) to generate a human-readable summary.

This is a highly useful workflow. It automates the exact grep and query commands you would run manually. But notice what is missing: it does not know anything you do not know. It only knows what you have explicitly logged and monitored.

Anthropic's Claude Code Routines operate on a similar principle for unattended dev automation. You can schedule it to run scripts, handle GitHub events, or trigger API workflows. But if a routine fails silently because a third-party API changed its response format and your code didn't catch the exception, the routine will just hang or report success incorrectly.

Comparing the Approaches

Let's look at how these different approaches stack up against each other in a production environment.

The Pragmatic Solution: Fundamentals First

The pragmatic approach to automated incident investigation is to treat these tools as powerful aggregators, not decision-makers. You do not want a system automatically rolling back a database schema at 3 AM because it misinterpreted a network timeout as a bad migration.

The best code is code you don't write, and the best automated action is the one you don't need to take because the system is resilient.

Before you adopt any automated triage tool, you must fix your fundamentals. If your system is a black box to you, it will be a black box to the agent.

Step 1: Deterministic Telemetry

Stop logging generic errors. If your logs say Error: Connection timeout, the agent will tell you "There was a connection timeout." That is useless.

You need structured logging with clear context. Why did it time out? What was it trying to do?

{
  "level": "error",
  "service": "payment-gateway",
  "action": "charge_customer",
  "customer_id": "cus_12345",
  "dependency": "stripe-api",
  "error_code": "timeout_5000ms",
  "message": "Failed to reach Stripe API after 5 seconds during charge execution."
}

When an automated agent reads this log, it can provide a summary that actually helps: "The payment gateway is failing because the Stripe API dependency is timing out after 5 seconds."

Step 2: Read-Only Boundaries

If you are going to use unattended automation or DevOps agents, you must enforce strict access boundaries.

Why? Because if you give an automated system write access to your infrastructure during an incident, you are handing a loaded gun to a blindfolded person. The agent should have permissions to read logs, query metrics, and list resources. It should never have permissions to mutate state (Update, Delete, Put*) without a human clicking "Approve."

Here is an example of the kind of IAM policy boundary you should establish before turning these tools on. We explicitly deny mutation actions to ensure the agent remains an observer, not an actor.

# Why we do this: We want the agent to investigate, not remediate.
# Remediation without human context leads to cascading failures.
Statement:
  - Effect: Allow
    Action:
      - cloudwatch:GetMetricData
      - logs:StartQuery
      - xray:BatchGetTraces
    Resource: "*"
  - Effect: Deny
    Action:
      - ec2:TerminateInstances
      - ecs:UpdateService
      - rds:DeleteDBInstance
    Resource: "*"

Step 3: The Human-in-the-Loop Workflow

Embrace the "Human-in-the-Loop" model. Use the agent to do the heavy lifting of data gathering, but reserve the remediation decision for the engineer.

What You Should Do Next

Technology is just a tool for solving problems. If you are struggling with incident response, do not start by installing a new tool. Start by looking at your system's observability.

1. Audit Your Logs: Pick a random incident from last month. Try to figure out what happened using only your logs, without looking at the code. If you can't do it, an agent won't be able to either. Fix your log context.
2. Map Your Dependencies: Ensure your services are emitting distributed traces (like OpenTelemetry). Automated investigation relies heavily on knowing which service talks to which.
3. Scope Your IAM Roles: If you are testing AWS DevOps Agent or Claude Code Routines, create strict, read-only roles. Never allow unattended mutation in production.

At the end of the day, the goal is not to replace the engineer. The goal is to let the engineer sleep until a problem actually requires human intuition. Build simple systems, emit clear signals, and use automation to surface those signals faster.

There is no perfect system. There are only recoverable systems.

Frequently Asked Questions