incident.io

It took us a few months to build an AI agent that could diagnose an incident.

It took us two years to build one we'd actually trust during an incident.

That gap has changed how we think about building with AI.

Prototypes are incredibly convincing now. Give a model access to your code, telemetry and incident history, ask it what broke, and you can get something surprisingly good back in an afternoon.

The problems start when "surprisingly good" needs to become consistently right.

For us, one of the hardest parts was simply figuring out how to measure that.

We ended up building a backtesting system that freezes old incidents in time and reruns Investigations against exactly the information that was available when the incident happened.

The "freezes in time" bit is harder than it sounds.

A post-mortem written three days later might contain the root cause. The fix is now sitting in the codebase. Someone eventually posted the answer in Slack. Let any of that leak into your test and suddenly your agent looks much smarter than it actually is.

And accuracy was only the first problem.

We needed to understand why it was wrong when it was wrong. We needed findings tied back to actual evidence. We added an adversarial agent whose job is to argue against the current hypothesis. We built ways for old findings to be discounted when new evidence contradicts them.

Then we discovered another problem: an accurate agent can still be incredibly annoying.

So now messages have to clear a bunch of checks before they're posted. Is this actually new information? Are responders already discussing it? Has the situation already been handled? Are we confident enough to interrupt people with it?

And then there's memory. Something the agent learns about your infrastructure today might be wrong next month because a service was renamed or a log format changed. So those memories have to be continually checked, updated and sometimes deliberately forgotten.

None of these problems are particularly obvious when the first prototype works.

That's probably the biggest lesson we've taken from building Investigations: calling an LLM is increasingly the easy part. Building the systems around it that let you measure, challenge, maintain and trust what it produces is where most of the engineering lives.

Martha wrote up a much deeper look at what we've learned building this over the last two years:

If Twilio goes down and our customers don't get paged, is that an  reliability problem?

We think so.

It's easy to define reliability around the systems you control. But customers don't care where your infrastructure ends and someone else's begins. They care whether the thing worked.

That's changed how we build and measure On-call.

For example, we run two telecom providers active-active. If one can't dispatch an SMS or phone call, we try the other. And when measuring notification latency, the clock doesn't stop until a provider actually acknowledges the notification.

We've found that framing reliability around customer experience, rather than ownership of individual systems, leads to some interesting engineering decisions.

We wrote about how we apply that thinking to our SLOs here:

We moved most of our reads to a Postgres replica. Then things started disappearing.

Not actually disappearing. But occasionally, one of our workers would try to fetch a record that we knew existed and get a NotFoundError.

The problem was a race we hadn't fully accounted for.

A request would write to the primary → publish an event → a worker would pick up that event → query the replica.

Sometimes our workers were fast enough to process the event before the replica had caught up with the write.

It only happened on ~0.1–0.5% of messages, but that was enough to make moving more traffic to the replica risky.

Our solution was to use Postgres Log Sequence Numbers (LSNs).

Every message we publish gets stamped with the primary's current LSN. When a worker picks it up, we compare that against the replica's LSN.

If the replica has caught up, process the message.

If it hasn't, put the message back on the queue and try again later.

That also gave us a useful backpressure mechanism: if the replica starts falling behind during a period of heavy load, work simply queues until it catches up instead of spilling back onto the primary and potentially making things worse.

We eventually extended the same idea to our API and made the replica the default destination for reads across our codebase.

Today, more than 60% of our read queries go to the replica, and we cut CPU utilization on our primary database roughly in half.

We wrote up the whole journey, including the consistency issues we hit, LSN stamping, connection pool routing, and some surprisingly effective optimizations:

For those running Postgres at scale: how are you handling read-after-write consistency with replicas?

Like a lot of companies, we'd accumulated a familiar data problem.

People had more questions than our data team could reasonably answer. Dashboards helped, but mostly when someone's question happened to match a chart we'd already built.

So the obvious next step was: give an LLM access to the warehouse and let people ask it questions directly.

Except LLMs are already pretty good at writing SQL. What they're bad at is knowing all the stuff that isn't written down.

Which join path should you use? Which filter is canonical? Which of four ARR-ish columns is actually the right one? Why is this table technically correct but definitely not the one you should query?

Basically, all the institutional knowledge a good analyst picks up over time.

So instead of building a text-to-SQL wrapper, we tried to build something closer to how we'd onboard a new data hire.

We created "themes" containing guidance for different business concepts, gave the agent vetted examples of previously answered questions, and connected those to our semantic layer and warehouse schema.

Rather than dumping the entire warehouse into context, the agent progressively loads only the guidance and schema relevant to the question it's answering.

We also put a fairly paranoid query layer between the agent and BigQuery. Every query gets dry-run, checked against an allow-list, rejected if it'll scan too much data, and logged along with the guidance the agent used to produce it.

One lesson that became pretty clear: the guidance itself has to be treated like code.

If your schema changes but the agent's instructions don't, or two pieces of guidance contradict each other, you can end up with something much worse than an error: SQL that looks completely reasonable but is quietly wrong.

We're now a couple of months in, and more than half the company uses the agent weekly.

We wrote up the architecture, how we bootstrap trusted examples from existing dashboards, how we stop the guidance from rotting, and a bunch of things we'd do again:

For anyone building internal data agents: how are you handling the institutional knowledge that isn't captured in your schema or semantic layer?

That was the goal, at least.

At , we process ~240 million messages a day across 800+ topics and 1,000+ subscriptions. For years, all of that flowed through Pub/Sub.

Pub/Sub has been extremely reliable for us, but having every event pass through a single provider left us with a pretty obvious problem: it was still a single point of failure.

So we decided to see if we could remove it without replacing it.

We added NATS as a second broker and built an active-active setup that normally splits traffic 50/50 between the two. If either broker starts failing, circuit breakers automatically shift publishes to the other.

The subscriber side ended up being the more interesting problem. We didn't want to double our workers just because we now had two queues, so we built a scheduler that shares the existing concurrency budget and prioritizes whichever broker has the oldest waiting message.

Once everything was running in production, there was really only one way to know whether we'd built this properly: break it.

We gradually shifted 100% of production traffic away from Pub/Sub. No customer impact, no engineers paged, and our On-call escalation events continued processing normally.

We wrote up the architecture, the tradeoffs we made, how the scheduler works, and what happened during the production chaos test here:

Curious how other teams running event-driven systems think about broker redundancy. Do you run multiple brokers/providers, or accept the broker itself as part of your failure domain?

That was the goal, at least.

At , we process ~240 million messages a day across 800+ topics and 1,000+ subscriptions. For years, all of that flowed through Pub/Sub.

Pub/Sub has been extremely reliable for us, but having every event pass through a single provider left us with a pretty obvious problem: it was still a single point of failure.

So we decided to see if we could remove it without replacing it.

We added NATS as a second broker and built an active-active setup that normally splits traffic 50/50 between the two. If either broker starts failing, circuit breakers automatically shift publishes to the other.

The subscriber side ended up being the more interesting problem. We didn't want to double our workers just because we now had two queues, so we built a scheduler that shares the existing concurrency budget and prioritizes whichever broker has the oldest waiting message.

Once everything was running in production, there was really only one way to know whether we'd built this properly: break it.

We gradually shifted 100% of production traffic away from Pub/Sub. No customer impact, no engineers paged, and our On-call escalation events continued processing normally.

We wrote up the architecture, the tradeoffs we made, how the scheduler works, and what happened during the production chaos test here:

Curious how other teams running event-driven systems think about broker redundancy. Do you run multiple brokers/providers, or accept the broker itself as part of your failure domain?