How Cognition evals, routes, and guarantees agent work
Russell Kaplan is the President of Cognition , the $26 billion company behind Devin, the AI software engineer. He started his career in machine learning at Tesla's Autopilot team, then led the ML org at Scale AI, which had acquired his computer vision startup Helia. In this conversation, Russell unpacks why more and more engineering work is becoming "intelligence saturated," how the sidekick architecture inside Devin Fusion cuts cost without cutting quality, and why Cognition is underwriting a $10 million productivity guarantee.
🎧 Watch the full conversation on YouTube, or listen & subscribe on Apple Podcasts or Spotify.
What We Learned
Model quality got so good that speed and cost became the bottleneck
Russell noted that frontier models jumped so much that "more and more of the tasks in software engineering are getting intelligence saturated." Most models can now handle them well. Developer priorities shifted with it, going from "wait, I need to be using the best possible model" to "holy cow, I'm spending so much money on my coding agents."
However, adversarial, competitive workloads remain the exception to this rule. If you're trading against a counterparty, you still want the most frontier intelligence you can get. For most production software, teams will likely start asking how to bring the cost down once quality clears the bar
Why Cognition created FrontierCode
When Cognition launched Devin, it scored in the teens on SWE-bench. Now, in Russell's words, "SWE-bench is totally saturated." Looking for a harder benchmark, Cognition found something evals weren't measuring at all: whether code that's technically correct is code you'd actually want to merge.
"Would you actually merge it? Would you feel happy? Would this improve the quality of your code base?" Those questions became the basis for FrontierCode, the harder benchmark Cognition built to replace it.
Devin triages its own team's Slack channels
Recommended by LinkedIn
Cognition runs Devin proactively. It's wired into a bunch of the company's internal Slack channels, triaging every message and deciding whether it needs to chime in or dig further.
The agent also carries context on who owns what across the codebase and company, so it can point the right person, or itself, at the right change.
Other Topics Discussed
Timestamps
People & Tools Mentioned During This Episode
Get More Max Agency
Hosted by Harrison Chase, CEO of LangChain, each episode goes deep with the builders designing, deploying, and learning from real agent systems in the wild. From architecture decisions to evals, tooling, and failure modes, Max Agency is for people who want to understand what it really takes to build useful agents.
