DX

94% of AI token consumption is now spent re-reading context. In our latest Q3 AI Impact Report, we found that cache-read tokens reached 94% of the total token mix over the past four quarters, while output tokens shrank. The data suggests AI spend is growing because the background context required to produce code is ballooning, not because models are generating more code.

“Whenever developers tell me coding agents slow them down, it usually comes down to one thing: the agent doesn't know what’s in their head.” — Max Kanat-Alexander Watch or listen to the full discussion with panelists Brian Houck, Eirini Kalliamvakou, and Margaret-Anne Storey: https://lnkd.in/gsdeVJS2

Median PR size is up 122% year-over-year. Teams have more code to review and the same review capacity. Published today, our Q3 AI Impact Report points to that volume as the main reason why first reviews are taking longer. Our research team suggests two other factors that could also be slowing first reviews: 1. AI-generated changesets take more effort to review, which ties up reviewers and pushes everything else back in the queue. 2. Reviewers hesitate to pick up large, unfamiliar, or AI-generated diffs because they're worried about missing a bug that's hard to spot. Read the full findings in our Q3 AI Impact Report: https://lnkd.in/gtD3CmBe

Last week in Seattle, Brian Houck brought together a group of senior engineering leaders for a conversation on how AI is changing the way engineering teams work. Set against the Seattle skyline in the Space Needle’s Skyline Room, the evening offered an early look at DX’s Q3 AI Impact Report, followed by thoughtful discussion, audience Q&A, and plenty of shared conversation over food and drinks. Thanks to everyone who joined us. The full Q3 AI Impact Report launches this week. Stay tuned. Photos by Into Dust Photography

Meta researcher Moritz Beller recently joined Brian Houck on the Engineering Enablement podcast to discuss diff authoring time (DAT), the metric Meta uses to measure the active time engineers spend building, testing, and reviewing a diff. Listen to the full episode to learn how Meta reduced its DAT by 40% year-over-year. https://lnkd.in/g52CHaji

An early finding from DX’s Q3 AI Impact Report: Teams generating high volumes of AI-authored code wait 105% longer for code reviews. Data from 500+ engineering organizations shows teams generating more than 80% AI-authored code wait more than twice as long for a first review, compared to teams writing predominantly human-authored code. Stay tuned for the full report.

“AI does not create a new need for good knowledge management; it dramatically increases the cost of poor knowledge management.” -Brian Houck Without reliable context, AI agents create three compounding operational risks: -Developer Experience: Forcing developers to diagnose errors and compensate for avoidable mistakes. -Cost: Accumulating unnecessary tokens and wasted inference spend. -Liability: Creating compliance and security exposure from stale inputs. To diagnose where these operational risks originate across engineering workflows, Brian and his co-authors introduce five properties for evaluating context quality for AI agents. Read more about the CAFE(S) framework: https://lnkd.in/g7cdbTjj

“Do we really need another framework?” is a fair question (especially when I’m the one bringing you another one). Hopefully the answer is yes, since last week we released CAFE(S). But this framework asks a specific and novel question: Is the context we give to our AI agents any good? This week, I unpack where CAFE(S) fits alongside the frameworks and evals you may already be using today. https://lnkd.in/gEwddN7j

“Can I use DX to implement the CAFE(S) framework?” CAFE(S), a new framework for improving agent effectiveness, influenced DX’s product roadmap. Within DX, teams can: - Assess whether their software environment is ready for effective agent use with the AI Readiness scorecard - Use Agent Experience to identify and address the friction that prevents agents from working effectively Coming soon: a new view of agent effectiveness across teams, repositories, tools, models, and types of work. Read more about how we’re bringing the CAFE(S) framework into DX: https://lnkd.in/eBEVJGdC

We recently shared the CAFE(S) framework, a new standard for evaluating AI context quality. While the industry invests heavily in model capabilities and harness orchestration, the translation from intent into context is what ultimately determines whether an AI agent succeeds or fails. CAFE(S) focuses specifically on this layer, giving teams a shared vocabulary to evaluate whether the context handed to an agent is actually fit for the task at hand. Read the full paper in ACM Queue: https://lnkd.in/g7cdbTjj