Cognition Labs, a startup with $21 million in funding, this week previewed what it describes as the world’s first software engineer, dubbed Devin, that is based on artificial intelligence (AI).
Devin is capable of responding to text commands and can be assigned tasks such as assessing the benchmark performance of an application. It will then build a plan and configure the required tools using its own command line interface (CLI), code editor and browser through which it can access, read and comprehend, for example, documentation using a reasoning engine and long-term planning capability based on advances in reinforcement learning.
Those capabilities make it possible for Devin to, for example, build a website, autonomously identify and fix bugs in codebases, deploy applications and even train other AI models.
Cognition Labs said a benchmark evaluation of Devin, using an SWE-bench tool that asks agents to resolve tasks, found Devin correctly resolved 13.86% of the issues end-to-end, far exceeding the previous state-of-the-art of 1.96%. Even when given the exact files to edit, the best previous models could only resolve 4.80% of issues.
Mark Hinkle, CEO of Peripety Labs, a consulting firm, said it’s too early to tell when Devin might be ready for use in enterprise IT environments, but the demonstrations showed how rapid advances in AI are about to transform how software is built and constructed. The user experience is substantially different from that of GitHub Copilot or CodeWhisperer from Amazon Web Services (AWS), which are more focused on helping developers write code rather than executing assigned tasks, he noted.
It’s not clear how much infrastructure is required to run Devin, which, given the ongoing shortage of graphical processor units (GPUs), may prove to be a limiting adoption factor, at least in the short term, added Hinkle.
Regardless of Devin’s current capabilities, which software engineers can only access by invitation right now, it’s clear advances to the reasoning capabilities of AI models will enable DevOps teams to assign tasks to an AI model much like they would any other member of their team. The impact that capability will have on demand for software engineers remains to be seen, but there is still a need to understand what to ask Devin to build and review how the software Devin builds has been constructed. The one thing that is certain is the pace at which software can be built and deployed is about to accelerate.
Undoubtedly, Microsoft, AWS, and other providers of software engineering tools are researching similar capabilities, so it may now be only a matter of time before advances in reasoning and long-term planning are broadly applied to software development.
In the meantime, DevOps teams might want to reassess their strategic plans for the coming years as it becomes easier to build software. Projects that might have required large teams of software engineers to complete might soon be accomplished using much smaller teams. In turn, the number of software development projects that could be launched by organizations large and small will increase as more of the toil historically required continues to be eliminated.
