I've built 12+ production AI agent systems across development, DevOps, and data operations. Here's why the current hype around autonomous agents is mathematically impossible and what actually works in production.
Yes, I’ve seen this as well. First of all, 16 devs is a tiny sample, a far bigger study would be needed to get any meaningful results here. Second, it really depends on how experienced people are at using these tools. It took me a while to identify patterns that actually work repeatably and develop intuition for cases where the model is most likely to produce good results.
METR measured the speed of 16 developers working on complex software projects, both with and without AI assistance. After finishing their tasks, the developers estimated that access to AI had accelerated their work by 20% on average. In fact, the measurements showed that AI had slowed them down by about 20%.
Yes, I’ve seen this as well. First of all, 16 devs is a tiny sample, a far bigger study would be needed to get any meaningful results here. Second, it really depends on how experienced people are at using these tools. It took me a while to identify patterns that actually work repeatably and develop intuition for cases where the model is most likely to produce good results.