Link to article (substack.com)
I was an early adopter of AI coding and a fan until maybe two months ago, when I read the METR study and suddenly got serious doubts. In that study, the authors discovered that developers were unreliable narrators of their own productivity. They thought AI was making them 20% faster, but it was actually making them 19% slower. This shocked me because I had just told someone the week before that I thought AI was only making me about 25% faster, and I was bummed it wasnât a higher number. I was only off by 5% from the developerâs own incorrect estimates.
I couldnât even guess at an expected productivity increase based on the few months Iâve used AI coding tools. Itâs just too different from the experience of writing code for me to draw any comparisons from subjective experience alone.
Writing code gets me in a flow state quickly which I find easy to maintain and time flies by, whereas prompting AI feels more like managing a clumsy apprentice; it feels more time consuming because the difference between what I expect the AI will produce and what it actually does disrupts any flow Iâm in before I even really get going.
Rather than a shortcut, AI coding feels like gambling on the possibility of a shortcut. Itâs possible that by staking a fraction of the time it would take me to implement it myself, I could write a prompt that gets me there quicker. If I lose the bet, I can either accept that the time spent prompting was lost or chase my losses and sink more time into trying to coax something useful out of it. Each roll of the dice increases the downside; that time is wasted, and I still have to implement it myself anyway.
I discovered that the data isnât statistically significant at any meaningful level. That I would need to record new datapoints for another four months just to prove if AI was speeding me up or slowing me down at all. Itâs too neck-in-neck.
That lack of differentiation between the groups is really interesting though. Yes, itâs a limited sample and could be chance, but also so far AI appears to slow me down by a median of 21%, exactly in line with the METR study. I can say definitively that Iâm not seeing any massive increase in speed (i.e., 2x) using AI coding tools. If I were, the results would be statistically significant and the study would be over.
Thatâs really disappointing.
You bet.
Maybe itâs the speed of iteration that makes people feel like theyâre getting somewhere faster, but whenever I find myself reaching for it, Iâm mostly hoping it saves me mental effort by getting it right so I donât have to think about it, rather than saving me time.
Consider this: with all you know about AI-assisted coding and its wide adoption, if I showed you charts and graphs of new software releases across the world, what shape of that graph would you expect? Surely youâd be seeing an exponential growth up-and-to-the-right as adoption took hold and people started producing more?
Given the current discourse, you would expect this to be a given.
The most interesting thing about these charts is what theyâre not showing. Theyâre not showing a sudden spike or hockey-stick line of growth. Theyâre flat at best. Thereâs no shovelware surge. Thereâs no sudden indie boom occurring post-2022/2023. You could not tell looking at these charts when AI-assisted coding became widely adopted. The core premise is flawed. Nobody is shipping more than before.
So if you're a developer feeling pressured to adopt these tools â by your manager, your peers, or the general industry hysteria â trust your gut. If these tools feel clunky, if they're slowing you down, if you're confused how other people can be so productive, you're not broken. The data backs up what you're experiencing. You're not falling behind by sticking with what you know works.
As usual, some people have gone a bit silly with it. There are cases where AI will speed programming up, but even when it does, the speed comes at the cost of familiarity and understanding. Over time, this is bound to be reflected in your general ability to understand the stuff that youâve outsourced.
I think the false expectations of future improvements, and of a world in which understanding the codebase doesnât matter because AI can do it for you, are flawed in treating it as a shift to a higher-level programming language. I can only safely ignore and forget low-level concerns with high-level programming languages because the outcome of my higher-level instructions are consistent, guaranteed, deterministic.
Even though models will improve, natural language is too verbose and ambiguous to describe complex behaviours reliably. Youâll still have to reach for code eventually.