OXD is in the midst of supporting one of our clients perform research on the roll-out of GitHub Copilot, the LLM-powered coding assistant, for their developers. They’ve onboarded and trained a cohort of developers on the tool and we’re helping with a three-month research project into the developer experience of using AI coding assistants. We’re just starting to see some of the emerging findings about how developers perceive using the tool, their trust and confidence with the code it suggests, and some of the benefits and costs associated with it.
The debate: Speed vs. code quality with AI
Github’s own research claims that developers are able to code more and code faster using Copilot. And they’ve also claimed that the quality of the code (and maintainability) has increased over time.
Yet earlier in 2024 and recently updated in February of 2025, Gitclear, a Seattle-based software company, published some interesting results on code churn and maintainability based on the analysis of 153M lines of code that suggest quality is heading in the other direction thanks to AI coding assistants.
From their research paper:
The 36,894 developers who answered ‘most important benefit you’re hoping to achieve with AI’ overwhelmingly picked ‘Increased productivity.’ Developers seem to view AI as a means to write more code, faster. Through the lens of ‘does more code get written?’ common sense and research agree: resounding yes. AI assistants do beget more lines.
But if you ask a Senior Developer ‘what would unlock your team’s full potential?’ their answer won’t be ‘more code lines added.’ To retain high project velocity over years, research suggests that a DRY (Don’t Repeat Yourself), modular approach to building is essential. Canonical systems are documented, well-tested, reused, and periodically upgraded.
This research aims to evaluate how today’s profusion of LLM-authored code is influencing the maintainability of tomorrow’s systems.
I was reminded of software engineering author of the 1990’s and 2000’s Steve McConnell and his expression, “the thing about quick and dirty is that long after quick is done, dirty still remains.”
And I was also reminded of how individual gains and short-term benefits can easily obscure longer term externalities and systemic costs. Individual car ownership can be fast and convenient and the air conditioning is great in the summer time, so long as you ignore the congestion, air and noise pollution, and healthcare system costs.
Evaluating AI coding assistant quality
Amidst all of the hype and promise and noise about AI tools, I’m glad our clients are taking the time to measure and evaluate, to treat AI as normal technology.
It doesn’t take a lot of effort to interview and survey developers (or other types of users of AI) and try to agree on some baseline metrics to gather, collect personal feedback, and to see more clearly the individual and systemic impacts of some of these tools. It turns out that speed limits, for both cars and AI, might just be a good thing to limit our collective risks associated with both.
I'm interested in learning about evaluations underway on AI tool pilots, adoption, and diffusion of these tools and their effects. If you've seen or conducted a good one in your organization, please share your insights with me using our contact form.
Sign up for our newsletter below to get our latest insights, case studies, and news delivered straight to your inbox.