AI Coding Assistants: Productivity Boost or Potential Pitfall?

There's a Problem WIth AI Programming Assistants: They're Inserting Far More Errors Into Code

Proponents of generative AI have claimed that the technology can make human workers more productive, especially when it comes to writing computer code. But does it really? A recent report conducted by coding management software business Uplevel, first spotted by IT magazine CIO, indicates that engineers who use GitHub's popular AI programming assistant Copilot don't experience any significant gains in efficiency. If anything, the study says usage of Copilot results in 41 percent more errors being inadvertently entered into code. For the study, Uplevel tracked the performance of 800 developers for three months before they got access to Copilot. After they got Copilot, Uplevel tracked them once again for another three months. To measure their performance, Uplevel examined the time it took for the developers to merge code into a repository, otherwise known as a pull request, and how many requests they put through. Uplevel found that "Copilot neither helped nor hurt the developers in the sample and also did not increase coding speed." "Our team's hypothesis was that we thought that PR cycle time would decrease," Uplevel product manager and data analyst Matt Hoffman told CIO. "We thought that they would be able to write more code, and we actually thought that defect rate might go down because you're using these gen AI tools to help you review your code before you even get it out there." All this information is not so surprising when you realize that GitHub Copilot is centered around large language models (LLM), which are often prone to hallucinating false information and spitting out incorrect data. Another recent study led by University of Texas at San Antonio researchers found that large language models can generate a significant number of "hallucination packages," or code that "recommends or contains a reference" to files or code that doesn't exist. Tech leaders are starting to get worried that making use of AI-generated code may actually end up being more work. "It becomes increasingly more challenging to understand and debug the AI-generated code, and troubleshooting becomes so resource-intensive that it is easier to rewrite the code from scratch than fix it," software development firm Gehtsoft CEO Ivan Gekht told CIO.

AI coding assistants do not boost productivity or prevent burnout, study finds

In a nutshell: Developers were supposed to be among the biggest beneficiaries of the generative AI hype as special tools made churning out code faster and easier. But according to a recent study from Uplevel, a firm that analyzes coding metrics, the productivity gains aren't materializing - at least not yet. The study tracked around 800 developers, comparing their output with and without GitHub's Copilot coding assistant over three-month periods. Surprisingly, when measuring key metrics like pull request cycle time and throughput, Uplevel found no meaningful improvements for those using Copilot. Matt Hoffman, a data analyst at Uplevel, explained to the publication CIO that their team initially thought that developers would be able to write more code and the defect rate might actually go down because developers were using AI tools to help review code before submitting it. But their findings defied those expectations. In fact, the study found that developers using Copilot introduced 41% more bugs into their code, according to CIO. Uplevel also saw no evidence that the AI assistant was helping prevent developer burnout. The revelations counter claims from Copilot's makers at GitHub and other vocal AI coding tool proponents about massive productivity boosts. A GitHub-sponsored study earlier claimed developers wrote code 55% faster with Copilot's aid. Developers could indeed be seeing positive results, given that a report from Copilot's early days showed nearly 30% of new code involved AI assistance - a number that has likely grown. However, another possibility behind the increase usage is coders developing a dependency and turning lazy. Out in the field, the experience with AI coding assistants has been mixed so far. At custom software firm Gehtsoft USA, CEO Ivan Gekht told CIO that they've found the AI-generated code challenging to understand and debug, making it more efficient to simply rewrite from scratch sometimes. A study from last year where ChatGPT got over half of the asked programming questions wrong seems to back his observations, though the chatbot has improved considerably since then with multiple updates. Gekht added that software development is "90% brain function - understanding the requirements, designing the system, and considering limitations and restrictions," while converting all this into code is the simpler part of the job. However, at cloud provider Innovative Solutions, CTO Travis Rehl reported stellar results, with developer productivity increasing up to three times thanks to tools like Claude Dev and Copilot. The conflicting accounts highlight that we're probably still in the early days for AI coding assistants. But with the tools advancing rapidly, who knows where they are headed down the line?

AI tools rewrite the coding playbook

Coding assistants have improved productivity and saved millions, but they come with the risk of bias and emerging copyright concerns. ET looks at the rise of coding assistants and pitfalls as the adoption increases.Six hours. That was all it took Kiran (name changed), founder of a climate tech startup, to design a static mobile application for his product using the latest AI coding tools available in the market. As a solo developer, he pegs it would take another couple of months to get the minimum viable product ready, in time for his pre-seed round six months later. A year ago, the whole process would have taken half-a-year, a team of 3-4 developers and a few thousand dollars. Thanks to AI-powered coding tools like Cursor AI, Copilot and others, this is possible with just a few hundred dollars and half the time, according to Kiran. Improvements in reducing hallucination, better large language models and prompt engineering, where the large language models were trained to generate relevant output with the right context, have helped in wider adoption. Then, it is hardly a wonder that AI coding assistants have become a mainstay in developers' life. These tools will have a significant impact on India given that it has one of the largest developer markets at 15.4 million and large Indian enterprises adopting them. Srikanth Nadhamuni, chairman, Khosla Labs and cofounder, Trustt, a fintech platform, told ET that when he had met OpenAI founder Sam Altman a couple of years ago, Altman had identified coding as one of the areas where GenAI could have a significant impact. At Trustt, Nadhamuni said that the company has introduced coding assistants and has seen substantial benefits. He said that what used to take two hours, it takes just five minutes with the right prompt. Since the launch of the very first version of Chat GPT in 2020 till date, several AI-powered coding tools have flooded the market offering code suggestions to generating an entire block of it, based on prompts. Enterprises are lapping it up. Take GitHub's Copilot. Till date, about 77,000 businesses have begun using Copilot. Shuyin Zhao, VP-product management at GitHub, said that Copilot helps developers stay in the flow and preserve mental effort during repetitive tasks, which drain and derail focus. Abhiram R, a Bengaluru-based Python developer, said that in the last six months, what used to take six hours a week, involving repeated functions like creating files, has come down to three hours now. This is a significant saving in developers' time, which can be used to create more software and innovate rapidly, say experts. Krish Ramineni, cofounder, Fireflies.ai, an AI note taking tool said, "AI helps our engineers become more productive and write more code in less time and helps with code reviews. We built our AI internally, an agent that can answer 30 to 40% of CS (support) tickets. So this has helped free up a lot of the stress." He added that there are certain tasks that they are able to complete almost 20-30% faster. Over time, it will save over 10 hours on average per month for an engineer. "The cost savings then adds up." These tools can also be used to fast-track learning. Paras Chopra, founder, Turing's Dream, a Bengaluru based AI residency, said that use of these tools can make a good developer better and help those who are starting out leapfrog their experience using these tools. Experts, however, point out that they are a double-edged sword since there are concerns of bias and copyright issues. Developer analytics tool GitClear analysed 153 million changes in lines of code authored between January 2020 and December 2023 and found that the code quality has taken a hit since use of AI tools. Developer security platform Synx highlighted that coding assistants have limited understanding of software and can make systems vulnerable. All this requires human oversight more than ever. Bias is another big issue. James Landay, co-director and cofounder, Stanford Institute for Human-centred Artificial Intelligence, earlier told ET that bias is one of the biggest issues with the LLMs. With no clarity on the kind of data that are used for training the LLM models, developers need to be aware of the pitfalls of using the model. Both Copilot and OpenAI are currently facing a class action suit from developers for using licensed code without attribution, violating copyright laws. The outcome of this case will have a huge impact on generative AI systems.