{"id":291,"date":"2024-11-18T20:46:43","date_gmt":"2024-11-18T20:46:43","guid":{"rendered":"https:\/\/redmonk.com\/kholterhoff\/?p=291"},"modified":"2024-11-22T14:29:17","modified_gmt":"2024-11-22T14:29:17","slug":"top-10-things-developers-want-from-their-ai-code-assistants-in-2024","status":"publish","type":"post","link":"https:\/\/redmonk.com\/kholterhoff\/2024\/11\/18\/top-10-things-developers-want-from-their-ai-code-assistants-in-2024\/","title":{"rendered":"Top 10 Things Developers Want from their AI Code Assistants in 2024"},"content":{"rendered":"

\"\"<\/p>\n

In the afterglow of GitHub Universe 2024, I\u2019m revisiting the \u201c10 Things Developers Want from AI Code Assistants<\/a>\u201d post I authored last year because in the past 12 months THINGS HAVE CHANGED. Sure, the technology has evolved and improved, but more importantly, developers are fundamentally shifting their workflows to accommodate these tools. Although the data on code impact is mixed, the 2024 DORA Report<\/a> indicates that AI code assistants have become a common element of developer tooling. 76% of respondents report relying on AI for tasks like code writing, summarizing information, and code explanation, and 67% of respondents report that AI is helping them improve their code.<\/p>\n

There is also a ton more competition in the marketplace. In addition to the options I listed last year of GitHub Copilot, Sourcegraph\u2019s Cody, Amazon CodeWhisperer (now Amazon Q Developer), CodiumAI, IBM\u2019s watsonx Code Assistant, Tabnine, MutableAI, AskCodi, Codiga, and Replit AI, I can now add Aider, Augment Code, Cline, CodeComplete, CodeGeeX, CodeGPT, Codiga, OpenAI Codex, Continue.dev, Cursor, Snyk\u2019s DeepCode AI, Cognition\u2019s Devin, Google\u2019s Gemini Code Assist, Microsoft IntelliCode, JetBrains AI Assistant, Refact.ai, Sourcery, SQLAI, Qodo Gen (formerly Codiumate), and Void (phew<\/i>). There is also a new breed of agents to accomplish specific tasks in the SDLC such as testing and QA (Copilot Autofix, Graphite Reviewer), as well as agents for bootstrapping entirely new apps like, Replit Agent, Stackblitz Bolt, GitHub Spark, and Vercel v0. These additions (which I do not claim to be encompassing) to my list are not necessarily new since last year, but rather they have appeared on my radar since then. Indeed, staying on top of the AI code assistant offerings and features is a full time job. Relatedly, if you\u2019re reading this, Forrest Brazeal<\/a>, can I get this list put to the Major-General’s Song, kthx?<\/p>\n

Here is some of my evidence that developer sentiment around AI code assistants is evolving. First, when I attended DevNexus<\/a> in the spring I was struck by the fact that every demo I attended used an AI code assistant in some capacity. Second, developers are forming communities expressly to discuss AI code assistant tooling. While r\/GithubCopilot<\/a> (6.2K members) and r\/tabnine<\/a> (141 members) were founded in 2021, r\/cursor<\/a> (5.5K members) was formed only this February. Beyond providing space to voice their enthusiasm, these and other dedicated communities (Hacker News, Dev.to, Tech conferences and meetups, etc) empower users to troubleshoot bugs, share tips, and negotiate best practices.<\/p>\n

A few table-setting notes: like last year\u2019s post, I am once again most interested in what developers are actually doing and saying. My research is qualitative. It is primarily derived from developer forums like Hacker News and Reddit, as well as private conversations I am having with practitioners and vendors. Also, this is not an inclusive list of features, and the desired features I listed last year absolutely endure today. Illustratively, I spoke with a Copilot customer recently who complained that language support for older coding languages needs improvement (2023 #4<\/a>). For the sake of avoiding redundancy while casting a wider net, I have for the most part endeavored to cover new territory rather than doubling-down on last year\u2019s claims.<\/p>\n

Here\u2019s the list:<\/p>\n

    \n
  1. Tab Completion: <\/b>According to many<\/a> developers<\/a>, tab completion is the killer feature in AI code assistants. These developers call not only for the ability to predict and accept a change using tab, but also to predict the next change after the current completion, enabling them to tab, tab, tab their way to happiness. Although vendors tend to make much of their chat capabilities, developers want to hit tab to accept autocomplete and move on\u2014especially<\/a> with new features like multi-file editing (see #9). Developers praise tab completion in Cursor. It is the first bullet point in Continue.dev\u2019s feature list. Tab completion is something developers have grown to appreciate in numerous<\/a> scripting tools and workflows, particularly on the CLI, and now demand it in their AI code assistants.<\/li>\n
  2. Speed: <\/b>Flow is essential to developers, and nothing pulls them out of this state more completely than lag. The issue of speed appears frequently in forums. For developers, accusations of a \u201csluggish<\/a>\u201d experience and being forced to wait is an absolute non-starter. Eliminating latency is the promise of smaller models, and I have heard several complaints that, despite the profound capabilities of OpenAI\u2019s o1, the slowness of this model makes it impracticable for AI code assistant use cases. As Kevin Kernegger<\/a>, Founder and CEO of Macherjek GmbH, explains<\/a>:\u00a0 \u201co1 takes some time to think before it answers. So its not instant like we’re used to.\u201d<\/li>\n
  3. High-Level: <\/b>Code assistants are not just for writing scripts and pushing pixels\u2014they assist at app building\u2019s planning stage. Tom Yedwab<\/a>, Data Architect at Khan Academy, argues<\/a> (via Reddit\u2019s r\/ChatGPTCoding<\/a>), that one benefit he gets from Cursor is the high-level perspective it offers of his projects:
    \n

    this tool feels like it is reading my mind, guessing at my next action, and allowing me to think less about the code and more about the architecture \u2026 I am building.<\/p><\/blockquote>\n

    As someone skeptical<\/a> of AI\u2019s ability to insert itself in every step of the SDLC I find this quote particularly exciting. Today, AI code assistants are helping developers shift left in their workflow in order to think big picture. Some developers report that this change has made them \u201cfeel more like a solution architect<\/a>.\u201d Others complain<\/a> that the technology isn\u2019t quite there yet and sometimes seems to be getting ahead of its skis (\u201cI’m aware of Devin and a few other higher-level systems, but they seem (a) like they’re still vaporware and (b) they’re actually aiming for an even higher level of functioning that relies on more judgment, taste, and design skills — too ambitious, don’t want to delegate *that* much.\u201d). Although use cases continue to center on autogenerating code rather than architecting it, this move to use code assistants to think high-level is deeply significant because it means that these assistants require full visibility and comprehension of an entire code base.<\/li>\n

  4. Superb Suggestions:<\/b> Obvious? Maybe. Developers naturally want the first suggestion to be the right one, but this requirement usually has more to do with the technical capabilities of the model than the assistant itself. In fact, ongoing debates around which model is better usually come down to which gives the best suggestions for a particular project (often this has to do with language support). What this means for vendors is that they must support multiple (all?) models so that developers are able to select the one that gives the most excellent suggestions for their individualized use cases. See #7 below for more on this.<\/li>\n
  5. Context:<\/b> Context is King. Developers frequently post questions to Reddit like \u201cHow to feed\/provide documentations to Github Copilot for context?<\/a>\u201d and \u201cLooking for an LLM Fully Aware of My Entire Project \u2013 Alternatives to GitHub Copilot?<\/a>\u201d Similarly on Hacker News, Lucas Jans<\/a>, VP of Product at Agency Revolution, explains<\/a>:
    \n

    I want to build my own agents so I can [have] my private domain specific awareness in the coding environment: PRDs, product docs, API docs, business goals, etc.<\/p><\/blockquote>\n

    Today, this sort of context-aware AI code assistant is promised by many vendors in the space including Augment Code<\/a> (\u201cIt’s the most context-aware Developer AI, so you won’t just code faster, you’ll build smarter.\u201d) and Cursor<\/a> (\u201cReference code with @ symbols to be used as context for the AI.\u201d). Meanwhile, the promise of GitHub Copilot is that it can be customized<\/a> to their own organization\u2019s private code and processes in addition to context from public repos. Competition on the context-front is fierce for very good reason.<\/li>\n

  6. IDE Fork or No Fork?:<\/b> There is no consensus about the VS Code fork that Cursor and Void use, except that forks add friction and some developers are mad about it. The folks at Cursor<\/a> chose to fork rather than build an extension because:
    \n

    VSCode extensions have very limited control over the UI of the editor. Our Command-K and Copilot++ features aren\u2019t possible as extensions. Same for much of what we want to build in the future!<\/p><\/blockquote>\n

    Although using a forked, closed-source version of VSCode is inconvenient, the benefits of Cursor have made the sacrifice worthwhile for some. Others opt to use Sourcegraph\u2019s Cody, Continue, and Copilot (among others) because they are VS Code extensions. As Francesco Belladonna, lead developer at Jane Technologies, writes<\/a>: \u201cHonestly, I’d rather have a buggish extension than having to change editor.\u201d Still other developers, wishing to avoid the drama altogether, are adopting Aider, which markets itself as \u201cpair programming in your terminal.\u201d At the end of the day, like everything else in developer tooling, it comes down to developer experience, and the code assistant that offers the best experience will win out.<\/li>\n

  7. Multiple LLM Support: <\/b>I\u2019m cheating a little here as this point is a simplified version of points 9 and 10 that I made last year, but, hoo boy, has it become more relevant. Developers are opinionated about their preferred models. A lot of developers are raving about Claude in 2024, it\u2019s been a huge market impact story. For example, Wes Bos<\/a>, host of the Syntax podcast, goes into raptures<\/a> about it. Probably the biggest announcement at Universe was the new models integrated into Copilot of Anthropic\u2019s Claude 3.5 Sonnet, Google\u2019s Gemini 1.5 Pro, and OpenAI\u2019s o1-preview. Thomas Dohmke is right to frame<\/a> this move as one of \u201cdeveloper choice,\u201d as it is very much something practitioners demand.<\/li>\n
  8. Multiple LLMs Simultaneously: <\/b>Developers want to use two or more models at the same time in order to leverage the strengths of each. According to one Redditor<\/a>, who uses ChatGPT and Cursor for unit testing:
    \n

    My ideal outcome is to have OpenAI specify the test plan, chat with Cursor who would execute the test, interpret the output and ask OpenAi any questions. This would be repeated until the unit test is passed.<\/p><\/blockquote>\n

    Other developers<\/a> in the space echo this desire and have had moderate success. My take away for vendors is that supporting multiple LLMs is now tablestakes; supporting multiple LLMs simultaneously is the future.<\/li>\n

  9. Multi-file Creation and Editing: <\/b>The ability to create and edit files is tablestakes. What makes some assistants stand out is how contextually aware these created files are (see point 5 above). According<\/a> to one Cline user:
    \n

    I’ve been using Cline and really like it, especially the way I can say “make a new function that does XYZ” and it can easily review all existing ones, and create as many files as necessary. Same with if something isn’t working, I can paste an error code and it goes through the files and comes back with “I see what the issue is…” and so on.<\/p><\/blockquote>\n

    Interestingly, multi-file editing is still “evolving”<\/a> and therefore buggy (see #10 below). Developers flock<\/a> to forums to discuss how to optimize, and some<\/a> have built tools to bridge the gap.<\/li>\n

  10. Mitigate Unintended Deletions:<\/b> Multi-file editing and creation is supported by more AI code assistants today, but with this capability has come a wave of disgruntled developers that are now grappling with unintended deletions. One Redditor<\/a> complains: \u201cOf the last 100 times that Cursor has deleted a file of mine, maybe 2 of those deletions were intended behavior that I actually wanted.\u201d This is undoubtedly user error, and the thread is full of helpful users pointing out ways to keep this from happening. However, as this is a post about developer experience, it is worthwhile to point out these areas of friction. The on-ramp to becoming a power user of these tools is steep, and the consequences of missteps can be serious. As more and more of the code creation process is automated and unreviewed this will only increase in importance.<\/li>\n<\/ol>\n

     <\/p>\n

    As I look ahead to 2025, I am reminded just how important developers will be in determining the winners and losers in this market. Indeed, if there\u2019s one thing my research made clear it is the importance of developer experience. As I explained last year, but is worth repeating:<\/p>\n

    Code assistant vendors are uniquely reliant on the success of their product\u2019s developer experience. These tools exist to assist<\/i>, which means they work in support of the developer and must cater to their needs and expectations. They are not an express train to a post-work future. No one is laying off their engineering teams and replacing them with robots. To excel in this competitive and potentially lucrative tooling space, the companies developing these tools ignore the wishes of their developer users at their peril.<\/p><\/blockquote>\n

    Interested to hear from the community. Do you disagree? Did I miss anything? Let me know here in the comments, on Bluesky<\/a>, or LinkedIn<\/a>.<\/p>\n

    Disclaimer: <\/b>AWS, GitHub, Microsoft, Google, and IBM are RedMonk clients.<\/p>\n","protected":false},"excerpt":{"rendered":"

    In the afterglow of GitHub Universe 2024, I\u2019m revisiting the \u201c10 Things Developers Want from AI Code Assistants\u201d post I authored last year because in the past 12 months THINGS HAVE CHANGED. Sure, the technology has evolved and improved, but more importantly, developers are fundamentally shifting their workflows to accommodate these tools. Although the data<\/p>\n","protected":false},"author":50,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":""},"categories":[32,12],"tags":[],"class_list":["post-291","post","type-post","status-publish","format-standard","hentry","category-ai","category-devx"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/posts\/291","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/users\/50"}],"replies":[{"embeddable":true,"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/comments?post=291"}],"version-history":[{"count":0,"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/posts\/291\/revisions"}],"wp:attachment":[{"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/media?parent=291"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/categories?post=291"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redmonk.com\/kholterhoff\/wp-json\/wp\/v2\/tags?post=291"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}