{"id":759,"date":"2023-12-14T10:04:49","date_gmt":"2023-12-14T17:04:49","guid":{"rendered":"https:\/\/redmonk.com\/rstephens\/?p=759"},"modified":"2023-12-14T10:04:49","modified_gmt":"2023-12-14T17:04:49","slug":"language-rankings-update","status":"publish","type":"post","link":"https:\/\/redmonk.com\/rstephens\/2023\/12\/14\/language-rankings-update\/","title":{"rendered":"What\u2019s Going On With Language Rankings?"},"content":{"rendered":"

As the inbound DMs and emails attest, at least a few of you out there having been waiting for RedMonk to drop our latest version of the programming language rankings. Our apologies about the delay: here\u2019s our update on the process.<\/p>\n

Normally we run these rankings twice a year, with the goal of using publicly available data sets to track if there are any meaningful changes in how people are using programming languages. We correlate cumulative language usage as seen through non-forked PRs on public GitHub repos against questions asked on Stack Overflow.<\/p>\n

These metrics have always told an incomplete story at best. There are languages that were under- or over-represented by using public GitHub repos; there are communities that were under- or over-represented by looking at discussions happening in Stack Overflow. These metrics were not perfect by any means, but the two large, publicly facing datasets created an interesting decade plus long trend for RedMonk to track over time.<\/p>\n

Our last set of queries, however, provided us with data that required further investigation.<\/p>\n

Stack Overflow<\/h2>\n

We\u2019ve been internally discussing how we\u2019re going to address the impact of AI-based code assistants on our language rankings since GitHub released Copilot in October 2021. However, it was when ChatGPT hit the market on November 30, 2022 and went from 0 to 100M users in two months<\/a> that we started seeing undeniable impacts on our source data.<\/p>\n

We\u2019d already seen questions asked falling off from their peak. However, when we pulled data for our June 2023 rankings the collective decline was notable.<\/p>\n

The below chart takes questions asked on Stack Overflow about the top 20 programming languages (as determined from RedMonk\u2019s analysis dated January 2023<\/a>) and look backed at historical semi-annual trends.<\/p>\n

\"Bar<\/a><\/p>\n

As you can see, the number of questions asked using these 20 Stack Overflow tags* declined almost 20% from the prior period. And this is just our first full period running these numbers post-ChatGPT. A cursory query about YTD indicates an even more stark change.<\/p>\n

It is undeniable that developer\u2019s ability to instantaneously ask questions of a non-judgmental AI assistant that will give answers in context is going to have a marked negative impact on the usefulness of the public datasets provided by Stack Overflow.<\/p>\n

GitHub<\/h2>\n

While we expected a change in Stack Overflow, we were not expecting significant anomalies in our data from GitHub (and to the extent that we were anticipating changes, we expected them to be upwards. The narrative about AI code assistants aiding development velocity and benefitting tinkerers<\/a> has been prevalent.)<\/p>\n

However, the data we saw from GitHub Archive actually showed a roughly 25% decline in pull requests in 1H2023 as compared to 2H2022 PRs that we were not expecting.<\/p>\n

The dataset we use is a public dataset on BigQuery, and so we asked questions of both Google in terms of how the data was pulled in and of the GitHub team to see if they had seen similar changes on in their internal data.<\/p>\n

In the end, the change appears to not be an error in the available data and is largely lacking an explanation. The best guess thus far is that there was an overhang in increased activity from the pandemic and this is a return to expected activity, but we have no way of confirming whether that storyline is accurate.<\/p>\n

As of now this is not a declining trend we expect to continue, but these are numbers we will continue to watch with interest.<\/p>\n

What Next?<\/h2>\n

The advent and rise of AI-based code assistants are already impacting the data that populates RedMonk\u2019s language rankings. As questions and knowledge sharing moves from public forums to private tools, our ability to ascertain meaningful trends from said public data will be indefinitely altered.<\/p>\n

We will continue to track these trends and make determinations about how this change in sample size will impact our ability to perform the rankings.<\/p>\n

As of now, look for our next rankings in January 2024.<\/p>\n

\n

* Tags queried based on last Top 20:<\/strong> JavaScript, Python, Java, PHP, C#, CSS, TypeScript, C++, Ruby, C, Swift, Shell, R, Go, Scala, Objective-C, Kotlin, PowerShell, Rust, Dart<\/p>\n

Disclaimer:<\/strong> GitHub and Google Cloud are RedMonk clients. Stack Overflow and OpenAI (ChatGPT) are not.<\/p>\n","protected":false},"excerpt":{"rendered":"

As the inbound DMs and emails attest, at least a few of you out there having been waiting for RedMonk to drop our latest version of the programming language rankings. Our apologies about the delay: here\u2019s our update on the process. Normally we run these rankings twice a year, with the goal of using publicly<\/p>\n","protected":false},"author":45,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"spay_email":"","footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false},"categories":[40,49,47,26,68,24],"tags":[],"class_list":["post-759","post","type-post","status-publish","format-standard","hentry","category-ai","category-featured-data","category-github","category-google","category-programming-languages","category-stack-overflow"],"jetpack_featured_media_url":"","jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/posts\/759","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/users\/45"}],"replies":[{"embeddable":true,"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/comments?post=759"}],"version-history":[{"count":0,"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/posts\/759\/revisions"}],"wp:attachment":[{"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/media?parent=759"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/categories?post=759"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/redmonk.com\/rstephens\/wp-json\/wp\/v2\/tags?post=759"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}