Is it possible to rank programming languages by their efficiency, or expressiveness? In other words, can you compare how simply you can express a concept in them? One proxy for this is how many lines of code change in each commit. This would provide a view into how expressive each language enables you to be in the same amount of space. Because the number of bugs in code is proportional to the number of source lines, not the number of ideas expressed, a more expressive language is always worth considering for that reason alone (e.g., see Halstead’s complexity measures).
I recently got a hold of a great set of data from Ohloh, which tracks open-source code repositories, on the use of programming languages over time across all of the codebases they track. After validating the data against Ohloh’s own graphs, one of the first things I did was try out my idea on expressiveness of programming languages. Sure enough, it gave me results that made sense and were surprisingly reasonable.
Some caveats to this approach :
- This assumes that commits are generally used to add a single conceptual piece regardless of which language it’s programmed in.
- It won’t tell you how readable the resulting code is (Hello, lambda functions) or how long it takes to write it (APL anyone?), so it’s not a measure of maintainability or productivity.
- Ohloh relies on opt-in subscription from open-source projects rather than crawling forges itself. That said, it’s a vast data set covering some 7.5 million project-months.
Time to let the results speak for themselves. Enough words, here’s the data (enlarge by clicking):
It’s visualized in the form of box-and-whisker plots, which are effective for showing a distribution of numbers relatively simply. What numbers are we showing? It’s a distribution of lines of code per commit every month for around 20 years, weighted by the number of commits in any given month. The black line in the middle of each box is the median (the 50th percentile) for that language, and languages are ranked by median. The bottom and top of the box are the 25th and 75th percentiles, while the “whiskers” extend to the 10th and 90th percentiles. The “Total” box indicates the median of each value across all languages (median of all 25th percentiles, median of all 75th percentiles, etc.) to show a “typical” language.
I’ve also colored them according to our most recent RedMonk programming language rankings (red is the most popular cluster, and blue is the second-tier cluster, while black is everything else), and restricted languages here to the ones popular enough to be included in that set of rankings.
What conclusions can we draw from this?
The trends generally make sense. If we focus purely on the tier-one languages shown in red, high-level languages (Python [#27], Ruby [#34]) lean toward better expressiveness while lower-level languages (C [#50], C++ [#45], Java [#44]) tend toward wordiness. Similarly in tier two, Fortran [#39/#52] and assembly [#49] are wordy, and “middle-aged” functional languages are intermediate while newer functional languages are best.
Expressiveness ranges broadly across languages. The medians go from lows of 48 for Augeas (#1) and 52 for Puppet (#2) to a high of 1629 for fixed-format Fortran (#52), which is a surprisingly large 31x variation.
Less expressive languages tend to show a much wider variability. There’s a clear, but not strong, correlation between the medians (black lines) and the IQRs (box heights). Languages with the largest IQRs also tend to have greater medians, and consistently expressive languages tend to also be more expressive.
Second-tier languages are well-distributed and reach into highly expressive languages. With 52 total languages on this list, the top ~17 constitute the highly expressive languages. Although none of those are first-tier languages, 9 of those 17 are second-tier — mostly functional with the exceptions of Groovy (#16), Prolog (#13), Puppet (#2), and CoffeeScript (#6).
Third-tier languages are heavily biased toward high expressiveness. Of the 15 third-tier languages on this list, 8 are in the top 1/3 of languages, leaving only 7 are in the remaining 2/3. Although these data do not directly show any correlation between age and expressiveness, it seems reasonable that newer, more expressive languages would begin less popular and may grow later.
Effects of language class/type
Functional languages tend to be highly expressive. On this list are Haskell (#10), Erlang (#22), F# (#21), Lisp variants (including Clojure [#7], Emacs Lisp [#14], Dylan [#12], Common Lisp [#23], Scheme [#31], and Racket [#11]), OCaml (#20), R (#17), and Scala (#18). Of those, only two fall below #30 out of the 52 languages included here.
Domain-specific languages are biased toward high expressiveness. Augeas (#1), Puppet (#2), R (#17), and Scilab (#19) are good examples of this, while VHDL (#38) serves as an outlier on the low end.
Compilation does not imply lower expressiveness. I was halfway expecting highly expressive languages to exclude all compiled languages but was proven wrong. Compiled languages in the top 17 include CoffeeScript (#6), Vala (#9), Haskell (#10), and Dylan (#12).
Interactive modes correlate with intermediate expressiveness. Languages with an interactive shell tend to be mid-range in expressiveness, with a few outliers on either side. For example: Lisp (#23), Erlang (#22), F# (#21), OCaml (#20), Perl (#26), Python (#27), R (#17), Ruby (#34), Scala (#18), Scheme (#31).
Specific language effects
Clojure (#7) is the most expressive of Lisp variants. There are a large number of Lisp variants that generally ranked quite well, described in more detail above in the functional-language section. In this context, it’s worth noting that the top one was the fairly popular Clojure, with a median LOC/commit value of 101, followed by Racket (#11) at 136 and Dylan (#12) at 143.
Among data-analysis languages, R (#17) and Scilab (#19) are most expressive. With a median of 193 LOC/commit for R, it’s a clear top performer. R is followed by Scilab and Matlab (#35) with medians of 225 and 445, respectively.
Although Go (#24) is getting increasingly hot, it’s not outstandingly expressive. We keep hearing about new use of Go across a variety of startups, but it’s little better than Perl (#26) or Python (#27) by this measure. Despite that, it does trump all the tier-one languages, so someone who only had experience with them could certainly see an improvement when trying Go.
What if we sort by consistency of expressiveness, instead of the median?
Ideally a language should be:
- Easy enough to learn that the vast majority of developers using it can be highly productive; and
- Equally expressive across nearly its entire domain of usefulness.
To measure that, let’s take a look at the interquartile range (IQR; the distance between the 25th and 75th percentiles) as a proxy for these two criteria, and rank languages by that instead (enlarge by clicking):
What you’re looking for here is the height of the boxes. It starts small on the left side, with CoffeeScript doing best at 23 lines and increases to the right side, ending with fixed-format Fortran at 1854 lines.
A few new insights specific to this plot before we move on to considering them both together:
- As alluded to earlier but illustrated differently here, inconsistency and wordiness are correlated, as are consistency and expressiveness.
- Tier-one languages put in a much stronger showing here, with four in the top 1/3 of languages (Python at #11, Objective-C at #13, Perl at #15, and C# at #17). Shell nearly makes the cut at #19. Those IQRs vary from 90–167 LOC/commit, a fairly large difference even among the best performers.
- Consequently, tier-three languages make a poorer showing here, although they performed unusually well at levels of expressiveness. They are nearly proportionate with their population with 5 of 15 showing up in the top third, and the remainder are evenly distributed across the moderate and low consistency groups as well.
- Java turns in the strongest performance of “enterprisey” languages (C, C++, Java) when considering both metrics. Java comes in with nearly identical expressiveness as C++ (both at 823 LOC/commit) but a vastly greater consistency (IQR of 277 vs 476).
- CoffeeScript is #1 for consistency, with an IQR spread of only 23 LOC/commit compared to even #4 Clojure at 51 LOC/commit. By the time we’ve gotten to #8 Groovy, we’ve dropped to an IQR of 68 LOC/commit. In other words, CoffeeScript is incredibly consistent across domains and developers in its expressiveness.
- The outliers are particularly interesting — the ones with unusually high or low medians compared to nearby languages. If the median is higher than neighbors, than it’s an unusually consistent yet less expressive language. Conversely if the median is lower than neighbors, then the language is unusually inconsistent (a.k.a. shifted to the right on this graph from the rough correlation between consistency and median expressiveness).
- Tier-one languages tend to be remarkably consistent, regardless of their expressiveness. In nearly all cases, their medians are higher than their neighbors, showing a general shift to the left from the expected placement. This suggests that a primary characteristic of a tier-one language is its predictability, even more so than its productivity.
- Conversely, in most cases where languages appear shifted to the right, they’re third-tier languages. The lack of predictability has often held them back from even reaching the second tier.
So, what are the best languages by these metrics?
If you pick the top 10 based on ranking by median and by IQR, then take the intersection of them, here’s what’s left. The median and IQR are listed immediately after the names:
- Augeas (48, 28): A domain-specific languages for configuration files
- Puppet (52, 65): Another DSL for configuration
- REBOL (57, 47): A language designed for distributed computing
- eC (75, 75): Ecere C, a C derivative with object orientation
- Clojure (101, 51): A Lisp dialect for functional, concurrent programming
- Vala (123, 61): An object-oriented language used by GNOME
- Haskell (127, 71): A purely functional, compiled language with strong static typing
Looking at the box plots again, I would tend to rule out eC based on the poor performance of the upward-reaching whiskers at the 90th percentiles, indicating a real lack of consistency as often as a quarter of the time (since the 75th percentile is quite good). I would also rule out Puppet and Augeas because they are DSLs.
Combining those with our RedMonk programming language rankings on popularity, the only highly expressive, general-purpose languages within the top two popularity tiers are:
If you’re considering learning a new language, it would make a lot of sense to put Clojure, CoffeeScript, and Haskell on your list, based on expressiveness and current use in communities we’ve found to be predictive.
No tier-one languages fall in the top 25 on both metrics, although 5 make the cut on consistency alone. Of the tier-one languages, lower-level ones tend to be both inconsistent and overly wordy, while higher-level ones have intermediate wordiness and very strong consistency. The most consistent languages are Python, Objective-C, Perl, C#, and shell, with the presence of Perl and shell supporting the initial assertion that expressiveness has little to do with readability or maintainability. Ruby is an interesting language, in that it violates the “rules” of expressiveness and consistency seen in the other higher-level languages. This could be an instance of a framework (Rails) truly popularizing a language that otherwise would’ve never taken off.
For projects that require an expressive language where it’s relatively easy to hire developers, Python is worth serious consideration. Of tier-one languages, Python, Perl, Shell, and Objective-C are the best overall performers, and I consider Python the strongest of those for general-purpose applications. In my opinion, it makes a lot of sense to take a polyglot approach to projects, writing as high-level as performance requirements allow. Fortunately many high-level languages like Python allow for modules based in more performant languages such as C. That means it’s easily possible to write the vast majority of a project in a more productive, more expressive language while falling back to high-performance languages where needed.
Update (3/26/12): I somehow missed Haskell on the final recommendations for second-tier languages, although it was on the initial list. Thanks to Chad Scherrer for pointing it out in the comments.
Update (3/26/12): I just wrote a post on the last day’s discussion and commentary about what this kind of metric means and what you can get out of it.
Update (3/26/12): I wrote a new post showing correlation of my data with external survey data on what languages developers think are expressive.
Disclosure: Black Duck Software (which runs Ohloh) is a client.