A Look at Unit Testing Frameworks

Testing is perhaps the most frequently spoken about, but most often ignored aspect of software development. As we have noted before CI is the onramp to cloud native, and to be successful with CI you need to continuously improve your testing.

The most basic level of testing that you can put in place are unit tests. It is, however, amazing to us the amount of people we talk with that do not even have basic unit testing in place. Part of this is historic, part of this is technical debt and part of it is a fear of what may be uncovered if more rigorous testing is put in place.

This last aspect, the fear of what may be uncovered, is perhaps the most insidious. It is also the easiest to address, just adding in a single test per developer per day on a project for existing functionality quickly adds up. Matt Curry of Allstate had a lovely comment about this at SpringOne Platform last year:

I think of my test suite like my bank account… I don’t get to just park $1mn in one day, but if I put in $10 a day, with compounding interest, it adds up /@mattjcurry on test automation #SpringOne

— DormAIn 🧟‍♀️ (aka "part of the problem") (@DormainDrewitz) December 5, 2017

In order to narrow the focus across the plethora of possible options available we have chosen the following criteria:

1. The unit testing framework is for one of the top five languages that appear in the RedMonk programming language rankings.
2. The framework supports the xUnit output format, making it easier to use with a tool like Jenkins.
3. The framework is an open source project.
4. The framework has a reasonable level of activity around it.
5. The framework has come up in our conversations with developers.

We have not looked at front end/UI testing (with some caveats), BDD testing, functional testing or integration testing in this analysis.

For each of the frameworks we have taken a look at the overall activity in terms of commits and issues. As always if there is a project or framework you feel we should consider please do let us know, either in the comments or on twitter.

JavaScript

JavaScript, unsurprisingly, has one of the largest arrays of potential test frameworks. We have chosen to focus on four popular frameworks, MochaJS, Jest, AVAJS and Tape.

Looking at StackOverflow data we see that MochaJS and Jest generate far more questions than either AvaJS or Tape. Jest has overtaken all the other frameworks in popularity terms on github but given its close association with React this is not particularly surprising.

 

MochaJS

The overall velocity of the MochaJS project, both in commits and issues continues to grow.

As is often the way with the most successful JavaScript projects around the node.js ecosystem, TJ Holowaychuk looms large, being by far the largest contributor. Although his contributions drew to a close in 2014.

AVAJS

AVAJS is very focussed on running tests concurrently, allowing for shorter test run times. The project is very popular but is just approaching a 1.0 release at the time of writing (March 2018).

Contributions to AVAJS are dominated by Sindre Sorhus, who accounts for over 50% of all contributions.

Jest

Jest is a JavaScript testing framework which is primarily developed by Facebook. As with many of the tools from Facebook you do need to keep in mind that the primary focus is on what Facebooks needs. Unsurprisingly Facebook dominate both commits and issues for the project, but the overall velocity is strong

While we excluded UI tests in our criteria above, Jest is frequently used for testing React based applications.

Tape

Tape, while supporting xunit output, is a TAP focused testing framework for JavaScript. It has a minimal core which is easily expanded.

Tape development is dominated by James Halliday, who accounts for almost 50% of all contributions.

Java

Java has had a strong testing culture since the very initial development of the language, and it is fair to say that jUnit is the primary reason for this. We have chosen to look at jUnit, versions 4 and 5, TestNG and Arquillian.

jUnit has a strong lead in terms of the number of questions on Stackoverflow, followed by TestNG. It is also the most popular java testing framework on github.

jUnit

The jUnit community create the 5.0 set of releases, which reached GA in September 2017, to address testing needs for Java 8 and beyond. The 4.0 train remains in extremely heavy use.

It is important to point out that jUnit has been around a long time, so our commercial attributions of contributions may have some gaps.  David Saff at Google remains the leading contributor, and we have not put any commercial attribution on the contributions of Kent Beck or Marc Phillip.

TestNG

TestNG originated out of some specific frustrations with jUnit, which its original author, and primary contributor, Cedric Beust had encountered.

 

There is a strong community around TestNG, and we hear TestNG mentioned reasonably regularly.

Arquillian

Arquillian is a testing framework which is part of the jBoss community. It has evolved in recent times to include a strong focus on testing microservices.

 

As it is a jBoss project, contributions to Arquillian are dominated by Red Hat.

Python

While there are a variety of testing frameworks for Python, we have chosen to focus on py.test and PyUnit. PyUnit is the in-built mechanism for running unit tests in Python, and py.test is the mechanism we hear about most often. As comparing PyUnit to py.test contributions is not directly possible, we only look at stackoverflow data, where we can see a strong bias towards py.test.

PHP

PHP has consistently been in the top five languages in the RedMonk Programming Language rankings, coming in at number four in the most recent rankings. As we have noted before people can be quite dismissive of PHP, but it is incredibly widely used and this usage shows no sign of abating. We looked at two testing frameworks, PHPUnit and Codeception.

PHPUnit is the undoubted leader in the PHP space. Codeception is included here as developers do use it for unit testing in PHP, although it leverages PHPUnit under the hood for this purpose.

 

PHPUnit

PHPUnit continues to be an extremely active project. The vast majority of code contributions come from Sebastian Bergmann, but the diversity of sources for issues shows a very vibrant community

Codeception

Codeception is used for a far wider set of testing, and as noted above we have included it here as we do hear it mentioned in connection with unit tests. Its community is also pretty vibrant.

 

C#

There are a number of commerical offerings for C# and .NET unit testing, but we have limited our research to NUnit and xUnit, which are by far the most frequently cited frameworks in our discussions. Stackoverflow activity shows a bias towards NUnit.

We have not included github stars for these projects, A more telling metric for both is the level of download activity on nuget. NUnit is just over 19 million downloads, while xUnit has recently gone past 13.5 million downloads.

NUnit

NUnit was originally a port of jUnit to dotnet, but has be rewritten in version 3.0 to take advantage of a specific .net features. Project leads Charlie Poole and Rob Prouse dominate the contributions here, but the community contributes a wide range of issues.

xUnit

Brad Wilson at Microsoft is by far the largest code contributor to xUnit, but the community contributes a wide range of issues. We have not broken out code commits here as authoring versus commits is not accurately reflected in our current data gathering methodology.

Credits: Ceramic Piggybank image , CC0 license by Pexels.

Disclaimers: Red Hat, Google, Microsoft, Pivotal, Oracle, IBM and Century Link are current RedMonk clients.