Checking In on the State of TDD

I recently asked Angie Byron, director of community for Aiven and core product manager for Drupal, if she had any feelings about test-driven development (TDD). “Lots” she assured me, before outlining testing’s role in Drupal core (more on that later). I tend to hear this degree of enthusiasm followed by opinionated takes from most every developer I ask about TDD. Although since the mid-2010s a vocal group of software engineers have abandoned TDD, with David Heinemeier Hansson’s “TDD is Dead” offering a particularly polarizing (but fun to read) artifact of this sentiment, I argue that rumors of TDD’s demise have been greatly exaggerated—at least in spirit.

Beginning in 1999, TDD became the gold standard for ensuring quality code. According to Kent Beck’s definition from his Test-Driven Development by Example (2000), this methodology starts with writing tests around requirements (the test will fail at this stage), then writing the simplest code that fulfills these specifications (the test should now pass), and then refactoring as needed (while running tests after each refactor). For years this system has been taught in CS programs, and upheld as the best means for ensuring that only high quality software is shipped. But in the past 15 years TDD has fallen increasingly out of favor. To quote DHH:

Test-first fundamentalism is like abstinence-only sex ed: An unrealistic, ineffective morality campaign for self-loathing and shaming.

I’ll hold off on digging into the nuance of this and other’s complaints against TDD (often disparaged as “test-first,” which has been around since the 1960s and is a principle of extreme programming) because my post is not intended to be a recap of this now well-trod backlash. Instead, I will be looking forward at the state of TDD today in order to weigh its legacy and speculate about where it will lead in the future. Much has changed in the QA and testing landscape recently owing to the rise of a new breed of testing tools and automation-driven techniques. I show that TDD continues to be a controversial subject among developers, and for very good reasons.

TDD, Agile, & Cloud

TDD’s longevity can be largely attributed to its close ties (historical and ideological) with two landmark shifts in the software development industry: 1) widespread adoption of Agile and 2) the rise of cloud. That is not to say that TDD emerged fully formed and without predecessors. It’s Red, Green, Refactor loop methodology is an extension of the industry’s longstanding Implement, Re-implement, rinse & repeat cycle (see Beck’s test && commit || reset (TCR)). But TDD’s close ties to Agile and cloud mean that it accomplishes something significant in the testing space that has transformed the way developers write code.

TDD appeared during the rise of Agile methodologies (the Agile Manifesto was published in 2001). Although the affiliation between Agile and TDD is intersectional rather than core (see this Stack Overflow question), Agile’s emphasis on speed made TDD necessary to its success. Software teams require processes in place to avoid pushing shoddy code into production, and TDD succeeded brilliantly in this. Leaders from the Agile community like AgileSherpa have advocated TDD because: “As the team embraces changing requirements as a part of a living, breathing system, they need to know that their changes will not cause a domino effect of broken code.”

TDD’s appearance at the turn of the century also occurred at a significant moment for cloud, coinciding with Salesforce’s introduction of SaaS. Cloud had tremendous implications for QA. Shipping patches and updates online was much easier than in the past when software was distributed by means of compact discs, or, going way back, floppys. It was relatively easy to test and ship web browser hosted apps, but this flexibility made testing seem less essential to the SDLC. What is more, many now perceive TDD as actively antithetical to software development best practices.

“TDD is Dead”

The anti-TDD movement is not the result of laziness or even DHH-style hyperbole. Some of the most disciplined, buttoned up, and conscientious engineers in the industry have fallen out of love with TDD (see Robert C. Martin (Uncle Bob) in 2016, Davide Fucci, et al. in 2016, Eric Gunnerson in 2017, and Neopragma in 2019). TDD is not appropriate for all use cases according to Ian Sommerville, who devotes a chapter of Engineering Software Products (2019) to TDD. He reflects:

I started with TDD on a class of system that met these [TDD-friendly] criteria and I liked it. It worked well for me. Then, I moved on to a system which was concerned with visualizing complex linked structures. Now, the thing about visualization is that (a) it’s often much more difficult to have clearly separate layers – the UI is the program and (b) it’s very hard to have pre-defined success criteria – you basically have to program by experiment and see what works and what doesn’t. TDD didn’t work.

On the whole, complaints against TDD follow Sommerville’s experience that this methodology is needlessly slow and rigid. TDD restricts the ability to tinker and experiment—a necessity in modern engineering practice. TDD’s strictness fails to accurately reflect the lived processes of developers by constraining intuitive authoring procedures. As Rajiv Prabhakar explains:

Insisting on writing tests first, often gets in the way of the exploratory work – work that is needed before you can iron out what the right interfaces, methods, and OO-structure should be.

This resembles PheonixPharts’s Hacker News comment:

The trouble with TDD is that quite often we don’t really know how our programs are going to work when we start writing them, and often make design choices iteratively as we start to realize how our software should behave.

If an increasing number of developers, including respected and established lights from the community, have moved on from TDD it is no great surprise that newer, often less cautious devs have absolutely no patience for it. Testing already has little place within the “move fast and break things” ethos (including even the less-rigid Behavior Driven Development (BDD)), so TDD’s test-first methodology just won’t fly. We see this particularly in the startup space. According to self-described “startup junkie” Daniel Markham:

TDD/BDD doesn’t fit the mold of startups. Here’s why:

TDD/BDD assumes you know the problem and are coding to create a solution. In startups, however, you do not know the problem. Sure, you can imagine some kind of customer that might want some kind of code to do something or another. But that’s all pipe dreams. You have no idea if method X is useful or not. Sure, you might know if method X meets the requirements of your imagination, or your architecture dreamed up in your imagination to deal with your imaginary customers with their imaginary problems, but you really don’t know.

So it’s a waste of time. Startups are all about providing value — not flexibility, not bug-free code, not code you can hang on a wall and take pictures of. Value. If you want to be the guy who can make business solutions happen, the guy that customers can come to with any problem and you can make it all happen, you need to bone up on this stuff. But in the business world, you’ve already got the money, your job is to make the solution happen. In the startup world, you don’t have the money yet. Big difference. Big, big difference.

Look at it this way: 19 out of 20 startups fail. That means that odds are that you will never see this code again. You’d be a fool to spend any more time on it than absolutely necessary. But the math works out completely differently in the commercial world, where most things you write stay around forever.

The overhead (labor, resources) needed to adhere to TDD does not necessarily make it a zero interest phenomena, but it does signal the popular perception that test-first is only practicable for maintaining obtuse legacy codebases. Greenfield projects have less need to adhere to TDD because there are less dragons lurking in the project’s unsounded depths. In the enterprise “commercial world,” caution is needed to prevent calamity, and TDD offers a time-tested means for ensuring success.

Software Craftsmanship & Open Source

While TDD is on the decline, the majority of professional software developers continue to feel strongly about the importance of Software Craftsmanship, meaning a quality-centered approach to writing code. Andrew Hunt and David Thomas espouse this philosophy in their landmark 1999 book The Pragmatic Programmer (which gave us rubber duck debugging) by suggesting that quality code is not the result of prescriptive methods, but comes instead from curiosity, experimentation, and critical thinking.

Engineers applaud high-quality code. They admire virtuosic programmers (only don’t get them started on the 10x engineer myth). They pride themselves on being able to determine how well an app is written by its Code Smell, a concept coined by Kent Beck and adopted by Martin Fowler and Robert Martin. Even if an app technically works, if it is not written well in terms of design, verboseness (DRY), large classes, etc, it stinks! While deeply subjective, code smell and the entire software craftsmanship movement signals the larger feeling among developers that quality is important, but TDD and quality are not coterminous.

The issue of code quality is a bugbear for many OSS maintainers. The democratic, volunteer-led, and loosely structured nature of many OSS projects has made testing necessary to account for the committers’ unpredictability. Because OSS contributors cannot be forced to fix issues they create in the codebase, testing is necessary to ensure that the project works as it should following every commit. For this reason many OSS projects continue to rely on TDD, or its less rigid sibling BDD, to avoid playing bug whack-a-mole.

In speaking with Byron about where TDD stands in 2023, she emphasized the need for testing in OSS projects like Drupal core. The sense that I got from this conversation is that the legacy of TDD very much alive because testing is essential and, even if it doesn’t come first, tests absolutely must accompany every PR. Commits to Drupal core must be accompanied by unit tests, which means that:

Drupal is very prescriptive about testing, with all the good and the bad that come with that.

Testing is a central part of Drupal’s community, with events like its chocolate-fueled Awesome Testing Party and its commitment to automated testing using PHPUnit, and I suspect this testing-focused approach extends to most other OSS projects (and a future post).

Software Development Post-TDD

So what is the state of TDD in 2023? Today, developers characterize their methods for ensuring code quality beyond TDD capaciously. In fact, many have coined new colloquialisms (at varying levels of tongue-in-cheek) to reflect the type of tests they actually write.

1. Behavior Driven Development (BDD): Perhaps the most established and popular variant. In this Agile team methodology users (often customers) write executable test specifications in plain language or pseudocode, and then developers translate these into unit tests (which fail), then write the simplest code to make the test pass.
2. Test During Development: allows for authoring code first and tests afterwards, but only where they make sense. For instance, anything you have to repeat in the console should be a unit test.
3. Mutation Driven Development: in which developers reach a state where they have both code and successfully passing tests and then inject bugs to ensure the test fails and thereby verify adequate test coverage.
4. Weak TDD: According to Hillel Wayne, this method involves “‘writing tests before code, in short feedback cycles’. This is sometimes derogatively referred to as ‘test-first’. Strong TDD follows a much stricter “red-green-refactor” cycle.”
5. Annoyance Driven Development: popular among game developers, who write tests to debug an irritating UI.
6. Just F***ing Fix It (JFFI, pronounced “Jiffy”): Tests are added as part of a bug fix effort.

Clearly TDD is far from dead, but it has evolved dramatically to fit the requirements of contemporary organizations. As the more relaxed testing practices listed above suggest, there continues to be an imbalance between the business requirement of velocity (particularly for startups), and need for stable, high quality code. No one doubts that tests can be helpful, even if they aren’t a core aspect of realworld development processes. However the need to follow more rigid testing practices like TDD is just not practicable or even desirable. What some have termed the Modern Testing ethos recognizes the diversity of requirements and situations that require tests outside the rigid TDD schema. Beyond unit tests, developers need to ensure the resilience of their code through integration testing, end-to-end testing, fuzzing, and property testing—which sometimes exceed TDD’s remit. The reality is that TDD has become an ideal, rarely adhered to but ever present as a light guiding software engineers to code excellence.

Disclaimer: Aiven is not currently a RedMonk client.