There is a paradigm shift in the SDLC happening around testing. The role of the QA engineer, which has become increasingly underappreciated and often underpaid in recent years, is changing rapidly thanks to an explosion of new technologies and workflows. Who writes the tests, when tests are written, and what is being tested are all subjects that have come under scrutiny thanks to the seismic changes—both structural and technical—reshaping the industry.
As a QA-engineer-turned-industry-analyst, I can confirm whisperings of a revolution brewing in the testing space. Powerful tools like Playwright Codegen, Cypress Test Runner, Applitools, Ghost Inspector, Katalon, GitHub Copilot, LambdaTest, and Percy by BrowserStack, are rapidly making test creation (and maintenance) more easy, intuitive, and efficient.
In fact, this new breed of testing tools and automation-driven techniques go further than their predecessors. Many harness AI and machine learning. Ever since it was open sourced in 2015, Meta’s static analysis tool Infer has boasted its use of AI to identify bugs in Java and C/C++/Objective-C code. More recent players in this space include Diffblue cover, a tool that uses AI to generate unit tests for Java code. Perhaps the most hyped product is Github’s AI coding assistant, Copilot. Although Copilot isn’t a testing tool per se, it boasts the ability to “[Speed] up test generation,” and as it matures will occupy an increasingly significant portion of this market.
Another development is the number of testing tools that utilize no/low code methodologies. Mabl markets its products as “low-code test automation,” while Katalon argues that its “Codeless test automation tools are the solution to the common problems for QA teams.” Now, developers’ opinions about low/no code tools tend to be polarizing, and there is a very vocal contingent opposed to these black box solutions. However, for engineers that have historically avoided test authoring because it is noncreative, time-consuming, and thankless, even a no/low code solution promises to be an improvement.
Interestingly, other tools have crossed over into the observability and monitoring space. The SaaS Playwright product Checkly downplays testing and QA in their marketing materials, and instead characterizes their approach as a “Monitoring as code workflow,” similar to Sumo Logic’s Sensu Go.
What interests me most about these new testing products is their impact on engineering culture. Can these tools successfully meet the calls to “shift left” the work of QA? Will they make it easier for folks who are still doing a ton of manual testing to automate a larger part of their tests?
Many in the space are making grand claims for their product’s revolutionary potential. According to Debbie O’Brien, Senior Program Manager at Microsoft specializing in Playwright, developers know they should be writing tests, but they don’t owing to time/money constraints. To meet this challenge she recommends using Playwright’s Codegen, a tool that generates tests by recording your actions while clicking through your application on the browser. The promise of Codegen is transformative:
So what if we could make it easier for developers, soften the learning curve when it comes to writing tests so they can at least get started, and then help them build their tests along the way without just telling them to go read the docs.
Codegen offers a vision of a fundamentally different future for QA—one in which developers write their own tests voluntarily because it is quick, easy, and fun. There may be something to this claim. According to the 2022 State of the Octoverse, Playwright is gaining in popularity and ranks at #5 in the category of OSS projects with highest contributor growth.
In addition to changing how developers test their applications and who writes these tests, this revolution in QA is reshaping how the industry defines quality altogether. Quality is indeed a fraught idea in the era of chaos engineering. While testing is a foundational part of maintaining high quality Clean Code, developers often profess to dislike writing tests. Beyond David Heinemeier Hansson’s 2014 “TDD is Dead” (a discussion I leave for a later post), engineering culture is undergoing a disruption around ideas of code quality and accountability. Some of this comes out of security concerns. Recent regulatory requirements regarding the software supply chain and SBOMs underscore the industry’s recognition that shipping quality code must be top of mind (for more on this topic see my colleague Rachel Stephens’s excellent “Developer Experience in Security“).
But this shift is also disruptive on a business and organizational level. Businesses have historically perceived testing as a cost rather than a value. Therefore, questions of who should be writing tests are particularly fraught. What will the work of traditional players in the testing space (QA engineers, test managers, test coordinators, test analysts) look like 5, 10 years in the future? Moreover, how will structural and technological shifts in the testing space overhaul what is being tested, both in terms of types of testing (unit tests; integration tests; e2e testing, etc.) and the different types of apps (or parts of apps) that are being tested: web apps and APIs are common, but where do backends, platforms, etc.
QA has a long and important history in the software industry, and this new generation of automated tools is fundamentally disrupting the ways in which developers work.
Disclosure: Microsoft (GitHub, Playwright) and Sumo Logic are RedMonk clients.
No Comments