Continuous Integration/Continuous Deployment (CI/CD) is the on-ramp to pretty much everything good in modern software development and operations. It’s easy to assume everyone has already made the transition to automated testing and agile approaches, but of course that is not the case, so rather than writing about the newest shiniest tools I just wanted to talk about to some basics.
As is so often the case, when talking about best practice in software it’s good to start with Martin Fowler
“The fundamental benefit of continuous integration is that it removes sessions where people spend time hunting bugs where one person’s work has stepped on someone else’s work without either person realizing what happened. These bugs are hard to find because the problem isn’t in one person’s area, it is in the interaction between two pieces of work.”
The interaction between two or more pieces of work, by different teams or individuals, is the norm now – indeed that’s what we’re optimising for. See microservices. Yesterday we looked at “canarying” in this context and I promised to look at “smoke testing”.
Smoke testing defines a successful build in an automated test process – you need to start the app and run some tests against it… “to see if smoke comes out” before it can be considered to have passed. Originally used in plumbing, before being applied to electrical engineering Steve McConnell applied the term to software engineering based on his experiences working at Microsoft. The Windows NT team used daily builds and smoke tests to keep themselves on track. In 1996 McConnell wrote:
“If you want to create a simple computer program consisting of only one file, you merely need to compile and link that one file. On a typical team project involving dozens, hundreds, or even thousands of files, however, the process of creating an executable program becomes more complicated and time consuming. You must “build” the program from its various components.
A common practice at Microsoft and some other shrink-wrap software companies is the “daily build and smoke test” process. Every file is compiled, linked, and combined into an executable program every day, and the program is then put through a “smoke test,” a relatively simple check to see whether the product “smokes” when it runs.”
McConnell continues
“The smoke test should exercise the entire system from end to end. It does not have to be exhaustive, but it should be capable of exposing major problems.”
You should really read the short essay above. It’s beautifully written and packed with wisdom. I particularly like the point that smoke tests help with morale, because developers get to see something working and improving in front of their eyes. As ever the interplay between good testing strategies and a happy team is what delivers good software.
More than 20 years later and smoke testing is more relevant than ever as software becomes more distributed and modular. I wrote recently about Ben Fletcher, Principal Engineer’s work at the FT:
Smoke tests are set up with CircleCI and AWS CloudFormation, with testing taking no longer than 5 minutes. Unique IDs allow for multiple builds from the same repo. CircleCI is also used for automatic nightly builds, though not automated deploys. The idea is to “make hygiene fun”. This is really critical – developers have to buy into testing approaches, and if you can make flossing fun people are far more likely do it.
I found this post from Assertible – Set up smoke tests for REST API in 5 minutes instructive. Here are some criteria for useful smoke tests.
- fast execution
- low false/positives
- reproducible
- test core features
As ever, outliers are interesting. Netflix, for example, has smoke tests that cover the simultaneous launch if a service in 130 different countries.
Smoke testing doesn’t have to be rocket science though. It’s good hygiene. As McConnell said of Windows NT:
“Those of us who work on projects of less staggering proportions will have a hard time explaining why we aren’t also reaping the benefits of this practice.”
No Comments