![](\"https:\/\/lh6.googleusercontent.com\/4aDZHQAfRp0PzTm6kZ9Oq6gYe08OcBz5tMtT1B9mgUyB1RlBTpio03EDT6yobze4IzYLzUW9Up6tM8QzyEVvFaE5TDonEiGi-ca5Zi5p4nSGfXSB5yBFza6Sd8D4YdYd0mQDLcyM\")
<\/p>\n
If you\u2019ve attended an infrastructure conference recently, particularly of the \u201ccloud native\u201d variety, you\u2019ll likely have heard the term \u201ccanarying\u201d. If you don\u2019t know what that means you\u2019ve probably nodded along sagely and thought, oh I better look that up. I\u2019ll do it when I look up \u201csmoke testing\u201d.<\/p>\n
So anyway – what is canarying?<\/p>\n
A canary test reduces risk by pushing out a change to a small subset of users before it is deployed to your entire infrastructure, and the rest of your users. Sounds simple, but obviously you need a pretty sophisticated development and operations infrastructure to do it properly. You start by deploying the change to a subset of a load balanced infrastructure that is currently not being used. Then gradually route a small number of requests to the new service. These might be random, or employees, or some other specific set of users first. As you become more confident in the change you route more users to it, and then begin to deploy change to more servers. You do want to run the older version in parallel for a while though, in case a problem emerges and you want to route users back to that.<\/p>\n
We used to call this kind of approach \u201cphased rollout\u201d but “canarying” sounds better, and the practice has come a long way over the last couple of years, in terms of technology and process – cloud, containers, microservices. For the state of the art in canarying at scale check out what Netflix is doing. I was reading a recent post on the Netflix tech blog about the Chaos Automation Platform<\/a> and at first I thought \u201cgee whiz pass the salt\u201d but then I paused and took a step back to appreciate the breath-taking achievements of the engineering team. Of course we\u2019re not all Netflix, and we shouldn\u2019t try to emulate everything they do – premature optimisation is a killer. But that doesn\u2019t mean we can\u2019t all learn from the company, and luckily they are pretty happy to share their experiences.<\/p>\n Here is the Netflix testing approach over time..<\/p>\n Netflix asks profound questions<\/p>\n \u201cThe best experiments do not disturb the customer experience. In line with the advanced Principles of Chaos Engineering, we run our experiments in production. To do that, we have to put some requests at risk for the sake of protecting our overall availability. We want to keep that risk to a minimum. This raises the question: What is the smallest experiment we can run that still gives us confidence in the result?\u201d<\/p>\n One of the ways Netflix is an outlier is in how many changes it makes over a relatively short term frame. When changes are being deployed to microservices that you are not in control of, canarying gets more interesting.<\/p>\n \u201cAny change to the production environment changes the resilience of the system. At Netflix, our production environment might see many hundreds of deploys every day. As a result, our confidence in an experimental result quickly diminishes with time.”<\/p>\n![](\"https:\/\/lh3.googleusercontent.com\/zkcUKQgXjwKcr724qlX1dcktpNaZvjURENnhLmr7vXK42mcb3j2ZRaa5buqZXiBIsIxov2JbOfJTdRSNVJeh8hf1eZgsSSrMMoDI1tgb64LtiTSyYwvLAh-noj7j6jDp5c2TxyvJ\")