KubeCon + CloudNativeCon Europe 2021 Virtual

Your K8s apps are instrumented for observability. You are using ingress controllers/service meshes in your production K8s cluster and can shift traffic between different versions of your app. You wish to take your CI/CD to the next level by introducing metrics-driven automated rollouts using live experiments like canary, A/B, and A/B/n comparisons. What could go wrong? We demonstrate how subtle differences in the design of the experiment---how metrics are collected, queried, and used; the traffic shifting policy; the number of requests sent to different versions during the experiment and its duration; and when/how it is terminated---can lead to dramatically different outcomes, and in turn, directly impact the version of the app chosen to run in production. We also discuss simple and statistically effective remedies for the above problem, so that experiments become repeatable and their outcomes are more accurate and trustworthy.