Releasing safely
Most broken checkouts at Mangrove start with a release. This lesson turns the order service's release into a series of small, watched steps: a canary that a machine judges against the old version, feature flags that switch behaviour without a deploy, and a rollback that happens by itself before anyone is paged.
By the end you can
- Separate deploying code from releasing a feature, and say what risk remains when a flag is off
- Explain what a Kubernetes rolling update checks and what it misses, and when blue-green is worth double capacity
- Write canary gates that compare the canary with the stable version, with a minimum sample size
- Set up an Argo Rollouts canary with a Prometheus analysis that aborts on its own
- Design percentage rollouts with stable hashing, and keep database changes and mobile apps safe to roll back around
The everyday picture
A restaurant kitchen is changing its most popular curry. The cooks learn the new recipe and keep the ingredients ready in the kitchen. That is the deploy: the new thing is in place, but no guest has tasted it yet. The release is the moment the waiter starts serving it.
A careful chef does not serve the new curry to every table at once. Two tables get it first, and the chef watches their plates next to the plates of tables still eating the old recipe on the same evening. If the new plates come back half eaten, the chef stops serving it at once and goes back to the old pot, which is still warm on the stove. Only when the new plates come back as clean as the old ones does the whole room get the new curry.
In this lesson the two tables are the canary, the other tables are the baseline, comparing plates on the same evening is canary analysis, and the warm old pot is a fast rollback.
Deploy and release are two different events
A deploy puts new code on production servers. A release is the moment users get the new behaviour. When the two happen in one step, every deploy is a bet on all users at once. When you split them, each step gets smaller and easier to undo.
Three things decide how much damage a bad change can do, and every technique in this lesson works on one of them:
- Who is exposed. The blast radius: all customers, 5% of requests, or ten staff accounts.
- How fast you notice. A dashboard someone checks after lunch, or a metric a machine reads every minute.
- How fast you undo. Rebuilding an old commit takes many minutes. Moving traffic back to pods that are still running takes seconds.
Shipping safely introduced rolling, blue-green and canary releases and feature flags. Here you make them concrete on the order service's Kubernetes cluster from Kubernetes for application teams, and add a rollback that no human has to start.
Splitting deploy from release does not make a deploy free of risk. With the flag off, the new image still starts, loads its config, opens database connections and runs any code outside the flag: a new library version, a changed middleware, a background job. That is why even a "flag off" deploy goes out as a canary.
Quick check
Version 2.14 of the order service is deployed with the new wallet retry logic behind a flag that is off for everyone. Which risk is still live?
A flag only guards the code inside its if. The rest of the new image runs for every request from the moment it starts, so library upgrades, config and background jobs are released by the deploy itself. A well-written flag falls back to its safe default (off) when the flag service is down, so that is the smaller worry.