Deploys, migrations and safe releases

People design for hardware failure, which is rare, and skip the most common trigger of downtime: someone shipped a change. Redundancy does nothing about a bug deployed to every machine at once.

The interesting question is not "how do you deploy" but "how do you make a bad change cheap": cheap to notice, cheap to stop, and cheap to undo.

Rolling deploy is the baseline, and the load balancing page covers the mechanics: take an instance out, drain its in-flight requests, update it, wait for the health check, put it back. The property that makes it work is that the new version has to run happily alongside the old one, because for a few minutes both are live. Everything else in this topic follows from that one constraint.

Canary goes further: send a small share of traffic (1%, then 5%, then 25%) to the new version, and compare error rate and latency against the old one before continuing. The value is that a bad release affects 1% of users for five minutes instead of 100% for however long it takes someone to notice. Say what you compare and what stops the rollout automatically, because "we would watch it" is not a mechanism.

Blue-green runs two full environments and switches traffic between them. Rollback is instant, which is its selling point. It costs double the infrastructure during the switch, and it does not solve the database, which is shared between both and cannot be switched back.

That last point is the important one: rollback is easy for code and hard for data. Once a migration has changed a schema or rewritten rows, going back is not a matter of redeploying. So the safe pattern is expand and contract, and it is worth being able to recite. Add the new column without touching the old one. Deploy code that writes both and reads the old. Backfill the existing rows in batches. Deploy code that reads the new column, and verify. Only then, in a later release, drop the old column. Five steps, several days, and at every point you can stop and go back.

Some specifics that make migrations survivable. Backfill in batches with a pause between them, because one enormous update locks a table and takes the site down. Adding a column with a default rewrites the table on some database versions (PostgreSQL before 11, for example). Know whether yours does. Adding an index should be done without blocking writes, which most databases support with an explicit option people forget to use. And a migration should be separate from the deploy that uses it, so a slow migration does not hold up a release, and a rollback of the code does not require a rollback of the schema.

Feature flags separate deploying from releasing: the code ships dark, is enabled for staff, then a percentage, then everyone, and switching it off does not need a deploy. Two disciplines keep them from becoming the problem. Flags need an owner and a removal date, or you accumulate hundreds and no combination is tested. And a flag that changes data shape is not a safe flag. Turning it off later leaves rows written in the new shape.

Finally, the rollback decision should be automatic and boring. Define, in advance, the error rate or latency that stops a rollout, and wire it up. An incident where someone has to decide whether to roll back, while being asked for updates, is an incident that lasts twice as long.