# Zero-downtime deploys aren't a platform feature. They're a schema-migration discipline. Every container platform will tell you it does zero-downtime deployments. App Runner, Fargate, Cloud Run, Kubernetes: they all start new instances, wait for them to be healthy, shift traffic, and stop the old ones. The application is never down. That part is true, and it's also the easy part. The hard part is the database, because the database doesn't roll. At the moment the new code starts serving requests, the old code is still serving requests too, and both are talking to the same schema. If the new code needs a column the old code doesn't know about, or the old code needs a column the new code just dropped, someone gets an error, and the platform's rolling deploy delivered exactly zero downtime to a system that is nonetheless broken. So the discipline isn't in the platform. It's in how you change the schema. This is the set of rules we run, the pattern that runs them, and the Lambda that applies them. ## The one rule everything follows from **Every migration must be compatible with the code that is currently running, and every code change must be compatible with the schema that is currently deployed.** Because a rolling deploy means the previous release and the next release run at the same time, for minutes, against one schema, the schema has to work for both. That's it. Every other rule is that one applied to a specific case. The consequence people resist: a change that in a development database is one step, "rename column `phone` to `phone_number`", is in production three or four deploys, and the schema is in an intermediate state for days. That's not a sign you're doing it wrong. It's what the rule costs, and it's much cheaper than the alternative. ## Expand, migrate, contract The pattern is old and it's still the whole answer. Every breaking change is split into an expanding step that only adds, a migration of data or code that uses both shapes, and a contracting step that only removes, with a deploy between each.
Rename phone → phone_number, with old and new code overlapping at every rollout 1 · expandADD COLUMN phone_number NULLold code ignores it · safe under N-1 2 · dual writewrite both · read new ?? oldbackfill job copies rows in batches 3 · switchread and write phone_number onlyold column present, unused 4 · contractDROP COLUMN phonenothing reads it any more running code during each rollout: v1 + v2 · both fine with a nullable extra column v2 + v3 · both write, both read with fallback v3 + v4 · old column unused by either v4 + v5 · drop is invisible to both Four deploys instead of one. Each one can be rolled back by moving the image tag, because the schema works for the release before it. The intermediate schema lives for days. That's the price, and it's the only price.
Some cases, spelled out, because the pattern is easy to nod at and easy to get wrong in detail: - **Adding a column:** nullable or with a default, in one step. Never `NOT NULL` without a default, because the old code doesn't set it. - **Adding an index:** `CREATE INDEX CONCURRENTLY` in Postgres, outside a transaction. A plain `CREATE INDEX` locks the table for writes for as long as it takes, which on a large table is the outage the rolling deploy was supposed to prevent. - **Renaming anything:** the four steps above. There is no shortcut. - **Changing a type:** add a new column of the new type, dual-write, backfill, switch, drop. Same four steps. - **Dropping a column:** only after a release has shipped that doesn't reference it, and after you've confirmed in production that nothing does. ORMs that `SELECT *` will surprise you here. - **Adding a constraint:** add it `NOT VALID` first, which enforces it for new rows only, then `VALIDATE CONSTRAINT` in a later step once the backfill has made the existing rows compliant. ## Migrations are not backfills A migration changes structure. It runs in seconds. A backfill changes data. It can run for hours. Putting the second inside the first is how you get a deploy that holds a lock on the orders table for forty minutes while every request waits. Our migrations are DDL only, plus at most a data change that's guaranteed to touch a bounded number of rows (a lookup table, a config row). Anything that touches "all rows of a large table" is a backfill, and a backfill is a job: a Lambda or a worker that processes a batch, records its checkpoint, and is invoked again, with a concurrency of one. It is idempotent, it can be stopped and resumed, and it runs *between* deploy 2 and deploy 3 for as long as it needs to, while the dual-write code keeps new rows correct. ```sql -- migration 0042, runs in seconds ALTER TABLE customers ADD COLUMN phone_number text; -- backfill, runs as a job in batches of 5,000 until it reports zero rows UPDATE customers SET phone_number = phone WHERE id IN (SELECT id FROM customers WHERE phone_number IS NULL AND phone IS NOT NULL ORDER BY id LIMIT 5000); ``` ## Where the migration runs: a Lambda in the deploy The migration has to run against the production database, from something that can reach it (it's in a private subnet), with credentials that can alter the schema (the application's own role can't, on purpose), before the new code starts but after the new image exists. That's a very specific set of requirements and it maps onto exactly one thing in our stack: a Lambda inside the VPC, with its own database role, invoked by the deploy script between "images built" and "service rollout".
build imagesnew code exists, not running schema-migrate Lambda · in the VPCpg_advisory_lock · apply pending, in orderrecord in schema_migrations · unlockown DB role with DDL · 5-minute timeout ok rolloutold and new code overlap, schema fits both verifyhealth · release SHA on failure: stop here · no new code has served a request · fix forward
The Lambda is about eighty lines. It opens a connection with the migration role, takes `pg_advisory_lock(42)` so two deploys can't race, reads the `schema_migrations` table, applies each pending file in order inside its own transaction (except the `CONCURRENTLY` ones, which can't be in a transaction and are marked as such), records each one, and releases the lock. It's invoked synchronously by the deploy script, which does not proceed to the rollout unless the invocation returns success. A five-minute timeout is the enforcement of "migrations are not backfills": if it takes longer than that, it was a backfill, and the deploy fails before it hurts anyone. The same Lambda, with the same code, runs against [each pull request's database](/en/blog/preview-environments-per-pull-request-on-aws) on stack creation and against staging on every merge to `main`. By the time a migration reaches production it has run a dozen times. ## Rollback, and why we don't write "down" migrations A code rollback is a tag change: point the service at the previous image, four minutes, done. A schema rollback is not a tag change, and pretending it is by writing a `down()` for every migration produces a false sense of safety, because `DROP COLUMN` in a down migration destroys the data that arrived since the up migration ran. So we don't write them. The safety comes from the discipline instead: because every migration is compatible with the previous release, rolling the *code* back is always safe, and rolling the *schema* back is never necessary. If a migration itself is wrong, the fix is a new migration that goes forward. In eighteen months we have wanted a down migration exactly zero times, and we've rolled code back six times, each one uneventful. ## The checklist we put in the pull request template - Does this migration work if the currently deployed code keeps running against it? If no, split it. - Does the code in this PR work against the currently deployed schema? If no, the migration ships first, in its own PR. - Any `NOT NULL` without a default, any rename, any type change, any drop? Then it's an expand/contract sequence: which step is this? - Any index on a table over a million rows? `CONCURRENTLY`, outside a transaction. - Any statement that touches more rows than you can count? That's a backfill; move it to a job. - Will this run in under a minute in production? If unsure, time it against a staging snapshot. Six questions. They add about ten minutes to a schema PR and they've removed the category of incident where the platform did its job perfectly and the users saw errors anyway. If your deploys are zero-downtime until they touch the database, [we'll help you draw up the discipline](/contact). It's mostly the checklist above and one Lambda.