Skip to content
Dev, staging and prod from day one: why a team of three shouldn't wait
← ← Back to Thinking Cloud

Dev, staging and prod from day one: why a team of three shouldn't wait

"We'll separate environments later, when we have users." We've heard that sentence from every small team we've worked with, and we've said it ourselves. It sounds like prudence: don't build infrastructure you don't need yet. It's actually a loan, taken out at the worst possible interest rate, and this article is the repayment schedule.

We wrote the how-to for running three AWS accounts from one CDK codebase. This is the why, aimed at the team that has one account with everything in it and a good reason to keep it that way for another quarter.

What "later" costs

Separating environments on day one is a day of work. Separating them at month six is a week. Separating them at month eighteen is a month, plus a risk you can't price. The cost doesn't grow linearly; it grows with the number of things that have accumulated state and names.

Effort to separate environments vs. project age day 1month 6month 18 1 day1 week1 month + risk: moving prod data out of a shared account 3 accounts, 1 config map, 1 OIDC role each database, secrets, DNS have names and data every integration points at the old account running cost of 3 envs from day 1: ~$60 / month

At month six the database has real data and a real name. Secrets have been copied into three places. DNS points at things. Someone's laptop has a profile called default that deploys to the only account there is. Moving production out of that account means importing stateful resources into new stacks, re-pointing every webhook and integration, rotating every secret that was shared, and doing it without downtime. We've done it. It's a project with a runbook, and the runbook has a rollback section, and you'd rather not need one.

The three things that go wrong in one account

We're not arguing from theory. Every one of these happened to a team we've worked with, and one of them happened to us.

Test data in production, or production data in test. With one account, "the database" is one cluster, and the separation is a schema name or a table prefix. A migration script with the wrong search_path, a seed command run in the wrong shell, an analytics query pointed at the wrong table: each is a one-character mistake and each one has happened. In separate accounts, the wrong-account mistake is still possible, but it requires assuming a different role, and the deploy tooling can refuse to do that without an explicit flag.

A deploy that was meant for dev. Our incident: a cdk deploy from a laptop, with the wrong context, deleted every App Runner service in an environment. It was the dev environment, in the dev account, and the total damage was an afternoon. The same command, in a single-account setup, deletes production. The account boundary didn't prevent the mistake; it bounded it. That's the whole value.

The "just this once" hotfix. A single account with no staging means the only place to test a fix is production. So the fix goes to production, and it works, and the next one does too, and eventually the team's deploy process is "push to main and watch". The habit isn't stupidity. It's the rational response to not having anywhere else to look. Give people a staging account that costs $20 a month and they use it, because it's there.

What "from day one" actually means

The objection to early separation is usually a picture of a landing zone: Control Tower, an organisation with a dozen OUs, service control policies, a shared-services account, centralised logging, a network account with Transit Gateway. That's a month of work and it's the right thing for a company with fifty engineers. It's the wrong thing for three, and it's not what we're proposing.

The minimum that captures nearly all of the value:

Do on day one Skip until you need it
An AWS Organization with three member accounts: dev, staging, prod Control Tower, landing zone accelerators
One config map in the CDK codebase, keyed by environment Organisational units beyond the default
One OIDC deploy role per account, scoped by branch Service control policies
RemovalPolicy.RETAIN and deletion protection in prod Shared-services or network accounts
A consolidated billing view tagged by environment Centralised CloudTrail into a security account
A deploy script that refuses prod without --env prod --yes Cross-account VPC peering

That's a day. The CDK diff to go from one environment to three is the config map and the env context value; everything else in the stack is already parameterised or should be. And the three accounts cost, at minimum size with Aurora paused and App Runner at 0.25 vCPU, about $60 a month more than one. That's the entire premium, and it's less than one hour of the engineer who'd otherwise be doing the month-eighteen migration.

What you get on day two

Separation isn't just a safety property. It's what makes several other good things possible, and those are the ones that pay for it:

  • Preview environments per pull request need a place where anyone can deploy anything without asking. That's the dev account, with a permissive OIDC trust policy that would be unacceptable in prod.
  • A production deploy that a human approves needs a GitHub environment with reviewers, and a role that only that environment can assume. That's the prod account's trust policy.
  • Honest cost tracking. "How much does production cost?" is a filter on account ID, not an archaeological project through tags.
  • Blast radius for credentials. A leaked dev key can't touch prod data, because it's not in that account. We deleted our long-lived keys anyway, but the boundary is the layer beneath that.
  • Hardening that differs per environment without branching in code. DESTROY in dev, RETAIN in prod, point-in-time recovery on in staging and prod, from one table in one file.

None of these are possible in a single account without building the isolation inside it by hand, with IAM policies that are harder to get right than the account boundary they're imitating.

The one-account case that's actually fine

If you're building a prototype that will be thrown away in eight weeks, one account is fine, and separating it is waste. The test is whether there's a database with data you'd be sad to lose. The day the answer is yes is the day you're past the prototype, and it's usually earlier than the team thinks: the first time a real user signs up, the first time an invoice is generated, the first time someone says "don't run that against the live one".

Do it on that day, not on the day of the incident.

If you have one account with everything in it and want the day-one setup retrofitted before it becomes the month-eighteen project, talk to us. We've done both, and the first one is much cheaper.