# The day CloudFormation deleted our App Runner services: a post-mortem and the four safety nets we added
On 30 April 2026, a single `cdk deploy` run from a laptop deleted every App Runner service in our dev environment. No data was lost, nobody outside the team noticed, and the services were back within the hour. It was still the most instructive incident of the year, because the tool did exactly what it was told to do and the person running it did nothing that looked wrong.
This is the post-mortem, written for engineers who use AWS CDK and CloudFormation and have never seen this failure mode. The platform is a client system we build and operate: three Next.js apps on App Runner, Lambda-backed GraphQL, DynamoDB, Aurora, all in one CDK stack per environment. We're not naming the client; the mechanics are what matter.
## What happened, step by step
Our CDK stack had a context flag, `hostingReady`, introduced during the initial bring-up. The first deploy of a new environment has to create the ECR repositories *before* any App Runner service can reference an image in them. So the stack was written as: if `hostingReady` is false, synthesize everything except the App Runner services; if it's true, include them.
The GitHub Actions workflow always passed `-c hostingReady=true`. The flag defaulted to **false**, which was the "safe" value for a first deploy and the wrong value for every deploy after it.
An engineer ran `cdk deploy PlatformStack-dev` locally to push a small change. No context flag. CDK synthesized a template without the three App Runner services. CloudFormation compared the new template with the deployed stack, saw three resources that were no longer declared, and did what a declarative tool is supposed to do: it deleted them.
Recovery was straightforward: re-run the deploy with the flag, wait for App Runner to pull the images and pass health checks. The container images were still in ECR, the databases were untouched, Cognito was untouched. The blast radius was "the apps were down for about an hour in dev".
## Why this is a design failure, not a human error
The tempting conclusion is "the engineer should have passed the flag". We rejected that, for three reasons.
First, CDK does not tell your app which command is running. By the time your TypeScript executes, `cdk synth`, `cdk diff` and `cdk deploy` look identical. You cannot write "if this is a deploy, refuse". Any defence has to work at the template level or at the CloudFormation level.
Second, the default value of a flag is a design decision. A flag that defaults to the destructive value is a loaded gun on the table. The one-time bring-up case should have been the opt-in, not the everyday case.
Third, CloudFormation's deletion behaviour is correct and will not change. Declarative infrastructure means "the template is the truth". If the template lacks a resource, the resource goes. The only question is whether you've told CloudFormation that certain resources are too important to delete without a fight.
## The four safety nets
We shipped all four within a week. They are independent layers; any one of them alone would have prevented the incident, and together they cover cases the others miss.
### 1. Safe-by-default flags
`hostingReady` now defaults to **true**. The bring-up case, where you need to create ECR repositories before services, is `-c skipHosting=true`, which nobody types by accident. Every context flag in the stack was reviewed with the same question: "if someone forgets this, which direction does the mistake go?" A forgotten flag must never remove a resource.
### 2. Termination protection on the stack
```ts
new PlatformStack(app, `PlatformStack-${config.stage}`, {
config,
env: { account: config.account, region: config.region },
terminationProtection: true,
});
```
This stops `cdk destroy` and the console's "Delete stack" button until an operator explicitly flips it off. It would not have prevented the April incident on its own, because the stack was updated, not deleted, but it closes the neighbouring door. It costs nothing.
### 3. A RETAIN aspect on critical resource types
This is the one that directly addresses the incident. A CDK Aspect walks every construct in the tree after synthesis and pins the CloudFormation `DeletionPolicy` to `Retain` for resource types we consider critical:
```ts
const PROTECTED_TYPES = new Set([
'AWS::AppRunner::Service',
'AWS::Cognito::UserPool',
'AWS::DynamoDB::Table',
'AWS::RDS::DBCluster',
'AWS::SQS::Queue',
'AWS::S3::Bucket',
'AWS::SecretsManager::Secret',
]);
export class ProtectCriticalResources implements IAspect {
visit(node: IConstruct): void {
if (!CfnResource.isCfnResource(node)) return;
if (!PROTECTED_TYPES.has(node.cfnResourceType)) return;
const current = node.cfnOptions.deletionPolicy;
if (current && current !== CfnDeletionPolicy.RETAIN) return; // respect explicit choices
node.applyRemovalPolicy(RemovalPolicy.RETAIN);
}
}
Aspects.of(stack).add(new ProtectCriticalResources());
```
With `Retain`, when a template stops declaring a resource, CloudFormation removes it from the stack's bookkeeping but leaves the actual AWS resource running. In the April scenario the services would have kept serving traffic, and the next correct deploy would have needed an import instead of a create. That's an annoyance, not an outage.
Two design choices worth noting. The aspect respects explicit decisions: if a construct set `DESTROY` on purpose (a dev bucket with `autoDeleteObjects`, for example), the aspect leaves it alone, because overriding it would either break synth or silently undo a deliberate choice. And the aspect applies in every stage, including dev, because the incident risk is the same wherever real users or real test data live.
The trade-off: a deliberate `cdk destroy` now leaves orphaned resources behind. Cleanup is a manual delete per resource. We accept that; it's the explicit price of safety.
### 4. Deploys happen from CI, and the CLI tells you so
The last layer is procedural. All three environments deploy from GitHub Actions via OIDC-assumed roles, each triggered by its own branch. A local `cdk synth` or `cdk diff` is fine and encouraged. A local `cdk deploy` is not, and since CDK can't block it, the app prints a banner whenever it's running outside CI:
```
⚠️ CDK is running outside of CI.
`cdk synth` and `cdk diff` are safe to run locally.
`cdk deploy` from a laptop is the cause of the 2026-04-30 hosting
incident — push to the dev branch and let GitHub Actions deploy.
```
The banner is suppressed by `CI=true` or by an explicit acknowledgement variable. It's not a technical control, and we don't pretend it is. It's a reminder at the exact moment when a reminder is useful, and it references the incident by date so nobody has to ask why.
## What we'd tell you to check today
You don't need to have this incident to benefit from it. Three questions to ask about your own CDK stacks this week:
1. **Which of your context flags or environment variables, if forgotten, removes a resource?** Flip their defaults.
2. **What's the `DeletionPolicy` on your databases, user pools and queues?** Run `cdk synth` and grep the template. If the answer is `Delete` or absent, that's a one-line aspect away from being fixed.
3. **Can a laptop deploy to production?** If yes, what stops a wrong `AWS_PROFILE`? Account pinning in `env` and a CI-only deploy path are both cheap.
One more lesson we learned from a second, smaller incident in August: CloudFront aliases and a certificate that had been attached by hand were wiped by the next CDK deploy, because CloudFormation replaces the whole distribution config. Same root cause in a different costume: **anything that's not in the template doesn't exist**. Retain policies protect resources; they don't protect properties. The only defence for properties is to put them in code.
If you'd like a second pair of eyes on your CDK setup before it teaches you this lesson itself, [get in touch](/contact).