# Backups you've never restored aren't backups. A restore drill, timed.
Every AWS data store has backups turned on by default or by a checkbox. Aurora keeps automated backups. DynamoDB has point-in-time recovery. S3 has versioning. The console shows green ticks, the compliance questionnaire gets a "yes", and nobody has ever actually restored anything, so nobody knows what a restore takes, whether it works, or what breaks when the restored copy has a different name.
We run a restore drill every quarter, into a separate account, with a stopwatch. This is the drill from last quarter: what we restored, how long each step took, the four things that failed, and what the numbers say about our real recovery objectives, as opposed to the ones on the slide.
## What's backed up, and how
The platform's state lives in three places: an Aurora Serverless v2 cluster, six DynamoDB tables, and two S3 buckets. The backups, all defined in CDK:
| Store | Mechanism | Retention | Where the copy lives |
|---|---|---|---|
| Aurora | Automated continuous backup, point-in-time | 35 days | Same account, plus a daily snapshot copied to the backup account via AWS Backup |
| DynamoDB | Point-in-time recovery, continuous | 35 days | Same account, plus daily AWS Backup to a cross-account vault |
| S3 | Versioning + lifecycle | Noncurrent versions kept 90 days | Same bucket, plus replication to the backup account |
| Secrets, parameters | AWS Backup doesn't cover these | n/a | Rebuilt from the [CDK definitions and a checklist](/en/blog/secrets-in-cdk-secrets-manager-ssm-or-nothing-in-the-template) |
The cross-account copy is the part people skip. A backup in the same account as production protects against a bad migration or a fat-fingered delete. It does not protect against the account itself being compromised, or against a `cdk destroy` with the wrong context [deleting things with RETAIN off](/en/blog/cloudformation-deleted-our-app-runner-services). AWS Backup with a vault in a separate account, with a vault lock, is what makes the copy survive the worst day. It costs about $6 a month at our size.
## The drill, with the stopwatch
The scenario: production data as of 14:00 yesterday is corrupt; restore everything to 13:55 and bring the application up on the restored data. In staging, from the backup account, with the smoke-test suite as the definition of "up". One engineer, no help, following the runbook.
| Step | Time | Notes |
|---|---|---|
| Aurora: restore cluster to point in time from the vault copy | 22 min | New cluster, new endpoint. Serverless v2, 0.5 ACU |
| Aurora: run smoke query, check row counts against the metrics dashboard | 3 min | Matched |
| DynamoDB: restore six tables from the vault, in parallel | 31 min | The largest table, 18 GB, took 28 of those |
| DynamoDB: re-enable PITR and TTL on the restored tables | 4 min | Restore does not carry these over. Neither does it carry auto-scaling, tags or the stream |
| S3: nothing to restore for this scenario; versioning check only | 2 min | |
| Repoint the app: write the new cluster endpoint and table names to SSM | 5 min | Should have been 1 minute; see failure 3 |
| Restart App Runner services and Lambdas so they read the new parameters | 6 min | |
| Smoke tests green | 4 min | |
| **Total** | **77 min** | |
Seventy-seven minutes for a full restore to a point in time, by one person following instructions. That's the real recovery time objective, and it's the number that goes in the runbook, not the "under an hour" that was on the slide before the first drill.
## The four things that failed
The first drill, a year ago, took four hours and didn't finish. Every quarter since has found something. Last quarter's four:
**1. The KMS key policy didn't allow the staging account to decrypt the Aurora snapshot.** The vault copy is encrypted with a key in the backup account. Restoring into staging needs that key's policy to grant `kms:Decrypt` and `kms:CreateGrant` to the staging account's restore role. It granted them to production's. The restore failed with a permissions error that took twenty minutes to read correctly. Fixed in CDK: the key policy now lists every account that might restore.
**2. DynamoDB restore drops the table's settings.** A restored table has the data and the key schema and nothing else: no PITR, no TTL attribute, no auto-scaling, no tags, no stream. Our app relies on TTL to expire sessions and on the stream to feed the search index. Without a checklist step for re-enabling these, the restored app "worked" and silently stopped expiring sessions. It's now a step in the runbook and, better, a small script that applies the settings from the CDK definition to a named table.
**3. One Lambda had the table name hard-coded.** Eleven of twelve functions read their table names from SSM parameters, so repointing was a parameter write. One, the oldest, had the production table name as a literal in its environment. It kept happily writing to the *old* table during the drill, which in a real recovery would have meant writing new data into the corrupt store. Found because the smoke test for that function failed. Fixed by moving the name to SSM like the others, and by a CI check that greps environment blocks for anything matching a resource-name pattern.
**4. The restore role in staging couldn't `PassRole` to the Aurora service role.** A one-line IAM omission, discovered at minute 30, fixed at minute 35, and a reminder that the restore path has its own set of permissions that nothing exercises except the drill.
None of these would have been found by looking at the green ticks. All four would have turned a real incident from "bad afternoon" into "bad week".
## The numbers that came out of it
Recovery point objective: five minutes for Aurora and DynamoDB, because point-in-time recovery is continuous; S3 is immediate through versioning. That part was true before the drill.
Recovery time objective: 77 minutes for a full restore, about 30 for a single DynamoDB table, about 25 for Aurora alone. Those numbers were fiction before the drill and they're measurements now. They're in the runbook and in the status-page template, so the message during a real recovery says "restore in progress, expected complete by 15:20" instead of "soon".
Cost of the drill: about $4 in restored resources for the afternoon, plus one engineer's afternoon. Cost of not doing it: unknown until the day it matters, which is the wrong day to learn.
## The runbook, condensed
- Declare the target point in time. Write it down before touching anything.
- Restore into the isolated account. Never restore over production; restore next to it and repoint.
- Aurora: point-in-time restore from the vault copy; verify row counts against a known metric.
- DynamoDB: restore all tables in parallel; then run the settings script for PITR, TTL, streams, auto-scaling, tags.
- S3: identify the object versions at the target time; copy forward if needed.
- Repoint: the application reads every resource name from SSM. Write the new names. Nothing is hard-coded, and CI enforces that.
- Restart everything that caches parameters.
- Smoke tests. Not "it loads": the suite that exercises checkout, login and the search index.
- Time every step. Update the RTO in the runbook if it moved.
- Tear down, and write the one-page note with what failed.
Quarterly. On the calendar. With a different engineer each time, because the point is that the runbook works for the person who didn't write it.
If your backups have green ticks and no restore history, [we'll run the first drill with you](/contact). Budget an afternoon and expect to find something.