# Rotating the database password without downtime: the thing you've been putting off
Everyone agrees the database password should rotate. Almost nobody's does, and the reason isn't laziness. It's that the first time you try, something breaks: a connection pool holding the old password, a Lambda that cached it at cold start, a background job that read it from an environment variable set at deploy time. So the rotation gets rolled back, a ticket gets opened, and the password stays the same for two years.
We rotate ours every thirty days, automatically, and the application doesn't notice. This is the mechanism, which is mostly AWS's, and the three changes to the application that made it safe, which are ours.
## Why the naive rotation breaks things
Rotation with a single database user goes: generate a new password, `ALTER USER app PASSWORD 'new'`, update the secret. Between the `ALTER` and the moment every consumer has re-read the secret, any *new* connection with the old password is refused. Existing connections survive, because Postgres authenticates at connect time, but a pool that opens a new connection during that window fails, and a service that reads the secret only at start-up fails on every reconnect until it's restarted.
The window can be seconds if everything re-reads promptly. It can be hours if something caches. In practice it's "until the next deploy", because that's when environment variables get refreshed, and that's the outage.
## Alternating users: the rotation that never invalidates a password in use
Secrets Manager's multi-user rotation strategy uses two database users, `app` and `app_clone`, with identical grants. At any moment one of them is the *current* user in the secret and the other is idle. Rotation changes the password of the *idle* one, tests it, and then flips the secret to point at it. The user that was current, whose password every consumer might be holding, is untouched until the next rotation thirty days later, by which time every consumer has re-read the secret many times over.
In CDK, on a cluster whose credentials came from a generated secret, it's a few lines:
```ts
cluster.addRotationMultiUser('Rotation', {
secret: appUserSecret, // the secret for the 'app' user, with masterarn set
automaticallyAfter: Duration.days(30),
vpcSubnets: { subnetType: ec2.SubnetType.PRIVATE_WITH_EGRESS },
});
```
CDK deploys AWS's rotation Lambda from the Serverless Application Repository into the VPC, wires it to the secret, and schedules it. The Lambda needs to reach both the database (it's in the VPC) and Secrets Manager (an [interface endpoint or a NAT](/en/blog/nat-gateway-the-most-expensive-line-you-do-not-see); we use the endpoint). The `app_clone` user is created by the Lambda on the first rotation if it doesn't exist, with the same grants as `app`, which is the one part worth checking by hand: if `app` has grants added later by a migration, `app_clone` needs them too. We handle that by granting to a *role* that both users are members of, so grants are made once.
## The three application changes
The rotation mechanism is safe by construction. The application still has to read the *new* secret at some point, and how it does that decides whether rotation is invisible or a slow-motion outage.
**1. Read the secret at connect time, not at start time.** The pool's connection factory calls Secrets Manager (through a cache with a five-minute TTL) each time it opens a connection. A long-lived service picks up a new secret within five minutes of a flip, without a restart. App Runner's `runtimeEnvironmentSecrets` injects the value at instance start, which is fine for a service that redeploys weekly and wrong for one that runs for a month, so for the database credentials specifically we read from Secrets Manager in code rather than from the environment.
**2. On an authentication failure, refetch once and retry.** Belt and braces: if a connection attempt fails with `28P01` (invalid password), invalidate the cache, fetch the secret again, and retry once. This covers the case where the cache TTL hasn't expired at the exact moment of the flip. It's ten lines and it has fired in production exactly as many times as we've rotated: about once a month, silently, in the logs.
```ts
async function connect(): Promise {
try {
return await open(await creds.get());
} catch (e) {
if (isAuthError(e)) { creds.invalidate(); return open(await creds.get()); }
throw e;
}
}
```
**3. Let RDS Proxy do it for the Lambdas.** The functions in the VPC connect through RDS Proxy, and the proxy authenticates to the database with the secret itself: it watches Secrets Manager and picks up rotations on its own. The functions authenticate to the *proxy* with IAM, so they never hold a database password at all. Rotation, for them, is a non-event by design. It also fixes the connection-storm problem that Lambdas have with Postgres, which is [the reason we'd recommend the proxy anyway](/en/blog/aurora-serverless-v2-review).
## The first rotation, in staging, with a stopwatch
We didn't turn it on in production first. In staging, we triggered a rotation manually (`aws secretsmanager rotate-secret`), watched the Lambda's log, and ran a load generator against the API throughout. Results from the first attempt:
| Moment | What happened |
|---|---|
| T+0 s | Rotation starts; Lambda sets `app_clone` password, tests it, flips the secret |
| T+4 s | Flip complete. Every existing connection still fine (they hold `app`) |
| T+0 to T+300 s | New connections from the App Runner pool still use the cached `app` credentials. Still fine, because `app` is still valid |
| T+300 s | Cache expires; next new connection fetches `app_clone`. Works |
| T+5 min to T+30 days | Both users valid. No consumer can be caught out |
Zero errors in the load generator. The one thing that failed was a cron job in a separate account that had the password in an environment variable: it kept working for thirty days (its user was still valid) and then broke on the *second* rotation, which is exactly the delayed failure the alternating strategy produces for consumers that don't re-read. That's the point of doing it in staging first. The cron job now reads the secret.
## The checklist
- Multi-user rotation, never single-user. The extra database user is free and it's the whole difference.
- Both users get grants through a shared role, so a migration that grants to the role covers both.
- The rotation Lambda lives in the VPC and needs a path to Secrets Manager: interface endpoint or NAT.
- Every consumer reads the secret at connect time through a short-TTL cache, or goes through RDS Proxy with IAM.
- Refetch once on `28P01`.
- First rotation in staging, manually triggered, under load, with the log open.
- Then grep every environment variable, `.env`, cron and script for the old password. Whatever still has it will break in thirty to sixty days. Better to find it now.
Thirty-day rotation, six months, six rotations, zero incidents. The password nobody has read has changed six times and nobody noticed, which is what a secret is supposed to feel like.
If your database password has a birthday, [we can get it rotating in a day](/contact), staging first.