Two stories about the same deploy pipeline, told together because the second exists because of the first.
The first is a bug that isn't a bug: we rebuilt a container image, pushed it under the same tag, ran cdk deploy, watched it report no changes, and spent an hour understanding why production was still running the old code. The second is what happened when our CI was unavailable for a stretch, for billing reasons outside our control, and we needed a way to deploy from a laptop that was safer than the way we'd once deleted three services from a laptop.
Part one: CloudFormation compares strings, not bytes
The App Runner service is a CloudFormation resource, and the image it runs is a string property: ImageIdentifier: 123456.dkr.ecr.us-east-1.amazonaws.com/api:abc1234. When you run cdk deploy, CloudFormation diffs the new template against the last one it applied. If the string is identical, there's no diff, no update, and App Runner keeps serving whatever it pulled last time.
That's correct behaviour, and it bites in exactly one situation: the tag is the same but the image behind it isn't. We hit it when we rebuilt the same commit with a changed build argument. Same git SHA, same tag api:abc1234, different bytes in ECR, and CloudFormation, correctly, saw nothing to do. The registry's tag now pointed at a new digest; the service was still running the old one; nothing anywhere was wrong except our expectation.
There are two honest fixes. The first is a rule: every deploy moves the SHA. If you want a different image, commit something. It's simple and it's what we did for a while, and it fails the moment someone needs to rebuild an image for a reason that doesn't involve a code change, like a base image security patch.
The second is to pin by digest. After the image is pushed, the deploy script asks ECR what the tag resolves to and passes the digest, not the tag, into the template:
digest=$(aws ecr describe-images --repository-name api --image-ids imageTag="$TAG" \
--query 'imageDetails[0].imageDigest' --output text)
npx cdk deploy platform-prod --context apiImage="$REPO/api@$digest"
// in the stack
imageRepository: { imageIdentifier: this.node.tryGetContext('apiImage'), imageRepositoryType: 'ECR' }
Now the template changes whenever the bytes change, and only then. A rebuild of the same commit with different contents is a real deploy. A re-run with identical contents is a genuine no-op. CloudFormation's behaviour didn't change; we just gave it the string that actually means "the image".
The digest shows up in the CloudFormation console and in cdk diff as an unreadable 64-character hash. We print tag → digest in the deploy output so a human can still tell what's going out.
Part two: deploying from a laptop, safely
CI went away for a while. The platform still needed deploys. We'd already learned, expensively, that cdk deploy from a developer machine with the wrong context deletes production resources, so "just run it locally" was not an option we'd accept again. The alternative was a script whose job is to make a laptop deploy behave like a CI deploy: same inputs, same guardrails, no way to ship what isn't in git.
It grew to about 300 lines of bash. Here is what those lines do, in order.
Preflight. The script refuses to run if the working tree has uncommitted changes, if --env prod is given without --yes, if the current branch isn't main for a prod deploy, or if the AWS identity it's about to use isn't the expected deploy role for that account. Each refusal prints one sentence saying what to do. About 40 lines, and the most valuable 40 in the file.
Worktree. It creates a detached git worktree at origin/main (after a git fetch) in a temporary directory and does everything from there. Whatever is in the developer's working copy, stashed, half-edited or experimental, cannot ship. The SHA of that worktree is the release identifier for everything that follows.
Archive to S3. git archive of the worktree, uploaded to a build bucket under the SHA. CodeBuild builds from that archive, not from a GitHub connection, so the build input is the same bytes whether triggered by CI or by the script.
Three builds in parallel. One CodeBuild project per application, started together, polled together. Building three Next.js images on a laptop's Docker would take eight minutes and heat the room; on three CodeBuild instances it takes three. The projects are the same ones CI uses.
Digests. As in part one. The script resolves every tag and prints a table of app → tag → digest.
Diff and confirm. cdk diff against the target environment, printed in full. If the diff contains any resource removal, the script highlights it and requires the operator to type the stack name to continue, not just y. This is the step that would have prevented the incident.
Deploy. cdk deploy with the digests as context and --require-approval never, because approval already happened in the previous step with better information than CDK's own prompt gives.
Verify and notify. Hit the health route until it reports the new SHA, grep the client bundle for the same SHA, and post a message to the team channel with the environment, the SHA, the operator's name and the diff summary. If verification fails, the script says so loudly and exits non-zero; it doesn't roll back automatically, because a failed verification usually means "look at it", not "undo it".
What it looks like to use
./scripts/deploy.sh --env staging
./scripts/deploy.sh --env prod --yes
./scripts/deploy.sh --env prod --dry-run # synth + diff, no build, no deploy
Three flags. Everything else is decided by the script from git and from the account. There is no way to pass a custom image, a custom branch or a custom context value, on purpose.
Would we write it again?
Yes, and we'd write it on day one rather than after the incident and the CI outage. Not because laptop deploys are good, but because the script is the pipeline. When CI came back, its workflow became: check out, assume the OIDC role, run ./scripts/deploy.sh --env prod --yes. One code path, exercised from both places, with the guardrails in the code rather than in a YAML file that only runs on someone else's machine.
The 300 lines are long for a shell script. They replaced a page of runbook that said "be careful", which is shorter and doesn't work.
If your deploy is a cdk deploy that someone runs from wherever they happen to be, we can help make it boring.