# Health checks that lie: six ways your service says "ready" when it isn't
A health check is a contract between your application and the thing that routes traffic to it. The router asks "can you take a request?" and the application answers. When the answer is wrong, the router does exactly what it was told: it sends users to an instance that will fail them, or it stops sending users to instances that were fine. Both are outages, and both were caused by a health check that was technically working.
We've been bitten by six distinct varieties of this across App Runner, ECS, Lambda behind an API gateway, and a Kubernetes cluster. This is the catalogue, what each one looks like from the outside, and the health route we ended up with.
## Two questions, not one
The root of most of these is that "healthy" collapses two different questions into one endpoint.
**Liveness: is the process alive and should it be left alone?** The answer is "no" only when the process is wedged: deadlocked, out of memory, spinning. The correct response to a liveness failure is to kill and replace the instance.
**Readiness: can this instance take traffic right now?** The answer is "no" while it's still starting, warming caches, waiting for a connection pool, or, temporarily, when it's overloaded. The correct response is to stop routing to it and check again shortly. Not to kill it.
Kubernetes makes these two probes explicit. App Runner and ECS give you one health check and use it for both purposes, which is fine as long as you know that a failing check gets the instance *replaced*, and design the check so that it only fails when replacement is the right answer.
## The six lies
**1. TCP is open, so we're ready.** The default on App Runner and on ECS target groups is a TCP check: if the port accepts a connection, the instance is healthy. Node opens its port in the first second of start-up, before the framework has loaded routes, before the database pool exists, before caches are warm. The instance enters rotation and serves the first thirty seconds of traffic slowly or with errors. We [wrote about this on App Runner](/en/blog/aws-app-runner-review-six-months): the fix is an HTTP check on a route that returns 503 until start-up is actually complete.
**2. A static 200.** The opposite mistake. `app.get('/health', (req, res) => res.send('ok'))`. It never fails. An instance whose database pool is exhausted, whose event loop is blocked for seconds at a time, whose disk is full, reports healthy forever, and the router keeps sending it traffic while the instance next to it, which is fine, gets its share too. This is the health check that most codebases have, because it was the first one someone wrote.
**3. The deep check that fails on someone else's outage.** The over-correction from 2: the health route pings the database, the cache, the queue and a third-party API, and returns 503 if any of them fail. Then the third-party API has a bad hour, every instance reports unhealthy, the platform replaces all of them, the replacements report unhealthy too, and you've turned a degraded feature into a total outage of a service that could have kept serving every request that didn't need that API. Readiness must not depend on things the instance can't fix by restarting.
**4. The check hits a different code path than traffic.** The health route is registered on a separate internal port, or before the middleware stack, or on a path excluded from the router that real requests go through. It passes while the real path is broken: a bad middleware deploy, a router that fails to load, a TLS misconfiguration on the public port. The check must go through as much of the real path as possible without doing real work.
**5. The cached result.** To make the check cheap, someone caches its result for sixty seconds. An instance that went bad at second one reports healthy for another fifty-nine, and a cluster-wide problem is invisible for a minute during which every instance is lying in unison. Cache the *expensive* sub-checks if you must, with a short TTL, but never the aggregate.
**6. The check that gets blocked.** The health route has no `User-Agent`, comes from an internal IP range, and doesn't carry the auth header, so the WAF's [`NoUserAgent_HEADER` rule](/en/blog/waf-for-a-nextjs-app-managed-rules-that-block-legit-traffic) or the rate limiter or the auth middleware rejects it, and the platform sees a 403 as unhealthy. Every instance is replaced in a loop until someone notices the WAF metrics. Health checks need an explicit bypass in whatever sits in front of the app, scoped to the platform's source and the health path only.
## The health route we ended up with
One route, `/api/health`, with a query parameter that selects the depth, because the platform's check and a human's check want different things.
```ts
// app/api/health/route.ts
const startedAt = Date.now();
let warm = false; // set true by the warm-up task after caches load
export async function GET(req: Request) {
const deep = new URL(req.url).searchParams.get('deep') === '1';
// readiness: only things a restart would fix
if (!warm) return json({ status: 'starting', uptimeMs: Date.now() - startedAt }, 503);
if (eventLoopLagMs() > 1000) return json({ status: 'wedged' }, 503);
const checks: Record = { app: 'ok', release: process.env.RELEASE_SHA ?? 'dev' };
if (deep) {
// informational only: never turns the response into a 503
checks.db = await timed(() => db.query('select 1'), 500);
checks.cache = await timed(() => cache.ping(), 200);
checks.payments = await timed(() => payments.ping(), 800);
}
return json({ status: 'ok', ...checks }, 200);
}
```
The properties that matter:
- **It returns 503 for exactly two reasons, both fixed by a restart:** the instance hasn't finished warming up, or the event loop is wedged. That's readiness and liveness in one route, with the failure conditions chosen so that "replace this instance" is always the right reaction to a 503.
- **Dependencies are reported, never enforced.** With `?deep=1`, the route checks the database, the cache and the payment provider, each with a timeout, and puts the result in the body. A human or a dashboard reads it. The platform doesn't call the deep variant, so a vendor outage can't take the fleet down. If the database is unreachable, the instance still says 200, because restarting it won't fix the database and the requests that don't need it (static pages, cached reads) still work.
- **It runs through the real path.** Same port, same middleware stack, same router as user traffic. The middleware has an explicit early return for this path that skips auth and the rate limiter but nothing else, and the WAF has a scope-down that exempts it from the User-Agent rule for the platform's health-check source.
- **Nothing is cached.** The route is cheap: two in-memory reads for the shallow version. The deep version does real I/O and is called by humans, rarely.
- **It reports the release.** The body carries the git SHA, which is what the [deploy script verifies](/en/blog/same-tag-deploy-and-the-deploy-script-without-ci) after a rollout and what you paste into the incident channel.
## Platform by platform
| Platform | What it supports | What to configure |
|---|---|---|
| App Runner | One check, TCP or HTTP, failure = replace | HTTP on `/api/health`, interval 10 s, unhealthy threshold 3, timeout 5 s |
| ECS on Fargate behind an ALB | Container health check (task definition) + target group health check | Both HTTP on the same route; target group is what gates traffic, container check is what triggers replacement |
| Lambda behind API Gateway | No health check; each invocation is its own instance | Provisioned concurrency for the warm-up problem; a synthetic canary that calls `/api/health?deep=1` for the visibility problem |
| Kubernetes | Separate liveness and readiness probes, plus a startup probe | Readiness on `/api/health`, liveness on the same with a longer period, startup probe with a generous failure threshold so slow boots aren't killed |
On every one of them, the number that matters most is the *unhealthy threshold times the interval*: that's how long a broken instance keeps serving before it's pulled. Ten seconds times three is thirty seconds of bad requests. We'd rather have 5 × 2, and pay for the extra check calls, which are free.
## The short version
A health check is a promise. Make it fail only when a restart is the cure, report everything else in the body for humans, run it through the real request path, exempt it explicitly from whatever guards that path, and never cache the answer. Then set the interval and threshold so a lie, when one happens anyway, lasts ten seconds and not a minute.
If your health route is a `res.send('ok')`, [we'll help you replace it](/contact). It's an afternoon, and it's usually the afternoon that ends the "random 502s after deploy" ticket.