Runbook Example for a Database Connection Exhaustion Incident

Database connection exhaustion looks like an outage, but the database is fine. The app servers simply can't get a connection to it, requests pile up, timeouts cascade, and users start seeing 503s while the database process itself sits there, healthy and idle-adjacent, wondering why nobody's calling. That distinction matters because it changes what an on-call engineer should reach for at 2 AM. A playbook tells your org how to handle any SEV-1; this piece is a runbook, built to tell one specific engineer exactly what to run for this specific failure, in order, without improvisation.

The test for whether a runbook is actually a runbook: can someone who joined six months ago follow it through a live incident without pinging a Slack channel for help? A failing answer here means what you have is closer to documentation than a working runbook. Connection exhaustion has a handful of common triggers worth naming up front, because triage depends on knowing which one you're looking at: a traffic spike the pool wasn't sized for, a deployment that causes every instance to reconnect at once, a slow query or long-running transaction holding a connection open indefinitely, or a pool ceiling that was set for a fleet size the org has since outgrown. The expensive version of this incident is the one where nobody alerted on pool utilization, so the first signal anyone gets is user-facing errors. By then the problem has already matured past the point where a five-minute fix would have worked.

What the runbook must contain before a single diagnostic step

Before anyone runs a query, the runbook needs a metadata block at the top, and it needs to be copy-ready rather than aspirational. Incident name ("High Database Connection Pool Exhaustion"), severity (SEV-1, or SEV-2 for partial degradation), the owning team, and a linked alert rule, something like db-connection-pool-utilization > 90% sustained for 5 minutes. The escalation path should name roles, not people: primary on-call, then database team lead, then engineering manager.

One field is non-negotiable: last validated date. An undated runbook can't be trusted at face value. If nobody can tell you whether these steps were checked against the current system last month or three years ago, you're trusting a document that might be describing infrastructure that no longer exists.

Preconditions come next, and they should be confirmable in under a minute: read access to the database host or its console, access to connection pool metrics in whatever observability tool the team runs, and a vault reference for database superuser credentials (never inline secrets in the runbook itself). The responder also needs to know, going in, whether a deployment or migration ran recently enough to be a suspect.

Three metrics should already be open in a tab before triage starts: dbpoolactiveconnections, dbpoolwaitingrequests, and application 5xx rate correlated against pool utilization. And every step from here forward should be idempotent where possible. Running a diagnostic command twice should never make the incident worse; that reliability is what lets a tired engineer move fast without second-guessing every keystroke.

Detection and initial triage: the first five minutes

Step one is confirming the alert is real. Check that dbpoolactive_connections is actually elevated in the dashboard, not a stale metric or a misfired threshold, and confirm the user-facing symptom independently: are 503s or timeouts showing up in application logs right now?

Step two establishes blast radius. Which services are affected, checkout, account pages, or everything? Is it one instance or the whole fleet? Is it read traffic, write traffic, or both? These answers shape which recovery action makes sense later, so they're worth nailing down even under pressure.

Step three checks for a precipitating deployment or migration. Correlate the alert timestamp against recent deploys. A reconnect storm caused by a rolling deployment is a distinct sub-case, and it changes the recovery path, because the fix there is about deploy pacing rather than database configuration.

Step four is the first real diagnostic query against the database itself:

SELECT count(*) FROM pg_stat_activity;
SELECT state, count(*) FROM pg_stat_activity GROUP BY state;

A large count of connections sitting in idle in transaction points to a fundamentally different root cause than a simple pool ceiling breach; it means something is holding connections open without releasing them, a query or transaction management problem rather than a capacity problem.

Step five identifies the worst offenders:

SELECT pid, usename, application_name, state, query_start,
       now() - query_start AS duration, left(query, 80)
FROM pg_stat_activity
ORDER BY duration DESC NULLS LAST
LIMIT 20;

Anything running past 30 seconds is a candidate connection hog. By the end of these five steps, the responder should know which of three situations they're in: a pool ceiling too low for current traffic, leaked or idle-in-transaction connections, or one slow query monopolizing the pool. Each has its own fix, and guessing wrong here wastes the minutes you don't have.

Recovery actions, ordered by risk and reversibility

The governing principle: start with the cheapest, most reversible action, and only escalate if it doesn't move the needle within a defined window. Escalating early because it feels more decisive tends to turn a contained incident into a self-inflicted one.

Action one, and usually sufficient on its own, is terminating idle-in-transaction connections:

SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE state = 'idle in transaction'
  AND now() - query_start > interval '5 minutes';

Watch dbpoolactive_connections for a drop within 60 seconds. This step is idempotent, so if the number doesn't move, run it again and move on; it costs nothing to retry.

Action two is lowering the application-side pool ceiling temporarily, if the exhaustion stems from a pool max set higher than the database can actually support. This is a config change that takes effect on pool restart, not on the database side, and it buys headroom without touching production database settings.

Action three carries more weight: raising max_connections on the database.

ALTER SYSTEM SET max_connections = 500;

This one needs a full database restart on most configurations; a reload alone won't apply it. Confirm this for your specific deployment before you commit, because assuming a reload is enough and finding out otherwise mid-incident is a bad way to spend fifteen minutes. Success looks like pool utilization dropping below roughly 70% and the wait queue emptying, typically within 15 to 30 minutes of the change taking effect.

Action four, where a replica exists, is shifting read traffic off the primary. If read queries are driving the pressure, this sidesteps the need to touch primary database configuration entirely, assuming your connection string or load balancer supports the split.

Action five, a connection pooler like PgBouncer sitting in front of the database, works well as a long-term fix but is a poor choice as a 2 AM action. Configuring a pooler correctly under incident pressure is how you turn one outage into two. Write it down as a follow-up, not a live step.

After every action, recheck the same two metrics: active connections and waiting requests. If they aren't trending the right direction inside the expected window, move to the next action. Waiting indefinitely on a step that isn't working is its own kind of risk.

Verifying recovery before closing the incident

Here's where incidents get closed too early. Connection count drops, the pressure lifts, and the responder, understandably, wants to go back to bed. Resist that. A momentary dip in pool utilization is not the same as recovery.

The checklist that actually confirms recovery: pool utilization below the safe threshold and holding steady for at least five minutes, dbpoolwaiting_requests at or near zero, 5xx rate back to baseline, and a real synthetic check or user-facing endpoint test passing, not just clean numbers at the database layer. Confirm no new alerts have fired in the same window.

If max_connections got raised as a stopgap, write the new value into the incident record explicitly and flag it as temporary. Treating that stopgap value as the permanent ceiling is how the next traffic spike catches you again. Worth a quick check, too: did any terminated idle-in-transaction connections leave a write half-finished? Rare, but worth ruling out before you call it done.

Closing the incident includes telling people it's closed. A one-line message confirming resolution and the window of user impact matters as much as the technical remediation itself, because the next person debugging a related issue needs that timestamp.

Post-incident actions that prevent the same page next month

The postmortem for this specific failure mode has its own questions, and generic "what went wrong" prompts won't surface them. What was the traffic or workload pattern that pushed connections past the limit? Was there any alert on pool utilization before users started seeing errors, and if not, why not? Was the pool ceiling sized for peak load or average load, and which should it have been sized for? Did a deploy or migration land right before the exhaustion started?

Instrumentation gaps usually show up here. Add a warning-level alert on pool utilization, somewhere around 75%, so there's a signal before the critical threshold fires and before anyone gets paged. Instrument idle-in-transaction count as its own metric, separate from raw active connections; a slow rise there is an early warning that something in query or transaction handling has drifted. Track waiting requests as a leading indicator too, not an afterthought.

Deployment practice deserves a hard look if a reconnect storm was the trigger. Staggering rolling deploys so instances don't all reconnect simultaneously is a small change with outsized payoff. If a migration ran alongside the deploy, consider decoupling migrations from application releases entirely. And for any feature expected to raise database load meaningfully, a load test that specifically checks pool sizing before rollout is cheap insurance against repeating this exact incident.

Update the runbook's last-validated date every time it runs. Note anything the responder had to improvise, since improvisation during an incident usually signals a missing step in the document rather than a clever save by the responder. If a pooler or a replica got added as a remediation, the recovery actions section needs to reflect that new topology, or the next person to open the runbook will be troubleshooting against a system that no longer matches what's written down.

Where the runbook lives and how it stays current

The runbook has to be reachable through a path that doesn't depend on the very alerting system that paged you. If the incident management platform itself is degraded, and sometimes it will be, the runbook still needs to be findable. That usually means a Git repository, versioned and diffable like the rest of the codebase, or a static docs site that lives independently of the observability stack, with the URL linked directly from the alert rule itself so the page includes the runbook rather than sending someone on a scavenger hunt through a wiki.

Runbook decay is a real cost, not a hypothetical one. Documents that don't get reviewed after incidents quietly accumulate references to services that were decommissioned, thresholds that changed six reorganizations ago, and recovery steps built for a topology nobody runs anymore. The wave of alerting-tool migrations many teams went through in 2025 surfaced exactly this pattern: teams migrating platforms discovered runbooks last touched years earlier, still confidently pointing at infrastructure that no longer existed. The fix isn't complicated, just consistent: every time a runbook actually gets executed in a real incident, that execution is the review. Update the date, fix what didn't match reality, and move on.

There's a shift underway in how much of this even needs a human to catch. A manual runbook stays essential, nothing here argues otherwise, and platforms that watch post-deploy telemetry can complement it by catching the early signs of connection exhaustion, rising pool utilization right after a release, a new query pattern quietly building up idle-in-transaction connections, before the threshold alert ever fires. OnePatch's approach treats every deployment as a verification event in itself: it correlates deploy timestamps against connection pool metrics and flags anomalies while they're still small, which means the runbook above gets run less often. That reduction comes from catching the triggering conditions upstream, not from anyone retiring the runbook itself.

That's the right division of labor. Automation doesn't replace the runbook; it just moves the runbook from first response to last resort, which is exactly where a well-built one belongs.