I've been paged at 3 a.m. enough times to have a strong opinion about high availability: most systems labeled "highly available" are nothing of the sort. They have a replica, a failover mechanism, and a diagram — and none of it has ever been exercised under real failure. HA that has never failed over is just a hypothesis.
After a decade running databases at Fortune 500 scale, here's what I've learned actually separates resilient systems from fragile ones.
Availability is a property of your worst day
It's easy to be available when everything works. The number that matters is how the system behaves when a primary dies mid-transaction, a network partition splits your cluster, or a disk fills at the worst possible moment. If you haven't watched your architecture handle those, you don't know your availability — you know your uptime on good days.
Real HA design starts from the failure, not the happy path.
Failover you haven't tested will fail
The single most common cause of a long outage isn't the initial failure — it's the failover that didn't work. Promotion hangs because a replica was lagging. DNS or connection strings don't repoint. The application pool doesn't reconnect cleanly. Every one of these is discoverable before an incident, and almost no one looks.
The discipline that fixes it is unglamorous: schedule failovers. Force them in production windows. Kill the primary on purpose and time the recovery. A failover path exercised monthly is a tool; one that's never been run is a liability with good branding.
Replication lag is a data-loss budget
Every asynchronous replication setup is quietly making a promise about how much data you're willing to lose. If your replica is 8 seconds behind and the primary dies, you just decided 8 seconds of writes were expendable — whether you meant to or not.
Resilient design makes that budget explicit. Synchronous commit where the data genuinely can't be lost, asynchronous where throughput matters more, and monitoring that treats replication lag as a first-class alert, not a dashboard curiosity.
The database is not the whole system
A promoted replica doesn't help if the application can't find it, the connection pool is still holding dead sessions, or the retry logic hammers a recovering node into the ground. High availability is an end-to-end property. I've seen flawless database failovers turn into full outages because the layer above wasn't built to notice.
Test the whole chain: database, connection layer, application retry behavior, and the humans who have to make a call at 3 a.m.
Resilience is a practice, not an architecture
You don't buy high availability by drawing the right topology. You earn it by rehearsing failure until recovery is boring. The teams with the best uptime aren't the ones with the fanciest architectures — they're the ones who've broken their own systems on purpose the most times.
— Jamaurice Holt, founder of EVE NeuroSystems LLC and Senior Database Administrator