Resources 8 min read

Designing Infrastructure for High Availability

Failure domains, correlated failure, N+1 against 2N, degradation modes and the dependency analysis that decides whether redundancy will actually help.

Transformer and switchgear equipment in an electrical substation under a clear blue sky.

Availability is designed before it is operated. The mechanics of distributing traffic and removing failed components are covered in High Availability and Load Balancing; this page is about the decisions made earlier, which determine whether those mechanics have anything useful to do.

The recurring theme is that redundancy is not the same as availability. Duplicating a component protects against that component failing independently, and a great deal of real-world downtime comes from failures that are not independent.

The target is a business decision

How available a system needs to be is a question about consequence, not about technology. What does an hour of unavailability cost, at what time of day, to whom, and does the cost rise linearly or fall off a cliff at some point?

Answering that first keeps the design proportionate. Without it, availability work defaults to whatever the technology makes easy, which produces estates with elaborate redundancy on systems nobody would miss and none on the one that stops invoicing.

Availability is not uniform across an estate

Treating every system to the same standard is expensive and usually wrong in both directions. A tiered approach, where systems are placed in a small number of availability classes with defined characteristics, is easier to operate and easier to explain.

Three or four tiers is normally enough. More than that and the distinctions stop being meaningful to anyone outside the team that wrote them, which defeats the purpose of having a shared vocabulary.

Failure domains are the central concept

A failure domain is the set of things that fail together. A server, a rack, a power feed, a switch, a hypervisor host, a data center, a region, a cloud account, a deployment pipeline.

Good availability design places redundant components in different domains, and the work is mostly in discovering what the domains actually are. Two virtual machines are redundant until you learn they share a hypervisor. Two racks are redundant until you learn they share a power distribution unit. The diagram rarely shows this and the facility documentation sometimes does.

Correlated failure is what defeats redundancy

Independent failure is what redundancy assumes. Correlated failure is what happens: a bad configuration pushed to every node, an expired certificate shared across the fleet, a memory leak that reaches the same threshold on both members of a pair at the same time, a dependency that both replicas call.

The design question is therefore not "are there two of these" but "what would cause both to fail at once, and have we addressed that". Staged rollout, configuration divergence between replicas and independent monitoring paths are the usual answers, and they cost more than duplicating a box.

N+1, 2N, and what you are buying

N+1 provides one spare beyond what is needed, which handles a single failure and leaves the estate exposed while the failed unit is being replaced. 2N provides a complete duplicate, which handles a failure and keeps full capacity during the repair window.

The gap between them is the repair window, and that is what should drive the choice. If replacement takes an hour, N+1 is usually adequate. If it takes three weeks because the part is on order, N+1 means three weeks of running with no margin, and the honest design is 2N or a faster supply chain.

Capacity has to survive the failure

Redundancy without headroom fails on the second event rather than the first. If losing a node means the survivors run at their limit, then the design tolerates one failure and nothing else, including the routine load spike that arrives while the replacement is being built.

The calculation is peak load against surviving capacity, redone whenever the estate grows. It is easy arithmetic and it is skipped routinely, which is why capacity exhaustion during a failover is one of the more common ways a resilient design produces an outage.

State is where the difficulty lives

Stateless components are easy to make redundant: run more of them and route around the failures. Everything hard about availability is about state.

Replication introduces the choice between synchronous and asynchronous, which is really a choice between latency and data loss. Synchronous replication means no loss and a write that waits for the far side; asynchronous means a fast write and a window of data that exists in one place. Neither is right in general, and the decision belongs to the business process rather than to the database administrator.

Cloud platforms encode the same thinking

Provider frameworks arrive at these principles from the same direction. Amazon's reliability guidance notes that in the cloud, there are a number of principles that can help you increase reliability, and lists automatically recovering from failure, testing recovery procedures, scaling horizontally and managing change through automation. See AWS Well-Architected Framework, Reliability Pillar.

Testing recovery procedures is the one that distinguishes a design from an aspiration. A failover path that has not been exercised is an assumption with infrastructure attached.

Degradation beats binary failure

The most useful availability property is often not staying fully operational but failing usefully. A service that keeps taking orders while the recommendations engine is down has degraded; a service that returns an error page because one dependency is unavailable has failed.

Designing for degradation means deciding in advance which features are essential and which can be switched off, and building the switches. It is comparatively cheap, it does not require duplicate infrastructure, and it converts several categories of total outage into a reduced service nobody escalates.

Dependencies decide the real availability

A system is at most as available as the things it cannot work without. A carefully redundant application depending on a single authentication service, one DNS resolver, or a licence server on a virtual machine somebody built in 2019, has the availability of the weakest of those.

Mapping dependencies and finding the shared ones is the highest-value analysis in this whole subject, and it usually produces an uncomfortable list. Identity, DNS and certificate infrastructure appear under almost everything and are redundant in almost no estate.

Where an alternate site belongs

Redundancy inside one facility does not survive losing the facility. Federal control catalogues treat this as its own requirement, with the ALTERNATE PROCESSING SITE control asking for a separate site and for agreements that permit the transfer and resumption of operations within the organization's recovery time objective. See NIST SP 800-53 Revision 5.

Whether a design needs one is a question about the business tolerance established at the start, and the honest answer for many systems is no. What is not honest is describing single-site redundancy as though it covered site loss.

Human error is a failure mode with no spare

A meaningful share of outages come from a change rather than a fault, and redundancy does not protect against a change applied correctly to every replica.

The controls are different in kind: staged rollout, a tested rollback, a second pair of eyes on changes that affect the whole fleet, and automation that makes the safe path the easy one. An availability design that considers only hardware failure has addressed a minority of what actually happens.

Complexity is a cost on the availability side of the ledger

Every redundancy mechanism adds components, states and failure modes of its own. Clustering software, replication, automatic failover and the health logic that drives them can all fail, and some of them fail in ways that are worse than the fault they exist to handle.

Split-brain is the classic example: two nodes each concluding the other is dead, both taking writes, and a reconciliation problem that is harder than the original outage. The design question is whether the mechanism is more reliable than the thing it protects, and the answer is not automatic.

Private, cloud and hybrid make different parts easy

Cloud platforms provide zone-separated capacity, managed replication and automated recovery as configuration, which makes several patterns cheap that used to require a second facility. Private infrastructure gives control over the physical domains, the hardware and the data location, at the cost of building the mechanisms yourself.

Neither is inherently more available. What differs is where the effort goes and which failure modes come by default. Private Cloud and Data Center Services covers the private side of that trade.

Write down what the design assumes

Every availability design rests on assumptions: that a failure will be detected in some time, that a replacement is available, that capacity suffices, that a dependency is more available than the system, that a person will be reachable.

Recording them makes them testable and makes them reviewable when the estate changes. Most availability failures, examined afterwards, turn out to be an assumption that stopped being true without anyone noticing that it had been an assumption.

Design, then operate, then recover

Availability design reduces how often recovery is needed and never removes the need. The two belong together: redundancy for the common failures, a recovery plan for the ones that defeat it, covered in Disaster Recovery and Business Continuity.

LABUSA's approach begins by agreeing the boundary and the objectives before the technical work starts, so that what is being protected, to what standard, and by whom, is settled rather than assumed. The wider service this design work belongs to sets out the rest.

About LABUSA

LAB Information Technology Incorporated (LABUSA) is a trusted provider of managed IT solutions, empowering organizations with secure, efficient, and scalable technologies. With expertise spanning cybersecurity, cloud services, enterprise software, and data management, LABUSA helps clients modernize operations, strengthen compliance, and optimize performance. Our customer-focused approach ensures tailored solutions that align with organizational goals while maintaining the highest standards of reliability and security. Headquartered in Houston, Texas, LABUSA serves government agencies, corporations, and nonprofits across the United States and internationally.