Availability, Regions, and the Arithmetic of Nines

Availability questions get answered with adjectives — “highly available”, “resilient”. They should be answered with numbers, because the number decides the architecture.

What the nines buy

# nines.py
MINUTES_PER_YEAR = 365.25 * 24 * 60

print(f"{'availability':>14} {'downtime/year':>18} {'per month':>14} {'per week':>12}")
for nines in (0.99, 0.995, 0.999, 0.9999, 0.99999):
    yr = MINUTES_PER_YEAR * (1 - nines)
    print(f"{nines*100:>13.3f}% {yr/60:>14,.1f} hrs {yr/12:>11,.1f} min {yr/52:>9,.1f} min")

$ python nines.py
  availability      downtime/year      per month     per week
       99.000%           87.7 hrs       438.3 min     101.2 min
       99.500%           43.8 hrs       219.2 min      50.6 min
       99.900%            8.8 hrs        43.8 min      10.1 min
       99.990%            0.9 hrs         4.4 min       1.0 min
       99.999%            0.1 hrs         0.4 min       0.1 min

Read the “per month” column. Four nines is 4.4 minutes a month — less than a single rolling deploy on many systems. Five nines is 26 seconds a month, which means the system essentially cannot have a planned maintenance window and every change must be zero-downtime.

“Before designing for availability I’d ask what the target actually is and what the business impact of an hour down is. Three nines is nine hours a year and is genuinely fine for a lot of systems. Four nines roughly doubles the infrastructure and adds on-call load. Five nines is a different engineering culture, not a different architecture — and it is rarely what is meant.”

Components multiply

# chain.py
def serial(*components):
    result = 1.0
    for c in components: result *= c
    return result

def parallel(component, n):
    return 1 - (1 - component) ** n

lb, app, db, cache = 0.9999, 0.999, 0.9995, 0.999

chain = serial(lb, app, db)
print(f"LB 99.99% x app 99.9% x db 99.95%  = {chain*100:.4f}%  "
      f"({(1-chain)*365.25*24:.1f} hrs/yr)")

app3 = parallel(0.999, 3)
better = serial(lb, app3, db)
print(f"three app instances in parallel     = {app3*100:.7f}%")
print(f"same chain with redundant app       = {better*100:.4f}%  "
      f"({(1-better)*365.25*24:.1f} hrs/yr)")

print(f"\nadding a hard dependency at 99.9%   = {serial(better, 0.999)*100:.4f}%")

$ python chain.py
LB 99.99% x app 99.9% x db 99.95%  = 99.8401%  (14.0 hrs/yr)
three app instances in parallel     = 99.9999999%
same chain with redundant app       = 99.9400%  (5.3 hrs/yr)

adding a hard dependency at 99.9%   = 99.8401%  (14.0 hrs/yr)

Three things fall out of this, and all three are usable answers:

• Components in series multiply, so the chain is always worse than its worst part. A 99.9% app behind a 99.99% load balancer is not 99.99% available.
• Redundancy at one layer has limits. Three app instances take that layer to effectively perfect, and the chain only improves from 14 hours to 5.3 — because the database is now the binding constraint.
• Every added dependency costs availability. The last line adds one more 99.9% service and gives back the entire gain from the redundancy. A microservice architecture with twelve synchronous hops has an availability ceiling set by that arithmetic.

That last point is the one worth volunteering: “a synchronous call to another service is a subtraction from your availability, which is the strongest argument for making a dependency asynchronous or optional.”

RTO and RPO drive different spending

                    RTO                          RPO
question            "how long can we be down?"   "how much data can we lose?"
measured in         minutes to days              seconds to hours
you buy             standby capacity             replication frequency
zero costs          a hot second region          synchronous replication
                                                 (and write latency)

They are independent. A reporting warehouse can tolerate an RTO of a day and an RPO of an hour. A payments ledger might accept an RTO of ten minutes and an RPO of zero.

# dr.py
PRIMARY_MONTHLY = 12_000

STRATEGIES = {
    #                        RTO          RPO         extra cost multiple
    "backup & restore":     ("6-24 hrs",  "1-24 hrs",  0.02),
    "pilot light":          ("10-60 min", "min-hrs",   0.15),
    "warm standby":         ("5-15 min",  "seconds",   0.50),
    "hot / active-active":  ("< 1 min",   "~0",        1.10),
}

print(f"{'strategy':<20} {'RTO':>12} {'RPO':>10} {'extra/mo':>11} {'total/mo':>11}")
for name, (rto, rpo, mult) in STRATEGIES.items():
    extra = PRIMARY_MONTHLY * mult
    print(f"{name:<20} {rto:>12} {rpo:>10} {extra:>11,.0f} {PRIMARY_MONTHLY+extra:>11,.0f}")

$ python dr.py
strategy                      RTO        RPO    extra/mo    total/mo
backup & restore         6-24 hrs   1-24 hrs         240      12,240
pilot light             10-60 min    min-hrs       1,800      13,800
warm standby             5-15 min    seconds       6,000      18,000
hot / active-active        < 1 min         ~0      13,200      25,200

Active-active costs more than double, not double — the 1.10 multiplier reflects that you run full capacity in both regions and pay cross-region replication traffic on every write.

The step from backup-and-restore to pilot light is the best value on this table: 7.5x the DR cost, but still only 15% on top of the primary, and it moves RTO from a day to under an hour.

“I’d want the RTO and RPO numbers from the business before choosing. The gap between backup-and-restore and active-active is a factor of fifty in DR spend, and the honest answer is that most systems asking for active-active actually need warm standby.”

Multi-AZ is not multi-region

aws rds describe-db-instances --db-instance-identifier prod \
  --query 'DBInstances[0].{az:AvailabilityZone,multiAZ:MultiAZ,secondary:SecondaryAvailabilityZone}'

{
    "az": "us-east-1a",
    "multiAZ": true,
    "secondary": "us-east-1b"
}

                    MULTI-AZ                  MULTI-REGION
protects against    data-centre failure,      regional service outage,
                    host failure, AZ network  regional control plane failure,
                    partition                 natural disaster, bad regional config
failover            automatic, 30-120 s       manual or DNS-driven, minutes
data                synchronous, RPO ~0       asynchronous, RPO seconds-minutes
extra cost          ~2x the database          ~2x everything, plus transfer
latency impact      1-2 ms on writes          10-100 ms if synchronous
complexity          a checkbox                a project

The row that matters: multi-AZ is a configuration flag; multi-region is an architecture. Multi-AZ protects against the failure mode that actually happens most often — a host or a facility — and costs one checkbox. Multi-region protects against the rarer, more dramatic one.

The honest framing for an interview:

“Multi-AZ first, always — it is cheap and automatic. Multi-region I’d only propose if there is a stated regional-outage requirement or a latency requirement for users on another continent, because it changes the data model. You have to decide what happens when the regions disagree, and that decision is the hard part, not the infrastructure.”

What redundancy does not fix

FAILURE MODE                        DOES REDUNDANCY HELP?
one host dies                       yes — this is what it is for
one AZ dies                         yes, if capacity exists in the others
bad deploy                          NO — it rolls out everywhere
bad configuration change            NO — it applies everywhere
expired certificate                 NO — same cert on every replica
data corruption                     NO — it replicates to the replicas
a delete that should not have run   NO — replicated within seconds
dependency outage                   NO — everything calls it
running out of a quota              NO — the quota is account-wide
DNS misconfiguration                NO — it is the entry point

Most of that column reads NO, and that is the point of the table. The largest real outages are overwhelmingly caused by changes, not by hardware. Redundancy addresses hardware.

What addresses the rest:

bad deploy               canary + automatic rollback on error rate
bad config               treat config as code; stage it like a deploy
expired cert             automated renewal, and an alarm at 30 days
data corruption          point-in-time restore, tested — backups are not
                         backups until a restore has been demonstrated
accidental delete        soft delete, versioning, deletion protection
dependency outage        circuit breakers, timeouts, cached fallbacks
quota exhaustion         monitor quota usage as a metric, not a surprise

“Redundancy protects against hardware and it is table stakes. The outages I’d actually plan for are changes — a deploy, a config push, a certificate. Those need canaries and rollback, and a restore that has actually been tested rather than assumed.”

Health checks decide whether failover works

aws elbv2 describe-target-health --target-group-arn arn:aws:elasticloadbalancing:...:tg/app \
  --query 'TargetHealthDescriptions[].{id:Target.Id,state:TargetHealth.State,why:TargetHealth.Reason}' \
  --output table

------------------------------------------------------------------
|                     DescribeTargetHealth                       |
+---------------+-------------+----------------------------------+
|      id       |    state    |               why                |
+---------------+-------------+----------------------------------+
|  i-0aaa       |  healthy    |  None                            |
|  i-0bbb       |  unhealthy  |  Target.ResponseCodeMismatch     |
|  i-0ccc       |  draining   |  Target.DeregistrationInProgress |
+---------------+-------------+----------------------------------+

Two health check designs, and the difference decides whether failover helps or hurts:

SHALLOW  /healthz returns 200 unconditionally
         → the process is alive. Says nothing about whether it works.
         → an instance with a dead database connection stays "healthy"
           and keeps receiving traffic that it fails.

DEEP     /healthz checks the database, the cache, and the queue
         → accurate, and DANGEROUS: when the shared database has a blip,
           every instance reports unhealthy at once, the load balancer
           removes all of them, and a recoverable degradation becomes a
           total outage.

The resolution is worth stating precisely, because it is a real design decision:

“I’d use a shallow liveness check for the load balancer — it should only remove an instance that is genuinely broken in a way its peers are not — and a separate deep readiness check for monitoring and for deciding whether to accept traffic at startup. Wiring a shared dependency into the load balancer’s health check turns a database slowdown into a full outage, and that is a real incident pattern rather than a hypothetical.”

Recognising it

QUESTION                                        ANSWER
"how do you make this highly available?"        ask for the target number first
"multi-region?"                                 only with a stated requirement; price it
"what's your RTO/RPO?"                          they are independent; both drive spend
"how do you handle a bad deploy?"               canary + auto-rollback, not redundancy
"how do you know backups work?"                 restore drills; untested backups are not backups
"what breaks when the cache dies?"              thundering herd against the database
"why did redundancy not help?"                  it was a change, not a hardware failure
"how do you test failover?"                     game days; a failover path never exercised
                                                does not work
"single points of failure here?"                NAT gateway, one AZ, the deploy pipeline, DNS

Practice

99.9% → 43.8 min/month      99.99% → 4.4 min      99.999% → 26 sec

14.0 hrs/yr → 5.3 hrs/yr, and one more 99.9% dependency gives it all back.

backup/restore +$240      pilot light +$1,800      active-active +$13,200

A database blip marks every instance unhealthy at once — total outage from
a recoverable degradation.

Next: scaling and queues — autoscaling that reacts too late, backpressure, and Little’s law.