Disaster Recovery & Resilience

Disaster Recovery & Resilience

1. The Strategy: RTO and RPO

Before choosing a technical architecture, you must define your business objectives. These two metrics dictate your entire DR strategy.

  • RPO (Recovery Point Objective): How much data can you afford to lose? It measures the time between your last backup and the disaster. If your RPO is 4 hours, you need to back up data at least every 4 hours.

  • RTO (Recovery Time Objective): How long can you afford to be offline? It measures the time it takes to get your system back up and running after a disaster occurs.

2. Disaster Recovery Strategies (Spectrum of Cost & Speed)

As you move from Backup & Restore to Multi-Site, RTO/RPO decrease (good), but costs increase significantly (bad).

🏪 Backup & Restore (Highest RTO/RPO, Lowest Cost)

  • How it works: Data is regularly backed up to cheap storage (e.g., AWS S3). No infrastructure is kept running in the DR region.

  • In a disaster: You must provision all servers, deploy code, and restore data from backups.

  • Metrics: RTO = Hours to Days; RPO = Hours.

🕯️ Pilot Light (Medium-High RTO/RPO, Low Cost)

  • How it works: Core data is continuously replicated to the DR region (e.g., live database replication), but application servers are turned off or kept as a template (AMIs/images).

  • In a disaster: You “light the fire” by quickly provisioning and scaling the application servers to sit in front of the already-replicated data.

  • Metrics: RTO = Minutes to Hours; RPO = Minutes.

🚶 Warm Standby (Low RTO/RPO, Medium-High Cost)

  • How it works: A scaled-down, fully functional version of your environment is always running in the DR region. It can handle minimal traffic but cannot handle full production load.

  • In a disaster: You scale up the existing instances (via Auto Scaling) to handle the full load and flip the traffic switch.

  • Metrics: RTO = Minutes; RPO = Seconds to Minutes.

🌐 Multi-Site / Active-Active (Near-Zero RTO/RPO, Highest Cost)

  • How it works: Full production environments run concurrently in two or more regions. Traffic is split between them (e.g., 50/50 via Route 53 or Traffic Manager).

  • In a disaster: If Region A dies, 100% of the traffic is instantly routed to Region B, which is already live and fully scaled.

  • Metrics: RTO = Real-time / Seconds; RPO = Near-Zero.

3. Data Sync: Cross-Region Replication (CRR)

To achieve low RPO in Pilot Light, Warm Standby, or Multi-Site setups, data must leave its primary region.

  • Asynchronous Replication: Data is written to the primary region first, then copied to the DR region shortly after. Best for Pilot Light/Warm Standby across long geographic distances (low impact on primary application performance).

  • Synchronous Replication: Data is written to both regions simultaneously. Hard to achieve across regions due to the speed of light (latency), but required for true zero-RPO Active-Active setups.

4. Execution: Failover Automation

Having a great DR environment is useless if it takes a human 3 hours to realize there is an outage and manually change DNS records. Failover automation relies on:

  1. Health Checks: Automated monitors (like Route 53 Health Checks) constantly ping your primary region.

  2. Circuit Breakers / Automated DNS Routing: If the primary region fails $X$ consecutive health checks, the DNS routing policy automatically shifts traffic to the DR region.

  3. Infrastructure as Code (IaC): Using tools like Terraform or CloudFormation ensures that if you need to spin up resources (like in a Pilot Light scenario), it is done via a single, tested script rather than human guesswork during a crisis.