The situation
What needs investigation
A production application may be healthy during normal operation but depend on one instance, one availability zone, one database path, or one untested failover action. The review defines which failures matter to the business and what must continue operating.
Signals we look for
Traffic
- ALB and target health
- DNS behaviour
- multi-AZ routing
Application
- instance or task redundancy
- session handling
- deployment failure modes
Data
- RDS availability
- replica role
- backup and failover behaviour
Recovery
- failure detection
- manual steps
- capacity during failover
How we investigate
Each step is validated against the workload before the next change.
- 01
Define service needs
Identify critical user journeys and tolerated interruption.
- 02
Map failure domains
Trace availability-zone, instance, database, network, and dependency boundaries.
- 03
Review redundancy
Check load balancing, health checks, replicas, and capacity plans.
- 04
Test failure behaviour
Exercise target loss, instance loss, and dependency degradation safely.
- 05
Plan recovery
Document detection, failover, and manual intervention steps.
- 06
Monitor the design
Add signals that show whether redundancy is actually available.
Architecture / decision path
Example engineering decision
Illustrative failure-domain review — not customer data
Application targetstwo targets in one Availability Zone
DatabaseMulti-AZ enabled
Session statestored in process memory
Engineering interpretation
Database redundancy does not protect a traffic path that is confined to one zone, and in-memory sessions can prevent clean recovery when a target is replaced.
Possible next investigation
- Spread targets across relevant failure domains.
- Move session state to an appropriate shared store.
- Test target termination and connection draining.
- Validate capacity when one zone or target set is unavailable.
Trade-offs we consider
Availability
More redundancy improves fault tolerance but adds components to operate.
Cost
Standby and multi-AZ capacity should match the service objective.
Data consistency
Failover choices affect write behaviour and recovery sequence.
Complexity
A design is only useful if the team can test and operate it.
What the engagement can produce
Typical outputs for this type of work
Scope depends on the environment and the problem being investigated.
- failure-domain map
- availability architecture findings
- health-check and redundancy plan
- failover validation plan
- recovery runbook
- monitoring recommendations