Presentation: Week-Long Outage: Lifelong Lessons
📰 InfoQ AI/ML
Learn from a 6-day outage to improve system resilience and psychological safety in your team
Action Steps
- Conduct a Failure Mode and Effects Analysis (FMEA) to identify potential system failures
- Implement shadow traffic to test system resilience
- Exercise rollback mechanisms to ensure smooth recovery
- Widen your circle of communication early to prevent information silos
- Appoint a defender to advocate for psychological safety
Who Needs to Know This
Developers, DevOps, and team leaders can benefit from understanding the technical and human elements that contribute to system resilience and team psychological safety
Key Insight
💡 Technical and human elements are equally important for building system resilience and psychological safety
Share This
🚨 6-day outage lessons: FMEAs, shadow traffic, rollback mechanisms & psychological safety 🚨
Key Takeaways
Learn from a 6-day outage to improve system resilience and psychological safety in your team
Full Article
Molly Struve discusses a brutal six-day outage that nearly sank a company. She explains technical lessons like the importance of FMEAs, shadow traffic, and exercising rollback mechanisms. She shares why the human elements - widening your circle early and having a VP who acts as a defender - are what truly build psychological safety. By Molly Struve
DeepCamp AI