News · 24 July 2024
After the CrowdStrike outage: resilience is a property you test, not assume
On 19 July 2024 a faulty content update to CrowdStrike's Falcon endpoint security software caused Windows machines to crash and fail to restart. Microsoft estimated that about 8.5 million devices were affected. Airlines, banks and hospitals reported disruption. The company later explained that a defect in its content validation allowed a problematic configuration file to be released.
It was not a cyberattack. It was a reminder that the most damaging incidents are sometimes self-inflicted, and that security tools themselves are part of the attack surface and the reliability surface.
Three engineering lessons
Stage your rollouts. The most robust defence against a bad update is not shipping it to everyone at once. Release to a small ring, watch the health signals, then widen. Customers can ask vendors how updates are staged and whether they can control the pace.
Know your recovery path before you need it. Many organisations discovered that recovering machines needed hands-on access, encryption recovery keys and a lot of staff time. A recovery runbook that has been rehearsed on a handful of machines is worth more than any policy document.
Monitor the thing that monitors you. Agents running with deep privileges deserve the same scrutiny as any critical dependency: version tracking, an inventory of where they run and alerts when their behaviour changes.
The health-care dimension
For hospitals, resilience means being able to operate when systems are down. That includes clinical workflows that degrade gracefully, paper or offline fallbacks, and an understanding of which systems depend on which. This is a data and architecture exercise as much as a security one: draw the dependency map, mark the single points of failure and rank them by clinical impact.
A short checklist
- Do you know every critical system that auto-updates, and who controls the schedule?
- Can you recover a representative machine within your target time, and has anyone tried?
- Do your dashboards show service health, not just security alerts?
- Have you tested what happens when a core supplier fails?
Resilience is not a feeling. It is evidence from rehearsals.
Written in July 2024; details reflect the situation at that time.