Cloudflare · 2019 · ~30 minutes
A single regular expression pushed CPU to 100% across the entire network
A new firewall rule shipped with a regular expression that could take catastrophically long to evaluate against certain inputs. Deployed globally, it pinned CPU to 100% on machines handling traffic for a huge share of the internet almost simultaneously.
Systems Involved
Timeline Replay
T+0m
A new WAF rule with a vulnerable regular expression is deployed globally.
T+1m
CPU usage spikes to 100% on edge machines worldwide as the regex hits pathological inputs.
T+2m
Customer traffic across the network begins failing or timing out.
T+17m
Engineers identify the offending rule and prepare a global rollback.
T+27m
The rule is disabled globally.
T+30m
CPU usage returns to normal and traffic recovers.
Root Cause
The regular expression used nested quantifiers that could cause catastrophic backtracking on specific input patterns. Because the rule was deployed to every edge machine at once, ordinary traffic was enough to trigger it everywhere within minutes.
Recovery
The team used an internal kill switch to globally disable the WAF's managed rules, immediately relieving the CPU pressure, then rolled back the specific rule before safely re-enabling the rest of the rule set.
Lessons Learned
- Regular expressions are code and deserve the same review, testing, and complexity limits as any other code path.
- A global kill switch that can undo a bad deploy in seconds is worth building before you need it.
- Progressive rollout (even a fast one) would have contained this to a fraction of the network.