April 10, 2023
Counting Forest Fires
One of the key tenets of How Complex Systems Fail is that change introduces new forms of failure. Static systems are safe because their failures — and fixes — become highly predictable over time. The set of unknown outcomes approaches zero the longer people observe and record the system’s behaviors.
Building software, however, continuously introduces new changes to a dynamic, complex codebase. Each release induces a period of rapid learning when teams attempt to understand the behavior of the updated system. When things inevitably break, incident teams can borrow a lesson from fighting forest fires: a firefighter prioritizes the time it takes to respond to a burning forest over the total number of fires.
If you were asked to evaluate how good crews were at fighting forest fires, what metric would you use? Would you consider it a regression on your firefighters’ part if you had more fires this year than the last? Would the size and impact of a forest fire be a measure of their success? Would you look for the cause—such as a person lighting it, an environmental factor, etc—and act on it? Chances are that yes, that’s what you’d do.
…The parallel to software is obvious: there are conditions we put in place in organizations—or the whole industry—and things we do that have greater impacts than we can account for, or that can’t be countered by individual teams or practitioner actions. And if you want to improve things, there are things you can measure that are going to be more useful. The type of metric I constantly argue for is to count the things you can do, not the things you hope don’t happen.
Counting Forest Fires: Incident Response Metrics, Honeycomb
Funding
Coast, a tool for creating demos for API-first companies, raised $2.1m in Seed funding.
Agnostiq, developer of software for quantum computing, raised $6.1m in Seed Extension funding.
Redefine, an AI-powered CI optimization platform, raised $8.5m in Seed funding.
Chroma, AI-native open-source embedding database, raised $18m in Seed funding.
Honeycomb, an observability platform, raised $50m in Series D funding.