By Leonard Greski, Chief Scientist
I recently participated in five episodes of the Agile Thoughts podcast with host Lance Kind and his guest Adam Tornhill, author of Your Code as a Crime Scene. Our conversation covered topics ranging from the economics of software quality to the relationship between code problems and people problems. A key technique Adam uses to find bad code is to identify hotspots, modules of code that change frequently. He cited multiple sources of research supporting his assertion that frequent changes to complex code predict declining quality (Tornhill, 2024). The “crime scene” metaphor of Adam’s book intrigued me because it aligns nicely with two key roles I always see on out-of-control software projects, the firefighter and the arsonist.
During our discussion I noted that often product quality problems are people problems because the way we work often rewards the wrong behaviors. Just like we can identify hotspots in code, we can identify hotspots in the organization. We typically call them “firefighters”. You know, the people solve urgent problems, “save the day” with key customers, and receive lots of visibility and praise in internal company communications. They are frequently well compensated. Very well compensated.
Having worked with scores of large organizations, I observe that wherever there is a culture that rewards firefighting, the firefighters are enabled by a network of arsonists, people who start fires for the firefighters to extinguish. A culture of firefighting is a major impediment to effective strategy execution because as companies organize around capabilities and establish a stable system of delivery, the need for firefighters is drastically curtailed.
Profiling the Firefighters
Criminal profilers take one of two approaches to connect crime scene information with potential perpetrators: top-down and bottom-up. With the top-down approach, investigators gather information at a crime scene and make inferences about the characteristics of the offender, categorizing him/her as either an organized or disorganized offender, fitting the details to existing types. A bottom-up approach starts with details, makes no assumptions about the offender, and uses the details to match patterns with databases of past offenses (McLeod, 2023).
I have examined hundreds of “on fire” software projects over my career, and every project has had at least one firefighter valiantly working to extinguish the inferno. My experience indicates that there are four basic top-down profiles of firefighters.
| The Wizard | Leverages deep business and technical knowledge to solve problems that are impenetrable to everyone else on the project team. While the Wizard may be friendly to other team members, s/he is typically introverted, and often so busy that the most critical information about the product or project is locked away in the Wizard’s brain, inaccessible to other team members until a crisis occurs.
|
| The Grinder | Leverages his superior physical stamina to outwork everyone else on the project team. During crunch time he typically puts in 20 – 40 hours a week more than anyone else on the team and is seen as indispensable due to his high levels of energy and commitment. What often goes unnoticed about the Grinder is that he is frequently the source of his own backlog of fires, as sloppy code and ill-advised fixes are deployed to production riddled with defects.
|
| The Zen Master | Takes control during production outages or other crises, calming the team and methodically restoring broken applications to service. Eminently unflappable, asks the right questions, directs the team to collect relevant data, and uses it effectively to triage problems. Able to manage scores or hundreds of anxious team members through outage triage while keeping senior executives apprised of progress and impact to customers.
|
| The Customer Advocate | Leverages deep customer empathy to encourage teams to take on heroic commitments. Seen by senior executives as someone who is clearly a leader because she confidently says “yes” to difficult assignments, enabling the company to close key sales opportunities. Able to put a positive spin on the team’s work by focusing on the heroic efforts being expended, even when failing to meet key delivery commitments.
|
The Path to the Dark Side: How Firefighters Become Arsonists
An effective firefighter extinguishes a blaze and ensures there are no flare ups in the smoldering debris at the fire scene. Aside from the rewards of heroism, firefighters experience a rush of adrenaline when working a fire that is similar to the experience of professional athletes participating in a high stakes “must win” game.
In business, as the firefighter effectively extinguishes fires by addressing root causes rather than symptoms, the frequency of fires decreases, lessening the need for the firefighter. Sometimes the absence of fires (and the subsequent reduction of adrenaline-fueled work) leads a firefighter to begin moonlighting as an arsonist, starting organizational fires that only s/he can put out. As firefighters grow in authority and rank in an organization, the most sophisticated ones figure out how to work the system to manipulate other people into unwittingly set fires to be extinguished by the firefighter. Even if the arsonist is ultimately caught and disciplined, the firefighter’s fingerprints weren’t on the gasoline can so s/he is handsomely rewarded.
And so the cycle continues.
“Say it ain’t so, Joe,” or Senior Executive Level Arson
Arsonists in the c-suite cause fires ranging from tens of millions to hundreds of millions of dollars of cost. Their approaches to fire setting typically come in one of four forms. First, there is the “delegation of faux authority.” In this scenario, a senior executive publicly delegates authority to make a strategic decision to a more junior executive. However, behind the scenes the senior executive mandates that the only allowable decision is the one preferred by the senior executive. The decision is likely to start a fire weeks or months later, when the executives’ attention is focused elsewhere in the organization. After the fire starts and grows into a five-alarm inferno, the junior executive is held accountable for the poor decision after the senior exec gets to save the day.
Another approach is the “wishful thinking” approach, where a senior executive constrains the team to plan only for the most optimistic scenario, doing things like back loading integration and stress testing or mandating a one year or longer requirements gathering effort before any working tested product is built.
Yet another approach that is periodically employed by sales leaders is to make unrealistic commitments to a high-profile customer, and use the importance of that customer relationship to bully the product team into an heroic effort. Sometimes this approach has the beneficial side effect of cannibalizing labor from a competing initiative that is disfavored by the executive, undercutting a competing executive’s priorities.
Finally, a variant of the “delegation of faux authority” scheme is “low man on the totem pole.” In this scenario, a senior executive artificially constrains a solution for purchased hardware or software by setting an unrealistic budget. The team proceeds to solve to the budget, not to the actual business need. The resulting Capital Authorization Request (CAR) has four or five layers of signatures, starting at the bottom with the architect who designed the solution, through Director, VP, up to Business Unit Head and Chief Operating Officer (depending on the size of purchase). Months later, after the purchase fails to meet the scale of business, the Legal team retrieves the CAR from files and holds the person whose name appears at the bottom of the signature list accountable for the poor decision.
Arresting the Arsonists by Establishing the Right Forensics
At this point the picture we’ve painted seems bleak. Firefighters gonna firefight. Senior execs gonna weasel their way out of accountability. However, effective practice of architecture, especially business and solution architecture, can establish the forensics needed to prevent fires from being started, as well as to limit executives’ ability to delegate poor decisions down the chain of command.
The first tool in our toolkit is the mission of every architecture function: to be the objective guide to decision-making about business and technology capabilities. With this purpose as the north star for an architecture team we can make executive decision-making transparent, establish objective decision criteria, and ensure accountability for results based on the economic value of decisions to be made. Starting with the business architecture, capability, financial, and usage models establish cost and benefit targets that constrain subsequent decisions as well as monetize the value of risks.
Next, solution architects consume the business architecture models and refine them by breaking them down into runtime availability, resiliency, security, and response time constraints that must be delivered within the overall cost budget established by the capability level financial model.
Finally, architects can influence product delivery teams to mitigate risk early in a project by scheduling work to load and stress test the architecture based on capability usage models. Architects can also ensure teams use risk management approaches like hedging and transfer to validate proposed solution well in advance of a go live date or implement viable contingencies if a high risk / high reward design fails to work as promised.
Yes, every firefighter eventually needs an arsonist when s/he runs out of fires to fight. Effective architecture practices can help companies prevent fires from being started by establishing objective criteria for decision-making, clarifying decision-making authority, and eliminating the delegation of responsibility without authority.
Len has over 30 years of experience helping large organizations generate billions of dollars in economic value by leading high risk, high visibility business and digital transformations. He currently serves as Chief Scientist at LeadingAgile, the leader in helping companies grow in value through business agility. He leads engagements
with large clients to help them improve flow by restructuring value delivery around their business capabilities, consistently delivering 30 – 85% improvements in productivity, quality, and business value. Len received Bachelor and Master of Arts degrees in Sociology from the University of Illinois at Chicago with concentrations in Organization Theory, Research Methods and Statistics. He can be reached at leonard.greski@leadingagile.com.
References
McLeod, Ph.D., Saul, Offender Profiling in Psychology, Simply Psychology website, November 2023, retrieved May 25, 2024.
Tornhill, Adam, Your Code as a Crime Scene, Second Edition (Beta): Use Forensic Techniques to Arrest Defects, Bottlenecks, and Bad Design in Your Programs, Pragmatic Programmers LLC, 2024.