Measuring Value in Architecture

(Editor’s Note: This is part three in a series of four articles that will appear each Monday. Lockhart is a Director at Point Management Group. The previous article was about defining value.)

 

The purpose of measuring value is to help make decisions on whether to invest or not, and once executed to determine if that investment is realizing the expected benefits. As a result, value measurements may require some hard work, math, and plenty of analysis. Measuring value can sometimes have limits where the return on investment is non-financial or where it merely enables some other value-producing effort. For example, to measure the Value of the SuperWidget to the company, we may be looking at any of the following:

• Improved Customer Satisfaction
• Reduced Production Incidents
• Improved Adherence to Regulatory and Compliance Standards
• Protection of the Company Brand
• Increased Revenue

And there could be others. This points to the importance of ongoing assessment  of outcomes delivered and they value they provide not just at the end of an initiative, but on a continuous basis. Given their skillset and typical presence across many domains of business technology, Architects are well-positioned to perform this valuation work.

In order to measure the value of an Architect’s contributions they are often stuck trying to determine whether or not the work-products produced, or activities performed by the Architect were ‘good’ and therefore contributed to successful outcomes. As this can become subjective, a useful proxy can be found by looking at the lifecycle of work in which the Architect is engaged and objectively measuring one or more of the following key attributes:

• the Quality of the output
• the Cost of delivery
• the Revenue realized
• the Timeliness of the effort.

Quality

Measuring Value by assessing quality of what has been delivered is useful across many delivery scenarios. In addition, it is especially useful in cases where risk and compliance, security and privacy or even simple customer experience issues are paramount. Did quality improve? If so, by how much?

Quality as a Value metric can be measured objectively in several ways:

1. Production Incidents Over Time

 2. User or Customer Sentiment

3. Speed of Delivery and “Doneness” of Requirements

In most development environments, especially those using Agile delivery, new iterations of applications are rapidly released to users. Rate of delivery is an important quality metric due to the fact that new releases typically contain fixes or enhancements to address user needs or wants. A higher frequency of releases that are delivered to the user should results in higher user satisfaction and better application quality.

This can be measured quantitatively in several ways:

1. 1. Number of releases – This is the basic count of how frequently new features are delivered to users.
   2. Agile stories which are “done” in a certain time period – Counting the number of “stories,” or user requirements, which are actually shipped to the user, provides a more granular measure of the rate of delivery and thus quality.
   3. User consumption of releases – For example, measuring the number of users who download or install a new patch or software update.

4. Exemption and Tollgate Pass Ratio

The rate at which applications progress through the Software Development Lifecycle is often controlled by quality checks at specific points in the process. These tollgates exist to prevent poor quality applications or applications that may generate incidents from moving into production as-is.

Assessing the number of times an application passes each of these reviews versus how many times it fails that check is a metric that can be used to describe quality. While the raw score lacks context (i.e. why did it fail the check?), it is a measurable and comparable metric. In addition the number of exemptions (failures during analysis which get overridden can be exceptionally valuable in understand architect influence and the value of engagement model and directly lead to the accrual of healthy vs unhealthy technical debt.

Not every approach is valid all of the time. But chances are good that one of these methods will work when faced with a scenario where the objective measurement of Quality resonates the most in articulating Value delivered.

Tenet: Aggressively Find Stakeholder Value Metrics.

The architect practice should consistently and aggressively discover objectives, KPIs and other capability analysis measures in common use in the organization. This should include all architects and tracked as a part of the engagement model and be included in the role and job descriptions.

Quality as a Measure

In order to leverage Quality as the measurement, an effective baseline and established KPIs are necessary. For example, to know what ‘good’ looks like in terms of Production Incidents, a baseline or target must have first been defined at the outset of the effort (Something an Architect can do, incidentally).

An Architect impacts Quality by delivering or performing any of the following:

• Engagement Models
• Exemptions/Pass/Fail Rates (Architecture Analysis)
• Risk/Compliance Analyses and Assessments
• Capture and Analyze Architecturally Significant Requirements
• Reference Architectures: Master, Context, Product-Specific
• Specialists in the Architecture Practice: Solution, Security, Data, Integration, etc.
• Standards and Frameworks (Principles)
• Define and Articulate Solution Options (Decisions)
• Objectives (Key Performance Indicators / Success Metrics)

Any positive Quality measurement is attributable, at least in part, to the Architect and Architecture. This would then become part of the Value Story and articulated as such.

Cost

A cost perspective on delivering Value would be useful in scenarios where operational efficiencies are being sought. For example, application rationalizations, budget cuts or post-merger integrations are prime scenarios for this type of measure. In these cases, a large reduction in run-rate operating cost (cost reduction) or a large, planned cost not incurred (cost avoidance) is associated with a large amount of Value delivered. What was the cost of delivering the output and was this less than expected?

Cost as a Value metric can be measured objectively in two main ways:

1. Cost Reduction
   Cost reduction, also called Cost-saving measures, are any actions that lower current spending, investment, or debt levels. They result in a tangible financial benefit for the organization and are measurable. The amount of money saved as a result of these measures should always be reflected in the financial statements and next year’s budget. Actual cost savings should be visible in the financial statements compared to prior periods; planned cost savings should be reflected in the budget.Some examples of cost saving measures are: reduction of employees, renegotiation of existing contracts, negotiated price decreases for materials, elimination of software or hardware costs.
2. Cost Avoidance
   Cost avoidance measures are any actions that result in avoiding costs in the future. They represent potential increases in costs that are averted through specific preemptive actions. These measures will never be reflected in the budget or the financial statements.Some examples of cost avoidance measures are: a reduction of a proposed price increase from a vendor, the elimination of the need for additional headcount through process improvements, or a change in maintenance schedules for critical equipment to avoid work stoppages. A common example is implementing a new process for $X that negates the need to continue to do something else that costs $2X.

In some cases, both reduction and avoidance are part of the same cost-based Value measure. However, most scenarios that Architects are familiar with (e.g. program delivery) cost avoidance is the metric that is most often associated with Value. That is, we planned on the SuperWidget project costing $350K to deliver, but because of the Workforce Management Analysis conducted by the Architecture team, we are able to save on labor and deliver for $200K. A cost avoidance of $150K. The Architecture team can and should claim some part of that Value delivery.

Cost as the measurement requires a baseline. In order to know what a Cost Reduction is worth in terms of value, the existing run-rate must be known. Similarly, for Cost Avoidance, it must be known up front what costs are expected to be incurred so that the avoidance can be calculated.

Tenet: How Many Business Cases Does the Team Write Themselves

The practice should be challenged and have a shared objective to write as many business cases as are possible and submit them through the same channels other business units do. This provides a measure of innovation.

An Architect impacts Cost when delivering or performing any of the following:

• Create and/or Review Business Case
• Manage and Prioritize Initiative Portfolio
• Capture and Analyze Requirements
• Define and Articulate Solution Options (Decisions)
• Cost/Benefit Definition and Data Collection
• Cost-Benefit Analyses
• Standards and Frameworks (For Financial Analyses)
• Workforce Management Analyses (Assignment)
• Budgeting (Benefits Realization)
• Financial Management and Reporting (Benefits Realization)
• Sequencing and Prioritization (Investment Planning and Prioritization)

Any successful Cost Reduction or Cost Avoidance effort is attributable to analysis and activities primarily performed by an Architect. It is a powerful metric for demonstrating the Value that Architecture can deliver.

Revenue

The measurement of Value most often used is based on the metric of Revenue. This makes sense since revenue is perhaps the most clearly visible result of the delivery of Project X or Program Y and is reported monthly, quarterly and annually. Measuring Value by comparing revenue is most useful in scenarios that involve discreet products and services or other offerings with a direct association to a line of business. What was the revenue derived from the output of the effort?

Revenue as a Value metric is usually measured objectively in one of two ways:

1. Existing Revenue Increase Over Time
   Organic Growth of existing revenue streams is a useful way of measuring the value of Project XYZ and the impact of architecture on growing the business. The example below is existing revenue growth that is PROJECTED, but the measurement of ACTUAL growth after the projects have gone live is similar.

2. New Revenue Added
Growth of revenue by adding new business as a result of Project XYZ that didn’t already exist previously is a variation on the theme. In the example below, specific projects are projected to lead to new revenue by way of mergers and acquisitions.

There is a fine line between these two Revenue metrics and usually only one is applicable in a given context. In the case of existing revenue lines that are enhanced, the context typically involves an existing product or service that is having an update or new version. That is, the Revenue baseline for that business activity already exists and absent any other changes the increase (or decrease) in Revenue post-delivery is attributable to the work effort. In the case of new Revenue, the context involves a new product or service offering where none previously existed. In this case, even without a baseline for the related business activity, resulting revenue is entirely attributable to the work effort.

An Architect impacts Revenue when delivering or performing any of the following:

• Create and/or Review Business Case
• Manage and Prioritize Initiative Portfolio
• Capture and Analyze Requirements
• Reference Architectures: Master, Generic, Product-Specific
• Other Architectures: Solution, Security, Data, Integration, etc.
• Capability Models (Business, Technical)
• Standards and Frameworks (Define, Publish, Socialize)
• Define and Articulate Solution Options
• Key Performance Indicators / Success Metrics
• Stakeholder Alignment
• Budgeting
• Financial Management and Reporting
• Roadmaps
• Stakeholder Communication
• Engagement Models

New Revenue or Increased Existing Revenue is a result of the entire effort. It is difficult to attribute any portion of this Value directly to, say, a really good Capability Model. But as in the examples above, the rigor that Architecture brings to delivering the product or service is an important aspect of the Value equation and should be front and center in the Value Story.

Timeliness

The fourth most common way to measure Value is to assess the time to market. The oft-quoted phrase that time is money has some bearing on this. This is a useful metric for determining Value in scenarios involving net-new products or services, partnerships with third parties, regulatory compliance efforts, and efforts that are tied to the calendar year (e.g. open enrollment at a health payor). How long did it take to deliver the output and was this less than expected?

Time as a Value metric is typically measured objectively in one of two ways:

1. On-time Delivery
   This metric effectively depends on two things: the SCOPE of what is being done and WHEN it is expected to be delivered. In the example tracker below, targeted savings are measured by initiative over time.

 

2. Accelerated Delivery
Value is measured by the degree to which various business and IT activities are sped up. These are quantitative measures that can be directly tied to dollars if need be.

 

These metrics are mutually exclusive and only one is applicable in any context. In the on-time scenario, the Value delivered is that expectations were met, i.e. things worked as they should have. This isn’t always the most powerful argument except in cases where the business is routinely impacted by delayed delivery. In those instances, the Value of delivering on-time is comparatively enhanced. Accelerated Delivery scenarios involve leveraging Architecture to accelerate the lifecycle usually by streamlining existing processes (for example via frameworks or new methodologies) or enabling large scale reuse of existing capabilities.

Time as a Value metric requires a baseline to be meaningful. If the program doesn’t know how long it expects to take to deliver the SuperWidget, then shaving 3 months off by reusing existing Integration Architectures won’t be perceived as Value delivery. This is true for any Time as a Value metric.

An Architect impacts Timeliness when delivering or performing any of the following:

• Manage and Prioritize Initiative Portfolio
• Capture and Analyze Requirements
• Reference Architectures: Master, Generic, Product-Specific
• Other Architectures: Solution, Security, Data, Integration, etc.
• Capability Models (Business, Technical)
• Standards and Frameworks (Define, Publish, Socialize)
• Define and Articulate Solution Options
• Stakeholder Alignment
• Roadmaps
• Sequencing and Prioritization
• Governance (Reviews, Toll-Gates)
• Stakeholder Communication
• Engagement Models

The Value Measure

While these four approaches to measuring outcomes will produce the metric needed to demonstrate Value delivery by Architecture, the metric alone is useless if it is not paired with a compelling narrative. In the case of Revenue as a Value metric, it is extremely difficult to tie part of a revenue increase directly to an Architecture activity or work-product. The case can still be made, however, with a good narrative produced for the right audience backed by the underlying data.

Understanding how to compose and deliver a Value Story that resonates with the audience is one of the most critical skills for an Architect. It isn’t just smoke and mirrors. It is a data-driven story that demonstrates how and where Architecture delivers Value.