By Gerben Wierda
Part One can be found at https://www.architectureandgovernance.com/applications-technology/beyond-chess-and-the-art-of-enterprise-architecture/
EA Chess was devised and written because orthodox instruments like ‘a future state and a set of design principles’ in the real world never work well enough. Eight years later, I have learned other lessons — reality is a great teacher if you allow it to be— and stumbled upon more insights, the most important of which are:
Complexity Crunch
That IT change is hard — and how to handle that — was the starting point of EA Chess. But I have been pondering deeper why it is so hard, because insight in the why might enable us to do better. I have come to the conclusion that the difficulty of change of digital landscapes is an unavoidable emerging property of their size. The reasoning is as follows:
The IT-revolution is a digital revolution. But digital technology in the end is nothing more than manipulating integers, which in the end can be expressed in the simplest of integers — bits — which we manipulate using classical logic (which is why we often call 0 ‘false’ and 1′ true’). We have thus created logical machines and like the physical machines drove the industrial revolution, the logical machines drive the digital revolution. With those logical machines, we have created logical landscapes of unimaginable size and power.
The logic we fundamentally employ in IT has some fundamental properties, which — because they are fundamental — lead to unavoidable outcomes in our digital landscapes. These properties are:
- Strength: logical operations are perfect and free from physical reality. That two plus two is four is not ‘false here and true there’ or ‘true now and false later’. This makes logic — and by extension: everything discrete — perfectly reliable (and thus perfectly transportable) and it thereby enables us to create these huge coherent landscapes of usable logic.
- Weakness: logic is inflexible, unforgiving. Logical operations are by nature extremely dependent on perfect inputs to produce their reliable outputs. That makes the large logical landscapes unavoidably brittle. We encountered this initially when our software programs started to grow large in the 1960s. The phrase ‘spaghetti code’ was coined for one such brittleness, for instance. A change ‘here’ would break something ‘there’. This pattern of dependencies has grown and we have effectively been fighting brittleness ever since. Mostly what we have been doing is creating abstractions — object orientation, service orientation, etc. — to make this manageable, and abstractions have enabled us to grow.
But abstraction is not a perfect tool. Mathematician founders of IT like Edsger Dijkstra argued it should be (e.g. mathematical proof of correctness), but in the end this turned out to be undoable. And there is probably a good reason: if these IT-systems need to affect the real world, then it is by definition impossible to catch ‘correctness’ in discrete rules. Ludwig WIttgenstein was right.
So, we end up in a world where we have added layer after layer and island after island of abstraction, but without perfect isolation. We might for instance be able to sometimes decouple technically, but we are seldom able to decouple logically let alone in reality. We still end up with ‘loosely coupled (logical) spaghetti’. This means that slowly but surely the growth of fundamentally brittle IT is leading to inertia. IT is getting harder and harder to change. Yes, two speed IT exists, but it is ‘adding’ that is relatively simple, and ‘changing’ that is particularly hard. We have been mistaking growth for change. Larger organisations now regularly have to go on many-year-long, expensive, and painful journeys to change key parts of their landscape. And not just organisations feel this — unavoidable — inertia, societies do as well. Politicians may want to change policies overnight, the IT changes needed to implement them may take many years. In short, the world has bought an enormous amount of productivity, but the price is being paid in agility, and we’re by far not done. E.g. more complex landscapes have more attack vectors and are more vulnerable. Demands on security, risk, compliance and assurance make life harder. Inertia increases everywhere. The human world is slowly petrifying because of its reliance on ever harder to change digital landscapes.
Taking this perspective, architecture becomes ever more about agility than about the actual strategic business goals you want to achieve. Architectural choices — the choices that are hard to change — have a life span of 15-20 years, business goals change every 4 year, maybe. Using a ‘target state’ based on the current business strategy was maybe doable in the 1960’s when orthodox architecture ideas were created, but that world only had very little IT. Orthodox approaches have long since lost their effectiveness (but the EA industry doggedly keeps doing and promoting them).
It also means people like Grady Booch and Martin Fowler were right when they defined architecture simply as ‘that what is hard to change’. Parts of our organisations and societies become petrified because of IT, and it is important to have as little of that rigidity as possible. And it becomes very important to really pay attention to which parts you accept to be ‘fixed’. I guess that at some point, our changes will sometimes have to become far more radical — not just technical, but also social, legal, etc. — in many ways to be possible at all.
If I descend even deeper, I see an even more fundamental, mathematical, issue. As I wrote above, digital computers handle integers, the ‘bit’ being the most fundamental integer there is. Digital computers do not handle reals, they give the impression, but that is fake. And as we know from foundations of mathematics, it is impossible to express the reals into integers. And that is a problem. In the end we need the power of the reals to be effective in the real world. This is why my personal motto since the 1990’s has been “The world is not Q, it is R” (a pun on math symbols which reads as “the world is not rational, it is real”) and why that is the motto of the EA Chess book. Again, it is not just a matter of scaling, no amount of scaling will overcome this fundamental handicap of digital technology: the fact that it is digital. IT’s strength is its weakness. Aside: this is why scaling of digital AI has always been it’s key problem. Watson was hailed as the step that would bring us true AI, but it fizzled, because scaling was no longer feasible. Large language models like GPT will show the same is my prediction. We might with a lot of engineering brilliance produce really usable tools, but the fundamental barrier stays as long as we work in a digital setup.
You don’t hear as much anymore about the singularity point. That is because the opposite is the case: we’re on our way to a (digital) complexity crunch. Pushing that back as far as possible has been the true focus of IT management for quite a while now, even if not many people were aware.
Looking to the future, approaches like analog and/or quantum computing (in effect computing with reals) will be required to escape the digital complexity crunch. But these are for the foreseeable future pie in the sky, and they come with their own very fundamental conundrums.
Consequences of Complexity Crunch for EA: Agility is key
The slow accumulation of inertia into complexity crunch also explains our move to Agile/DevOps methods. Agile is: “flexible humans adapting to inflexible IT”. Agile is simply the next step in abstraction: object-orientation, but now at organisational level (the ‘product’ being the ‘object’). This ‘organisational architecture’, however, presents an important challenge to IT-architecture. The idea that ‘the best architectures emerge from self-organising teams’ is both true as well as frightfully naive. The worst architectures also emerge that way. We’re still finding out how to marry local (team) autonomy (object-orientation at business level) with good IT-architectures. I fear that apart from Continuous Integration and Continuous Deployment, this also requires Continuous Discussion, and that the risk of ‘talking issues to death’ is another price we pay for increased complexity. Here, consent-based decision making really becomes essential.
The ‘agile’ consent-based, collaborative, mostly bottom-up EA Chess approach with checks & balances has held up well, I think. A hierarchical setup of alignment and peer-review boards to discuss architecture seems to work to make sure what emerges as architecture of the organisation is as healthy as we can make it. Each team provides a ‘architecture’ team member to the first level, and then up and all the discussions are decidedly about the design choices we make. A hard part is: when and where do you intercede with checks & balances if there is no true clear moment anymore (as there was when we were more project-based and each project came with a clear up-front design)?
It turns out so far that the outcome is an uneasy tension between continuous change on the one hand and clear designs being written on the other. Better ways to document designs are still needed. Of course, that assumes that you are really doing Agile, and not some sort of ‘Dark Scrum at Scale‘ where the entire organisation has become one big project with lots of top-down steering and where we call the smallest project work-package a sprint.
At the strategic level, the change to Agile is not yet recognised across the board (or by every board). At the strategic level, organisations are historically and naturally very goal-oriented, be it urging ‘extreme customer focus’ or ‘outside-in views’. The problem is that less and less of architecture can be steered that way. These days, I argue that the strategic plane should be partitioned into four different parts:
- Forecasting
- Backcasting
- Uncertainty Planning (a.k.a. Scenario Planning)
- Strategic Agility
The first three are more or less established. Forecasting is trying to forecast the future and deciding your strategy to handle this. It was big in IT in the 1980’s and 1990’s but fell in disgrace because, well, forecasting is almost impossible. There are parts which can be reliably forecast, however. Demographics is most often mentioned as an example, but sometimes technology is predictable too. In 2016, for instance, it was pretty clear for those understanding the technology that containers would very likely become big, and getting ready for it — as containers are by no means a free lunch — was a good move.
The classic idea of having a ‘future state’ (where ‘state’ is about as silly a word as can be if you think about it) is backcasting: where do I want to be and how do I get there? Backcasting has been the main focus of orthodox EA for decades now.
Often, forecasting and backcasting are thrown into the same bucket, called ‘the future state’.
Uncertainty planning is kind of the necessary second part of Forecasting. Here you consciously look for uncertainties that are (a) truly uncertain/unpredictable and (b) of consequence for your organisation. From an architectural perspective this is very important but it is seldom done. Architecture are those things that are hard to change and thus an inventory of potential big changes can help you decide on where you want to be flexible and where not. For architecture, you do not need the full scenario approach, just making sure you manage your portfolio of strategic uncertainties is enough.
In recent years, I have been promoting the idea that the growing inertia of IT-landscapes — especially now that the speed of change of architecture is sinking below what is acceptable from the perspective of Forecasting and Backcasting — requires a fourth independent strategic element: I have dubbed it Strategic Agility. Strategic Agility says that you do not only need to execute your current strategy, you also need to make sure you can change your strategy later. Just to give some push back to the fore- and backcasters, I sometimes exaggerate that from an architecture perspective, the organisation’s business strategy doesn’t really interest me, because architecture is about choices that have a far longer life span than the current business strategy. And I’m only partly joking when I say that.
An Agile EA Manifesto
If there was to be an ‘agile EA’ manifesto — it could be summed up as:
- Strategic agility and uncertainty analysis over fore- and backcasting
- Requirements over principles
- Collaboration over division of labour
- Design skills over design principles
- Create your abstractions ‘risk based’
- Govern your architecture through ‘checks & balances’
- And last but not least:
- IT skills over powerpoint skills
These together give an organization the means to fight inertia instead of freezing when confronted with it, or fleeing into make-believe simplified or plannable worlds. They are also largely in complete opposition of what I would call ‘the EA orthodoxy’.
The Psychology of Architecture
The analysis of EA Chess, and even more what has come after, is a hard sell, in part because the simplistic ideas of orthodox EA are as seductive as they are mistaken.
Concepts like Strategic Agility are the opposite of seductive. You’re in effect telling top management: “it’s not simply about executing your own, current strategy, it’s also about enabling your successor to kill your darlings”. It also requires people to accept ‘complexity crunch’ when for decades everyone has been told (sold) ‘rapid change’ — and a singularity that is around the corner — and then — to top that off — management has to accept that they have to pay attention to IT because of … IT and not only because of business goals. Real EA is becoming the mother of all ‘hard sells’.
What makes it even harder is that the EA world is awash with people selling simplistic and seductive stories. Edsger Dijkstra’s comment that “[IT] is the discipline with the highest quack density in the world” is brought back to mind time and again. And yes, I know what I am saying would make me a quack in Edsger’s world, but that is because he was smitten by the mathematical clarity of young Wittgenstein and I have been convinced by the older (and wiser) version of the philosopher.
All of this trouble to convince people has made me dig into the question “how do humans make and change convictions, anyway?” Because architects are not just in the game to think of the right design choices at many levels, they also are tasked to convince others of their — assumed — valuable insight. And this musing coincided with how conspiracy theories and such have florished in the world.
It has become clear to me that while we humans believe we build our convictions based on observations and analysis, research has shown this is largely a myth. We are far less intelligent than we think. Our intelligence is above all geared for speed and energy-efficiency, but that means that we seldom really look deep or analyse. Psychological research shows that our convictions have far more effect on our observations and our memories than the other way around. I have concluded that these convictions need to be stable so we can act fast and cooperate easily in groups, hence changing convictions is unavoidably hard (but not as hard as changing IT…). We all — not just the conspiracy theorists — ignore facts that are in conflict with our existing convictions. They’re not crazy. They’re simply human. We all are limited.
Now, the business of an enterprise architect is the convictions of engineers and management. And they are human too… Therefore, architects should probably pay more attention to getting a seat at the right tables than simply pay attention to creating the best designs or arguments. An architect should be a ‘trusted advisor’ and trust comes from regular interaction, not the other way around. Good luck with getting a seat at the management table when EA has been so often a failure and the world is awash with seductive quacks. Good luck with getting a decent relation with the engineers when your added value is mostly bureaucratic and has only a tenuous relation with IT-reality.
At board level
Organisations have wrestled with IT almost as long as we have had IT. In general, the most used pattern has been to make a single board member responsible for IT. At some point this was the Chief Information Officer or these days the Chief Technology Officer. But this pattern, I think, hides a fundamental flaw: it separates ‘IT’ from ‘the business’ when it has become obvious that this separation doesn’t exist anymore. If your organisation has a complex enough landscape that it needs architecture, the IT is everywhere. That means that the role a Lead Architect of an organisation needs to play should be like that of the Chief Legal Counsel. You need someone in that role who is your trusted advisor on the hard decisions with an IT component, and that must be someone who understands the bits and bytes of technology in the same way your Chief Legal Counsel understands the bits and bytes of the law.
Gerben Wierda is author, occasional speaker, and a lead architect at a large financial institution. He has written this in a personal capacity. His blog is at R&A IT Strategy & Architecture.