IT Landscape as a Complex System of Systems – Part 1

“Excessive complexity is nature’s punishment for organizations that are unable to make decisions.”

– Gregor Hohpe’s Law

By Vasily Yamaletdinov, Principal Consultant, Zühlke Group

Increasing interest in business digitalization today affects almost the entire spectrum of organizations – from start-ups to large enterprises, and in a variety of fields. In recent years, innovative digital products have dramatically changed the way they deliver value to consumers and have increased their responsiveness to market changes, which is forcing large enterprises to change their attitude towards investments in IT, perceiving it not just as a cost center, but as one of the key assets that bring profit. However, the desire of enterprises, with their IT landscapes inherited over the years of existence, to participate in the digital race along with startups often leads to an even greater increase in the complexity of the entire IT infrastructure, which in turn entails an unjustified increase in the cost of owning IT assets, risks of breaching security and business continuity, as well as an increase in Time-To-Market when releasing new products.

Systems Thinking

Systems thinking is an approach that allows you to manage attention and reasoning in the process of analyzing, designing, and implementing complex objects.

To begin with, it is necessary to remember that a system is a whole, consisting of parts, due to the interaction of which it manifests its properties (the so-called “emergence”) in the external environment. According to systems thinking, systems are recursive – each part of the system, in turn, is also a system (i.e., a subsystem in relation to the system), and the system itself is included in a certain supersystem, and so on up and down all system levels continuum.

Figure 1: Systems Continuum

For cyber-physical (engineering) systems, there are 3 types of interaction between the system and its surrounding systems – materials, energy, and information. Obviously, for the IT landscape as a system consisting of applications, it is the information interaction between them that is the main one.

Since we are dealing not with biological, but with cyber-physical systems, they, unlike living organisms, do not create themselves, this requires a special third-party system that controls the life cycle of the system of our interest, the so-called constructor (enabling) system. Quite long chains of constructor systems can often be observed.

The system-of-interest is usually understood as a system that has value for the end user (and who will pay us for it), i.e. a certain final product (goods or service) of the activity of the enterprise as a constructor system.

Another important definition for us is the system of systems (SoS) – a set or arrangement of systems that results when independent and useful systems are integrated into a larger system that delivers unique capabilities.

The key properties of a system of systems that distinguish it from conventional systems (Maier, 1998) are:

• independent systems for constructing/enabling its subsystems (the SoS manager does not have the authority to prescribe a general development-modernization for the systems included in the SoS).
• subsystems that can operate independently of the existence of the target system (no authority to tell system owners how to operate).
• emergence from merging into a system (one wants to get a function/behavior from the target system of systems that cannot be obtained from working with individual systems included in the system, and joint operation of these independent systems is required).
• evolutionary development (understanding what will happen in the system of systems at each next step of the system creation program requires research and additional approvals, because there is no role that knows how the system of systems is arranged at every moment – the subsystems of the system of systems as autonomous systems change independently. Therefore, there is a series of upgrades, and not a one-time designed project action, i.e. the creation of a typical SoS can never be considered fully completed.
• geographical distribution of the systems included in the SoS.

ISO 21839:2019 distinguishes four types of SoS, which differ in the degree of connectivity of their constituent systems – from tightly coupled to completely autonomous:

• directed, in which there is a designated architect who can issue orders to the project teams of the constituent systems and a manager who manages shared resources. Strictly speaking, this type of SoS is practically indistinguishable from a conventional system.
• acknowledged, in which there is a recognized architect, but he can only persuade the owners of the SoS components of the systems to change their systems according to the architecture developed by him (that is, the architect does not have a supervisory/governance authority, only mentoring about the architectural decisions he makes).
• collaborative, in which the owners of all systems agree with each other on each issue, but there is no architect, project manager or similar dedicated organizational role involved in the creation and development of SoS at the level of the whole system.
• virtual, in which the owners of the systems included in the SoS do not know anything about each other at all, and thus they do not influence each other explicitly.

System of Systems

Modern large enterprises, as a rule, have a very complex multi-level IT infrastructure that has been formed over decades of continuous development. Hundreds and even thousands of applications, databases, servers, storages, network devices and channels located both in their own data centers and in the “cloud” – all this is combined by tens of thousands of relationships into a system that is usually called the enterprise IT landscape.

However, for the sake of simplicity, by IT landscape we will only understand the totality of all applications (software and data) and their integrations since they are usually the focus of attention when it comes to the complexity of the landscape.

So, we are focusing on “our” system called “IT landscape”. The first thing to do to consider this system is to deal with its supersystem, the constructor system, and most importantly, with what is the system-of-interest for us.

Suppose that our enterprise is a retail bank that provides individuals with lending products and savings. In this case, the systems-of-interest for us will be the actual results of the provision of the relevant services – the credit funds issued to the borrower and the storage of funds received from the client in the account in a non-cash form.

The bank itself in relation to these systems-of-interest will be the constructor system.

Figure 2: Systems-of-interest and enterprise as constructor system.

Having dealt with the supersystem, it’s time to look inside the system of our enterprise and find out which parts of it implement the emergence required by customers within the systems-of-interest.

As part of the banking system, there should be subsystem providers for the corresponding products, as well as their constructor systems, the role of which is played by the relevant business units. Departments come up with the concept of using the services provided by their products to customers, create a description of the concept of the products themselves (what functional components they consist of, how these functions interact, what roles are involved in the processes, what work practices are used by these roles, what documents are generated, etc.).

Figure 3. Enabling and service provider systems before IT era.

Some 100 years ago, this could have been stopped, since all the indicated information processing functions within the product were performed exclusively by people using auxiliary tools (i.e. stationery), without requiring any additional services. At the same time, the systems-providers of products did not depend on each other.

With the advent of computers, the picture has changed significantly. Almost all functions began to be implemented by services provided by IT systems (applications) and front- and back-office employees attached to them. As a result, “our” system arose – the IT landscape, as well as its system-constructor – the organizational unit “IT Department”, initiating the division of the organization into “Business” and “IT”.

At first, everything looked relatively simple. There was one core system in the IT landscape that, as a building block, embodied almost all the required functions for business system providers (which is ideal from a systems engineering point of view) and at the same time practically did not interact with other systems. In the case of a bank, this was the Core Banking System (CBS).

Figure 4. Early IT landscape.

The frequency of the appearance of new products (and therefore new constructor systems for them) and changes in existing ones was extremely low, with which a small IT department quite coped by changing the settings of the COTS system, especially since its architecture, usually, was modular, which made it possible to buy additional necessary standard modules as the business evolves and the need for new products arises. At the same time, the overall level of development and penetration of information technologies in the clients and partners environments was relatively low, which did not require the provision of any enterprise services through digital channels. It is worth noting, however, that product enabling systems now have a shared dependency on a single enabling system, the CBS, which introduces business continuity risks.

However, the subsequent explosive development of information technologies with the simultaneous expansion of their availability led to a multiple acceleration of changes in the product line and the emergence of fundamentally new channels for its delivery to customers. The era of digitalization has begun, in which products and services are either provided to the client in a completely digital form (digital media, online banking), or physical products and services can be received by the client using digital services (for example, online carsharing). The range of COTS IT systems and technologies available on the market has expanded significantly. Naturally, all these changes could not but affect the structure of the IT landscape. The atomization of the IT landscape began, where, along with the CBS as the main operating and accounting module, many additional applications appeared to implement individual functions in product supersystems of the business, which made it possible to improve the quality of services to a certain extent, as well as increase the speed of their change. However, at the same time, the complexity of the IT landscape itself also increased markedly, since applications had an independent life cycle and, usually, were developed by autonomous organizational units (as constructor systems). In addition, the systems had to exchange data as part of integration IT solutions to support the functionality of product systems, both online and offline (in batches), which led to the complexity of the integration infrastructure. Thus, 3 system levels began to stand out:

• The level of the IT landscape itself (Enterprise Architecture) is a collection of all enterprise applications (sometimes including applications of its partners and customers) and their integrations.
• IT solution level (Solution Architecture) is a subset of IT landscape applications that interact with each other to implement some target behavior (process) in a specific business supersystem.
• Application level (Software/System Architecture) is a separate application that provides its behavior within an IT solution through programmatic (API) and user (UI) interfaces.

An additional factor in the increase in complexity was the use of the same applications as building blocks in a large number of new IT solutions as enabling systems for product system providers, which began to lead to conflicts both between supersystems of product providers (one product needs an attribute in some database entity, while such an attribute will interfere with another), and conflicts between the system levels of the IT solution and the application as its subsystem (an IT solution requires the application to perform a certain function, but the application constructor system considers this function to be non-targeted). Not surprisingly, these conflicts escalated from the level of the IT landscape up the chain, leading to conflicts between the final systems- constructors – business and IT organizational units. These conflicts were resolved, as a rule, either by introducing new applications with duplication of the functionality of existing ones, or by “pushing” non-target functionality and data into existing applications, which led to an even greater complication of the IT landscape.

Separately, it is worth noting such a factor in the complexity of the IT landscape as numerous mergers and acquisitions, as a result of which in the joint enterprise there appeared duplicating systems-constructors of products and IT applications supporting them.

Figure 5. Increasing complexity. System conflicts.

The next stage of increasing complexity is associated with the introduction of Agile approaches to creating systems-providers of products. From the point of view of the system approach, Agile is the empowerment of system constructors in business (product business units) with resources and authority not only to develop the concept of using the product provider system and its functional description (as it was in the classic “waterfall” model of interaction between business and IT ), but also independently synthesize its modular structure, as well as develop the application modules themselves (usually in a microservice architecture), and do all this iteratively with quick feedback from clients. This approach has led to improved communication between business and IT when creating products, increased product autonomy, and also to the emergence of a two-level topology of the IT landscape and its constructor systems, defined by Gartner as Bimodal IT:

• Mode 1 “Core IT” is traditional and consistent, focused on security and reliability. Optimized for more predictable and understandable areas. It focuses on using what is known while upgrading a legacy environment to fit the digital world. Mainly supported by monolithic applications developed by the respective IT departments and deployed as part of an enterprise-wide release cycle.
• Mode 2 “Agile IT” is exploratory and non-linear, providing product flexibility and Time-To- Market speed, experimenting to solve new problems and optimized for areas of uncertainty. Mostly applications in microservice architecture developed by cross-functional Agile

From the point of view of the structure of the landscape, “Core IT”, due to inter-level conflicts, over time acquires the features of “platform” – monolithic core applications provide microservice applications of “Agile IT” with their behavior through a layer of standardized programming interfaces (API). In “Agile IT” a platform is also being formed into which infrastructure and business services common to all applications (authentication, logging, generation and recognition of documents, etc.)

Unfortunately, due to such “intertwining” of business and IT, the discussion of new business initiatives often occurs directly based on constructive/modular system descriptions in terms of IT solutions (applications and their integration), bypassing the functional breakdown of the product provider system. Designing a system without a clear understanding of the target process of its functioning leads to a large number of suboptimal architectural solutions.

An important factor in the increasing complexity of the IT landscape in Agile is also the growth in the number of IT landscape applications that support business product systems developed by independent teams, as a consequence of Conway’s law – “organizations, who design systems, are constrained to produce designs which are copies of the communication structures of these organizations.”

It is obvious that with all the advantages of Agile, in the absence of appropriate oversight (governance), an increase in the number of application constructor systems that are also created/changed at different rhythms leads to further complication and even some chaos of the IT landscape system.

Figure 6. Agile and Core ITs. Platforming.

Returning to the concepts of a systems approach, we can confidently say that the IT landscape has all the features of a system of systems (SoS) according to Maier’s criteria:

• Applications, which are subsystems of the IT landscape, are created by independent system constructors (agile teams and IT departments).
• Applications, especially when it comes to COTS software, can function independently outside the context of the supersystem of the IT landscape of a particular enterprise.
• Applications interacting within the framework of an IT solution manifest their emergence in the form of provided services for provider systems of products.
• The development of the IT landscape is ongoing through the implementation of IT solutions for business, or IT modernization projects aimed at reducing architectural debt.
• The IT landscape of a modern enterprise, as a rule, is geo-distributed in order to ensure its disaster tolerance, and also as a result of the active use of “cloud” services.

Depending on the level of maturity, size, and style of IT management in an organization, IT landscapes can be encountered that belong to the entire range of SoS types according to the ISO 21839:2019 classification:

• Directed – in the presence of the Chief Architect Office with sufficiently broad powers, including the disposal of project team resources for the implementation of architectural tasks and tasks for closing architecture debt. This IT landscape is highly structured with minimal architecture debt, but with a relatively low rate of change due to strict regulations. It is observed mainly in medium and small companies.
• Acknowledged – in the presence of the Chief Architect Office, which can communicate effectively with application development project teams in order to introduce an architectural approach into their practices. The presence of the Architecture Board as a management body ensures the governance of the IT landscape. It is characterized by a rather low structuredness and, at the same time, high connectivity of applications. The level of architecture debt can be quite high. Observed among large companies.
• Collaborative – with a small number of project teams that own applications. Typical for small companies. Architecture debt is not tracked.
• Virtual – in the absence of IT management as such, the IT landscape in our understanding is absent and is mainly considered only as an IT infrastructure (servers, storages, and network channels).

Thus, we make sure that the IT landscape in large companies is a system of systems (SoS) with all its inherent properties.

Zühlke is a global innovation service provider that empowers ideas and creates new business models by developing services and products based on new technologies – from initial vision through development, deployment, production, and beyond.