At a high level, application integration means leveraging technical infrastructure to make diverse applications communicate with each other to achieve a business objective. The integration needs to be seamless and reliable, regardless of platform and geographical location of these applications.
The move toward service-oriented architecture, business process management, and software as a service has accelerated the recognition that application integration can increase business efficiency. To ensure that the integration is both beneficial and feasible, one should closely examine the business processes that are being supported before focusing on the systems and technologies themselves. Understanding this is the key to determining how to select the most suitable integration technologies.
This article examines the application integration challenges while considering the enterprise diverse technology landscape and architectural concerns. The presented “scenario-driven approach” describes how to successfully implement application integration standards at the enterprise level, leveraging the TOGAF methodology, and, ultimately, supporting the business capabilities.
CHALLENGES OF APPLICATION INTEGRATION
Making suboptimal or incorrect choices when selecting your technology tool set can lead to complex architectural issues, which in turn lead to tightly coupled systems and support and maintenance problems. Moreover, guidelines and best practices espoused by architecture groups are not consistently written down, nor are they consistently followed by application development teams at large. Without clear standards on when to use which technology, and for what purpose, one runs the risk of creating an unnecessarily complex technology environment.
Many enterprises have also suffered from organic growth and now have every integration technology of the last two decades in place: JDBC/ODBC, database links, remote method invocation (RMI), enterprise Java beans (EJB), Web services, Java message services (JMS), MQSeries, and flat files abound. This technology diversity has resulted in increased support costs and has adversely affected system performance.
Additionally, there are organizational barriers that may impact your application integration efforts as well. These issues typically arise from the fact that enterprise IT systems span multiple departments in the organization. Different development teams in the enterprise, if not properly guided or monitored, may tend to choose the path of least resistance or resort to technologies that they are familiar with, which can add to integration complexity.
Organizational issues in application integration can be tricky in a larger organization. However, these companies often have the greatest need for an effective application integration environment. Therefore, it is particularly important to clearly define standards, so that development groups can focus on business needs.
LEVELS OF APPLICATION INTEGRATION
Application integration can occur at many different levels of a system’s architecture, including the data layer, the application layer, the service layer, and the presentation layer.
Integrating applications at the data layer can sometimes be the quickest implementation approach, due to the simplicity and power of data layer integration technologies. Data layer integration includes the use of database links, shared database catalogs, and direct database queries. However, this approach leaves many internal application details exposed, and upstream and downstream application changes result in significant impact, rework, and testing.
With application style integration, low-level implementation details may or may not be exposed or accessible from other systems. Application layer integration includes the use of flat files, message queues, and remote procedure calls. An “API” of some level is leveraged. Although this approach gives a better degree of separation, there are still problems with this approach.
Message-based technologies such as JMS and MQSeries are examples of popular queuing solutions but are based on proprietary implementations. Error handling can be problematic as messages can be lost on “undeliverable mail” queues. The level of application details that is needed to expose for interfacing applications is one of the major disadvantages of integration using remote procedure calls (RMI and RPC style of integration). In short, integrating applications at the application layer is preferable to data layer integration, but still has its own issues.
Service-based integration includes the use of Web services. The advantage of Web services integration is interoperability, even though the integration can still be point to point. Each Web service client still has the responsibility of knowing which service to call, so the addition of new endpoints will result in additional coding.
An enterprise service bus (ESB) combines the strengths of existing service-based integration technologies but provides more abstraction and interoperability. Application integration using an ESB combines message-oriented processing and Web services, which is the foundation for an event-driven SOA. In our opinion, services-based integration, especially when paired with an ESB, is the preferred method for all application integration in an enterprise.
NOW WHAT?
To get your application integration under control, we recommend that you work within your existing landscape of systems and technologies and leverage the TOGAF methodology, instead of kicking off a large “EAI project.”
We use a “scenario-driven approach” that focuses on aligning business scenarios with supporting technologies. This approach consists of the following steps:
- Leverage TOGAF architecture principles.
- Identify technology standards and create building blocks.
- Identify usage scenarios and map scenarios to the technology standards.
Leverage TOGAF Architecture Principles
The main reason to integrate applications is to support a business process. Taking a technology-first approach can lead to inflexible solutions that can be costlier to maintain as the business environment changes.
Architecture groups do not typically create technology standards in a vacuum for idealistic purposes (although some development teams may have differing opinions). When developing standards, you need to make sure there is direct traceability to enterprise architecture principles such as:
- Creating loosely coupled interfaces.
- Setting platform independent, open standards.
- Developing reusable, shared services.
- Minimizing application impact.
- Promoting data consistency.
- Recognizing that business logic is a strategic business asset and should not be placed in closed vendor solutions.
These principles are based on the default TOGAF enterprise architecture principles; however, they may be too vague for some project teams and leave much room for interpretation. To address this vagueness, the architecture group needs to identify acceptable integration technologies and map usage scenarios to those technologies to control the complexity of the integration environment.
Identify Technology Standards and Create Building Blocks
Based on organizational strategy, solutions in place, and staffing and skill levels, you should then identify the technology standards that are preferred at your company. Determine if your strategy is to be a custom Java shop, a Microsoft shop, or an SAP enterprise. Regardless of the technology, set clear standards for technology usage and avoid allowing developers to choose their favorite technology to use for integration.
TOGAF promotes the use of building blocks to support effective enterprise architecture. Building blocks are simply a package of functionality defined to meet business needs. For example, an architecture building block is a high-level, abstract architectural pattern. A solution building block is a specific instance of a technology or product. Every organization must decide for itself what arrangements of building blocks work best for it.
For each architecture building block, create the corresponding and solution building block(s). For example, ETL is an architecture building block; it can be realized by a solution building block consisting of products such as Informatica PowerCenter or Business Objects Data Integrator.
A well-specified catalog of building blocks will lead to improvements in application integration, facilitate interoperability, control technical diversity, and provide flexibility in the creation of new systems and applications. Table 1 lists a sample of the approved architecture and corresponding solution building blocks.
Map Scenarios to Standards
As a final step, map the integration technologies to various usage scenarios. You will need to develop usage scenarios that can be used to identify types of application interaction.
Our set of usage scenarios includes:
- Perform domain entity validation
- Synchronize business information in multiple systems.
- Notify of state change between applications.
- Notify other systems of event of interest.
Notice how our usage scenarios are not technology focused—the scenario is not “put a message on an application queue”; the scenario is “Notify other systems of event of interest.”
We use a matrix to map the preferred integration standards to various usage scenarios at various layers, i.e. data, business logic, and presentation. The goal of this matrix is to enable consistent architecture between system interfaces by establishing a common set of prescribed integration and usage patterns.
For example, if a usage scenario is to synchronize information in multiple systems, the preferred integration standard would be to use message bus or ESB as suggested by the matrix. Any deviations from preferred integration techniques would require an exception process to be followed.
The preferred technology for a particular usage scenario is indicated on the matrix with a + sign; a – sign indicates that this technology is acceptable but not preferred. Also, some technologies such as FTP and DB links are not included, meaning that they are not permitted and their usage would require an exception approval.
The matrix acts as a guide to architecture and solution delivery teams. Basing decisions on this matrix minimizes the decision variability across teams and provides a base for service orientation that ensures interoperability and true integration in a heterogeneous enterprise landscape.
CONCLUSION
The key to successful services-based integration is to focus on why applications need to exchange information. Our scenario-driven approach enables the application teams to use the matrix as a guide to identify approved integration technologies based on business requirements. This approach helps to control technical diversity as well as fosters consistent integration standards across the enterprise.
