In this blog we provide you an overview of three system integration approaches and point out the benefits and drawbacks of the approaches. Integrating systems directly, also known as Point to Point, is compared with using a dedicated integration platform to connect systems. We also introduce an integration approach known as API-Led or a 3-tier API architecture.
Integration Patterns
Three high-level application integration patterns have emerged over the past few decades. These three patterns hold true if we are talking about Web Services, REST, Database Connectivity, File Transfers, or Messaging systems like Rabbit or Kafka.
The patterns are Direct Point-to-Point Connections, using an ESB or an Integration Platform as an intermediary, and most recently, 3-Tier API Led. Let’s discuss what this means.
Direct Connections between Systems (Point-to-Point)
This approach involves inserting code into a system, initiating the integration by calling an API, accessing a database, producing an import/export file, or publishing an event. On top of this connectivity effort, things like data transformation, exception handling, and handling multithreading to provide scalability needs to be considered and built into the integration.
This effort needs to be duplicated in the integrated target system. This system implements the API that is called, reads the file created, or consumes the published event. Once again, data transformation, exception handling, and scalability need to be built into the receiving end of the integration. Moreover, security is also a concern for the integration of the system on the receiving side.
Direct Connectivity
Direct Connectivity Over Time
When a pattern of direct connectivity between systems is used to integrate applications, it results in an exponential growth in the number of interconnections that need to be built and maintained individually.
Pros of Direct Connection
- The fastest to implement if the integration is simple
- No learning curve outside of the knowledge of your existing Applications
- Works well for File Exports and Imports
- It may work when calling simple REST APIs
Cons of Direct Connection
- Very hard for developers not familiar with the current systems to find or update the integration code
- New integrations are done by copying/pasting existing code (if it can be found)
- Data transformation is cumbersome in a procedural language
- Complex scaffolding code is required to manage connections and security
- Scalability is usually very limited (single threads, small result sets)
Point-to-Point Flows in an Integration Platform as a Service
This approach involves creating a single flow in an iPaaS to act as a pipe between an object or data source inside the system initiating the integration and an object in the target system. This moves aspects like system connectors, data transformation, process orchestration, threading, and exception handling into an integration platform designed specifically to solve these problems. However, implementing the end-to-end connectivity in a single flow results in initiator authentication, target system security, data transformation, and process orchestration combined in a single deployable component. This makes reuse nearly impossible to achieve in a realistic scenario. Teams using this overall pattern in ESB and SOA projects were more productive and efficient than teams directly connecting applications together, but integration service and API reuse continued below.
Point-to-Point Flows
Point-to-Point Flows Over Time
When a pattern of building individual flows to handle connectivity between two systems is used to integrate applications, it results in an exponential growth in the number of interconnections that need to be built and maintained individually. The complexity is the same as directly connecting applications, albeit with the advantage that all the integration complexity is located in the same integration platform instead of being scattered across multiple applications.
Pros of Point-to-Point Flows
- Tools are designed for Integration use cases
- iPaaS platforms have drag and drop to speed up development
- Code written is limited to Transformation Logic
- No boilerplate scaffolding code required
- Faster for medium and high complexity requirements
- Easier for future developers to find and update integration code
Cons of Point-to-Point Flows
- The potential learning curve to get producing in iPaas
- This pattern is essentially implementing Point to Point inside a tool. The iPaaS is simply hiding this fact
- Reuse is very hard to accomplish because Integrations are hardwired to connect specific systems together.
API-Led Connectivity
API-Led connectivity is an approach to integrating systems by building and reusing APIs. A fundamental tenet is to design the APIs following a specific pattern that groups the APIs into up to 3 types of layers. Generally speaking, all APIs in the design are implemented as a group of synchronous or asynchronous REST APIs. However, this pattern can also support database triggers, events, or scheduled batch processes. API-Led Connectivity has been proven to have higher reuse levels than creating a single API that is implemented as a direct connection between two systems, as described in the Point-to-Point Flow section above.
System Layer: The System API Layer is the simplest to understand but maybe the most complex to implement. A system API (SAPI) is designed to expose back-end applications and simplify connectivity, authentication, exception handling, retry mechanisms, and data translation from a native format (XML, CSV, Binary, SQL Resultset etc.) to JSON (or perhaps someday another common format).
Process Layer: The Process API Layer is responsible for implementing processor data orchestration logic. In the case of a business transaction that involves coordinating updates to multiple objects, the transaction would be managed at the Process Layer. Any business rules for formulas should also be implemented at the process layer. For an API that calculates a tax value based on the cost of goods, the price should come from the system layer, and the execution of the formula ( for example, Tax = 5% of the cost) should occur in a process API (PAPI). The Process layer can be omitted in simple scenarios that don’t have business orchestration, transaction management, or custom business logic.
Experience Layer: The Experience Layer acts as a proxy designed to give the initiating system the easiest and lowest friction API to access Process or Systems APIs that sit lower in the architecture. Its authentication method should sync with the system initiating the integration that will be calling it, and its data format should also match the initiator. This avoids custom-coding data transformation and translation inside the consuming application. Although reuse is a goal of API-Led, the reuse is not expected to be found at the Experience Layer. Each system that initiates an integration will generally have its own Experience Layer API (XAPI).
One way of thinking about API-Led is that the integration team focuses on building reusable API products that expose systems and processes instead of building bespoke pipes that directly connect systems together.
3 Tier API-Led (Building a layered set of Composable APIs that work together to integrate systems)
If no Orchestration is required, omit the Process Layer
Over time, the API-Led approach results in an “API Network” forming, which results in the reuse of the system and process layers.
Pros of API-Led Connectivity
- The focus is on exposing IT Products, not building Pipes
- Microservice based architecture
- Supports Domain-Driven Design
- Orchestration, Transformation, Security, Data Access, Event Processing, Connectivity details are separated into discrete layers.
- Reuse is a proven outcome
- Easy to find the APIs in a Portal
- Easy to secure the APIs
Cons of API-Led Connectivity
- It can take longer to implement, especially when you are new to API-Led
- More computing resources get used (especially with CloudHub)
- More network hops can increase latency
API-Led Case Study
Organizing APIs into multiple standard tiers will allow developers to find and reuse the created APIs. Separating the Experience Layer, the Process Layer, and the System Layer will enable IT teams to move from building bespoke Pipes between systems to building reusable APIs that multiple systems can use over time.
Have questions on how to use these systems integration approaches in your business architecture to solve your challenges? Wish to know more about how we are accelerating digital transformation with integration as a service? Lets connect to lead your next digital transformation project.
