Why Integration Must Be Hybrid

Integration is the central piece that ties your enterprise together. The need to integrate has only increased with the proliferation of SaaS applications, IoT sensors and devices, B2B e-commerce supply chains and mobile app development in addition to the vast number of applications that still reside in on-premises data centers which also host mainframe, legacy and custom applications. One of the biggest drivers today is digital transformation—organizations are rapidly trying to become digital enterprises in order to differentiate themselves from the competition and develop new business models to create new revenue channels.

In the meantime, APIs are becoming the standard way of connecting applications, data and devices, and providing services to partners and end users. The API economy is the term used for creating new products as part of an organization’s transformation journey. APIs are also rapidly becoming a new channel for B2B partner interaction. Since APIs and integration are two sides of the same coin, this allows organizations to quickly realize the value of integration to unlock data from different silos, drive consumption of this data and leverage business benefits for the application of this data in various end user and consumer apps.

It’s a Hybrid World

The most common understanding of hybrid is that organizations have applications both on-premises as well as the cloud. Pure cloud integration is a fantasy since most organizations will have applications and systems that have to stay on-premises for a variety of reasons: the cost and risk of redeveloping core capabilities (eg. An ERP application that has been heavily customized for the business or a core banking application that is mission critical), the lack of critical services in the cloud, concerns about security and sunk cost of on-premises applications and data sources (mainframe and legacy applications that just work and will be extremely expensive to rip and replace with a SaaS application).

However, the definition of hybrid is much broader and according to Gartner, such a strategy must address a set of challenges that go across personas, domains, endpoints and deployment models. Lets explain each of these in more detail:

• Personas: Integration development today is not just the purview of IT specialists but extends into ad-hoc developers and business users (often called citizen integrators) who despite not being technical cannot wait for IT to develop integrations for them.
• Domains: Integration has always been a continuum supporting a variety of use cases that go across applications, B2B partners, data (for the purpose of doing analytics) and process (to support automation and orchestration).
• Endpoints: The sources and targets for integration started originally with on-premises, then expanded to the cloud along with the need for mobile connectivity and more recently has exploded with IoT with the proliferation of sensors and devices.
• Deployment Models: This covers the more traditional definition of hybrid and includes on-premises, cloud and hybrid (cloud and on-premises) but is now expanding into a new style called embedded, using a headless deployment model which is required for SaaS applications that need underlying integrations to satisfy their end user requirements.

The Hybrid Integration Platform

A true hybrid integration architecture makes integration available across all environments, for all the personas, overlapping all the domains and connecting all the endpoints. This will span on-premises ERM, CRM and mainframe, cloud and SaaS environments, edge devices and sensors and B2B partners–we call this integrating edge to app.

A key architectural principle in hybrid integration is to allow integrations to be done close to where the applications are in order to reduce latency and increase performance. From an application connectivity perspective, hybrid requires applications hosted on-premises, public cloud and private cloud to all be connected seamlessly so an integration developer does not worry about where they are and only needs to easily wire together the mapping to move data from one application to another.

From a B2B perspective, hybrid requires data from B2B partners (customer and suppliers) to easily be identified, exchanged and processed within the integration layer. From an IoT perspective, hybrid requires that data from sensors and devices either directly or aggregated after analysis is integrated with back-end applications. Finally, all the integrations need to be exposed via an API layer for internal business units and external customers to consume the data in a secure and privileged way through mobile or custom apps.

These requirements have created the need for a hybrid integration platform (HIP) that supports hybrid deployment across on-premises and cloud, democratizes integration development across IT and business users, is versatile enough to support all integration patterns and is extensive enough to integrate everything.

Components of a HIP

A lot of organizations originally solved the integration problem by adopting enterprise service bus (ESBs) and integration platforms. These are still widely in use and in many cases powering mission critical applications for organizations. These integration products solved the application to application (A2A) problem and in conjunction with message oriented middleware (MOM) provided capabilities to decouple publishers and subscribers and allow new applications to be easily added.

However, ESBs are undergoing an evolution since they are considered a monolith and organizations are seeing the need to deploy light weight integrations as microservices. This has led to the emergence of a microservices runtime (MSR) that can be scaled very quickly to meet the needs of the business. MSRs are typically deployed using Docker containers and scale using a container orchestration standard such as Kubernetes and are subsequently monitored using open source tools, such as Prometheus.

A big requirement for deploying microservices is the need for DevOps which provides the discipline to manage the dev/build/test/deploy process using automated techniques. Many of these organizations also adopted B2B gateways to extend the reach of integration to their trading partners for the purpose of exchanging e-commerce documents such as purchase orders and invoices via standards, such as EDI and XML.

A lot of transactions still occur using files leading to the need for a managed file transfer (MFT) offering to securely manage file interactions. These in conjunction with adapters and connectors provide reach to a wide variety of applications and data sources. A newer component is API management, which allows existing integrations to be exposed as APIs for access to a variety of consumers while providing policy based enforcement and security of the underlying services. APIs needs end-to-end lifecycle management and governance to control the creation to retirement process and also drives the need for asset management to track dependencies and relationships and provide impact analysis.

IoT integration is becoming a common use case. An IoT platform along with edge capabilities will allow data from sensors and devices to be easily integrated and analyzed for the purpose of moving to other systems or to take action. In-memory is a key embedded capability for ensuring high performance and scale.

Finally, iPaaS is a pure cloud offering which enables integration in the cloud but in conjunction with an on-premises integration platform, or agent, it can ensure data is seamlessly exchanged between cloud and on-premises with end-to-end visibility and monitoring across the different environments. Here is a quick summary of the components:

• ESB or integration platform (with broad connectivity capabilities)
• Microservices and DevOps
• API management and governance
• B2B and MFT
• Messaging
• In-memory
• IoT platform
• iPaaS

A Practical Approach to Hybrid Integration

A HIP can be extremely powerful but involves a number of components which can also make it overwhelming to tackle all at once. A practical approach is to add components (using a building blocks approach) as the need arises to address various integration use cases.

Most customers already have an ESB or integration platform, so that is usually the starting point for integrating on-premises applications. Once an organization starts adopting SaaS apps then there is a decision to be made whether to simply do cloud integration using the on-premises ESB or to start adopting an iPaaS. A critical factor in this decision process is to follow the general guideline and allow integrations to be done close to the applications. The decision matrix for such a scenario can be described as follows:

• If the use case is mostly cloud to on-premises and most of the applications are still on-premises (80/20 rule) then this can be done within the ESB by adding cloud connectivity options.
• If the use case involves a number of cloud-to-cloud integration needs then it may be worthwhile to invest in an iPaaS for efficiency reasons (there is no need to go from cloud to ground and then back to cloud).
• If the application mix gets switched to mostly being SaaS apps then an iPaaS should be adopted since most integrations will be done in the cloud anyway.

Once you bring together an ESB and iPaaS you are in a great position to start leveraging API management and add B2B capabilities to round out most of your integration needs and start deriving value from these integrations.

Business Benefits

Hybrid integration provides a way for customers to move ahead with critical projects and strategic transformation by exploring more flexible solutions. Not long ago, integration had been painted as a back-office capability with little direct impact on the business. However, integration needs have moved to the front-office and addressing these requirements are key for successful digital transformation initiatives.

The business benefits are as follows:

• Organizations are seeing more demand from customers for a better customer experience. Customers have many choices, the best experience wins the business.
• Businesses need to react faster to disruptive changes by increasing the delivery cadence and deploy integrations faster.
• The technology world is experiencing many disruptions. Businesses need flexible architectures to disrupt instead of being disrupted. They need to be able to implement new projects, scale rapidly to take advantage of short-term opportunities and try new fast-fail ideas to leapfrog the competition.
• Gain benefits of the cloud. There is constant pressure to reduce TCO, use an opex model instead of capex and be able to access and leverage all data available from new IoT and SaaS sources.
• Democratize integrations. IT departments are at breaking point and can no longer meet the needs of the business which means business analysts need to develop integrations on their own in a self-service way, which allows more people to contribute to the company’s goals.

Real World Customer Case Studies

Retail Pharmacy: Our first case study is a large global enterprise with more than half a million employees and hundreds of distribution centers. One of the biggest changes in the company’s history was the merger of two iconic brands. The company needed a global platform strategy to stay competitive and remain ahead of Amazon and other digital disruptors in the retail space, and coined their version of a HIP: Independent Federation Layer.

Software AG provided a solution to standardize global operations—a “cook once, serve many” solution. Rather than buy more tools, the company would invest in a unified set of software capabilities using webMethods HIP to integrate all systems. They set up a 200-person integration center of excellence. All retail stores are being migrated onto webMethods for a mission-critical retail transformation program to simplify and standardize sales and inventory processing. The next phase will be standardized processes for warehouse and order management, replenishment management, financial management and more. With webMethods HIP, this retail pharmacy has the agility to react to market challenges and opportunities.

Transport Company: Our second case study is a multinational public transport company that operates bus, train and tram services in Europe, Canada and the U.S. They needed to lift and shift services to the cloud in order to create a new business offering that could scale easily. They refactored their webMethods integration code base and moved key services to the webMethods integration cloud to enable customers to purchase tickets using mobile and web platforms via third-party websites. They provided secure access to on-premises databases by exposing APIs to the cloud using webMethods API gateway. In the process, they created a reusable ticketing platform that they’re offering to other transportation providers. They now have a new line of business.