IT integration has two major forms: application integration and data integration. To understand the two and what they mean for your business, it is necessary to learn about their history, the technologies they are based on, what each one is meant to accomplish, and their future.
Origins and underlying technologies
Data integration is the combination of information residing in two different locations in order to provide the user with a unified view of the two formerly separate data sets.
It typically deals with data that is batch-oriented and at-rest, meaning that the processes that created the data have already been completed. With the growing demand for data, integration is often performed during data warehousing through a process known as “extract, transform, and load” (ETL). In data warehousing, large amounts of data are extracted, converted, and uploaded from a variety of sources into a single, integrated stream for review, analysis, and archiving. It is not necessary for data integration to be conducted in real time, although enterprise IT’s insatiable appetite for large amounts of data (okay, let’s call it Big Data) has significantly increased demand for real-time integration. Data integration focuses on the large-scale movements of large, multiple-source data sets. Therefore, it involves considerable data delays.
Application integration, a relative newcomer to IT, deals with integrating live, operational data between two or more applications in real-time and is mainly used to connect the disparate systems that make up an enterprise. It’s trigger- and event-based and involves the invocation of an integration flow that updates and enriches data in other applications. It generally involves more complex processes and challenges than data integration and can become even more complicated when integrating on-premise, legacy systems with newer, cloud-based services. Application integration is based behind the firewall, emphasizing logic and work. It works better with continuously interacting systems that are limited in their data movement capabilities, and its transformations are often less sophisticated than those of data integration.
Which integration approach to use
If you’re asking which integration option is most effective for you and your business, you’ll first need to ask yourself what you’re trying to accomplish. Each type of integration is meant for solving different problems.
The goal of data integration is to synchronize data among multiple data sources in order to standardize quality and consistency of data across a database when traveling between applications. It is used for the large-batch data analysis of files and databases. ETL, the primary data integration technology, is entity-oriented and is used for making business integrations, resulting in better decision-making.
The goal of application integration is to ensure the reliable and timely exchange of messages among applications rather than data sources. Application integration directly links different apps at a functional level, transforming and transporting data from one application to another. It is very much a real-time sharing of information between online transaction processing (OLTP) systems and requires some sort of enterprise application integration (EAI) middleware that enables two or more different applications or databases to communicate quickly and easily with one another. In short, application integration actively manages the flow of information between applications, often via an application programming interface (API). EAI, the most common application integration tool, is generally process-oriented and is used for IT. EAI deals with transactions, not entities, within the process.
Why is it important to know the difference between data and application integration? When evaluating integration platforms — especially platforms for on-premises and cloud-based services integration — it’s easy to forget that each data integration approach can require different sets of platforms and methodologies to accomplish what they do.
Data integration will always be a part of complete integration. There will always be a demand for ETL, data cleansing, and MDM; however, application integration is the only way to deal with the changing requirements of integration. Capturing data as part of the transaction means it can be acted on immediately, whether that involved synchronization with other applications, corrective action, or escalation.
Despite the differences in their goals and what they do, data and application integration have similarities. They both involve open APIs, metadata, transformation, utilized connectors, SOA, and they both provide business intelligence and reporting.
The idea of integration convergence
Should application and data integration co-exist? Gartner Group conducted a survey of 329 companies to determine what integration should look like from a departmental perspective. 47% of respondents planned to evolve both application and data competencies into a single or integrated team. Just 13% expected their application and data teams to remain independent of one another. The same assimilation that occurred between the IT and telecom departments in the 2000s is now going on in the integration field.
In fact, the old rules of what delineates application integration and data integration no longer apply. There are common requests for both types of information. The modern-day boundary-less enterprise can no longer establish what’s on-premise and off-premise, since data is being moved around with a much greater frequency from locations. Data now needs to be transformed before it can be inserted and updated. This requires the use of metadata (data about data), which both application and data integration share. Data warehousing, which was traditionally employed through data integration-based ETL technology, can now occur at remote locations, off-site, and in the cloud. In application integration, the real-time transaction flow has business processes on- and off-premise, and the data needs to be referenced in real-time.
EAI, message brokers, message queues, ESBs
ETL , MDM
Business process automation
Limited in data movement
Amounts of Data
Large to enormous
Better decision making
Agility, real-time, event-driven
Live, in motion, event-based
Data at rest, bulk movement
Single view of customer
Data entry once
Intelligence and Reporting
In a future blog post, we will talk about how the silos of application integration and data integration have paved the way for hybrid integration.