5 Best Practices for Data Integration Techniques

Compared to two decades ago, when data integration techniques were a more straightforward operation for most organizations, data integration has become the connective tissue that holds the modern IT environment together. In the past it was a bit like a small-town road system, with few connection points, no traffic jams, and few road hazards with which to be concerned. 

The IT environment of today looks very different. On one level, the enterprise is simply dealing with more requirements for data integration. Companies have deployed multiple ERP systems, CRM, marketing automation, point-of-sale systems, mobile applications, and much more. The small-town road system of yesterday has been replaced by a network of highways and interchanges that need to accommodate higher traffic volumes, with substantially more sophisticated security and reliability mechanisms.

It is easy to see how data integration has developed into a key competency for business success. Here are some data integration techniques and best practices for planning your roadmap.

1. Begin with the end in mind

Anthony Scriffignano, Chief Data Scientist at Dun & Bradstreet, advises: “Never lead with a data set; lead with a question.” Every data integration initiative should begin by establishing clear objectives for the project. Is the organization trying to optimize operations and gain efficiencies? Are you aiming to better understand your customers and prospects to gain a competitive advantage? Does the business need to modernize its landscape and be better prepared to adopt new technologies?

Well-executed data integration projects can yield tangible results. What results are you targeting, and how will you measure them? There are many reasons to embark on a project to improve data integration. It is vitally important that IT leaders be clear about their reasons for doing so.

2. Decide which data sources to include

Decisions regarding which data to include should be driven by the business case.

Traditional mainframe and IBM i systems continue to play a key role in the operations of most large enterprises and even many small and mid-sized companies. These systems hold core transaction data that is essential to most data integration initiatives. 

Next, companies should look at the disparate software systems throughout their landscape and identify the role that data from each of those systems should play in meeting the objectives laid out in the business case.

Finally, look beyond the boundaries of the organization itself to determine whether and how you will incorporate data from external sources. This could include third-party data for analytics (such as consumer location data and traffic analysis that can be used to drive site selection decisions for a retail store).

3. Minimize complexities in data integration techniques

Data transformation is fraught with complexity. For example, mainframe data can be especially challenging due to some of the anomalies associated with variable length records and COBOL copybooks. To make matters worse, it is getting more and more difficult to hire people who have the skills required to make sense of these complexities as well as the skills needed to handle newer technologies.

Enterprise-grade integration tools can simplify the process by managing complex data types behind the scenes. IT can focus its attention on the important matters of designing and validating the integration roadmap, instead of mastering the arcane details of mainframe data.

4. Determine data communication methods

There are numerous things to consider when determining how data will be communicated. Real-time integration is the gold standard and the one for which most organizations aim. Batch-mode integration can address a number of scenarios adequately and can even be built to meet an “almost real-time” standard. It is important to consider both current and future volumes of data to assess whether pipeline capacity will be adequate to handle the traffic.

Ultimately, the communication method you choose will depend on your objectives. Where do you want the data to live, and what do you plan to do with it? Again, it comes back to the business case. If your objective is to consolidate information from multiple sources into a big data platform for analysis, then your strategy will look very different than if you are pushing changes on a customer record from a legacy system to a marketing automation system. 

5. Consider future needs

Today, businesses have access to a range of new technologies that were unknown five to ten years ago. We have seen an explosion in the amount of data available. Innovation has followed suit, as businesses have discovered new ways of using new technologies and data to increase efficiency and drive competitive advantage. 

In today’s world, data integration is generally not a ”once-and-done” project. As the organization evolves, as new technologies emerge, and as new sources of information become available, your data integration techniques should be periodically reassessed and realigned to serve business objectives.

Keeping that in mind, it is important that companies establish a strategy for robust, adaptable tools and frameworks that can support their organizations as new technologies emerge, as organizational needs evolve, and as new opportunities come to light.

For more information, get our checklist of the 10 key features when hiring a data integration vendor. Click to read The Data Integration Top 10.