Trending Topics in Data Integrity

Today, most companies have made the shift toward thinking about data as a strategic asset and an essential tool for making data-driven decisions. Nearly two-thirds of data leaders as noted in the IDC Spotlight report and are expected to make data-driven decisions, yet only about a quarter of respondents stated that they trust the data they’re using. Significant gaps exist between expectations placed on the use of data and organizations’ capacity to trust and use data in delivering data-driven business outcomes.

IDC’s recent spotlight report, Improving Data Integrity and Trust Through Transparency and Enrichment, describes data integrity in a nutshell: Trust comes through transparency, provided by intelligence about data, backed by enrichment to provide context and deliver integrity.

Trust in data is not unlike trust in anything else, such as a product, a person, or a company; people tend to trust those things which exhibit integrity. That requires transparency which, in turn, requires data intelligence. Data integrity provides the foundation for fast, confident decisions that drive your organization forward.

To build data integrity, organizations must understand where the data is and what it means from both a technical and business-value perspective. They need to know their level of data quality, who is using the data, and how they are using it. Data enrichment and location intelligence add context, further enhancing data integrity.

Closing the Data Integrity Gap

Most companies still aren’t delivering when it comes to data integrity. The IDC spotlight report revealed that the people who are closest to the data – including operations, sales and marketing, and human resources – tend to have the lowest levels of trust in their company’s data assets. Those who rely on those assets to drive critical business decisions, including the CEO at most companies, trust it the most.

Another interesting finding was that trust in data tends to be highest among the people who are directly responsible for creating and maintaining it. As those assets are made available to other operational personnel across the organization, though, confidence in data integrity is diminished.

To close these data integrity gaps, companies must take a proactive approach to this problem. Stewart Bond, IDC’s Research Director for Data Integration and Data Intelligence Software points out that most companies aren’t very good at collecting and managing data intelligence. Metadata management software is one of the last pieces of technology implemented at many organizations.

The pressure is mounting to solve this problem. The IDC spotlight report outlines several key factors driving the push towards greater data integrity:

1. Artifical Intelligence is Gaining Momentum

Artificial intelligence, powered by machine learning models, relies on trusted data to train algorithms accurately and reliably. The risk of getting it wrong is high because, unlike general business analytics, AI/ML does not ultimately benefit from routine human review and oversight. In other words, when automated decisions take the place of human judgment, there is substantially less likelihood that a person familiar with the data will be looking at what’s happening with a critical eye.

As organizations move forward with AI/ML initiatives, they are keenly aware of this issue. Consequently, many are scrambling to build data integrity into everything they do.

AI technology also benefits from contextual richness. The more information an AI algorithm has, the better equipped it will be to discover meaningful patterns. Data enrichment and location intelligence are therefore becoming much more important across the board.

2. ESG Reporting Is Big

ESG reporting is another important trend that has been accelerating over the past year. ESG stands for Environmental, Social, and Governance reporting, and although it is currently voluntary, many predict that ESG reporting will become a de facto requirement for most companies. Recent decisions by regulatory and standards bodies have affirmed that the drive toward universal ESG reporting is very much on the ascent.

ESG is accelerating the need for greater transparency. Accreditation is a key component of the ESG model, so companies will be held to high standards for the integrity of their data with respect to environment and sustainability, in particular.

For complex organizations especially, ESG will necessitate stronger data integration. The IDC spotlight report found survey respondents placed a high priority on data ingestion and consolidation. These ranked as the second most important capabilities that organizations must focus on within the next three years.

3. Cloud Adoption

Enterprises have been steadily shifting more and more of their data to the cloud, due primarily to the advantages of scalability and elasticity that cloud computing offers. That offers opportunities to improve data integrity; at the same time, it’s driving the need for greater trust.

Cloud data platforms and cloud analytics are improving data democratization efforts at many companies. That creates opportunity, but it also underscores the critical importance of data integrity, as more and more users are leveraging the power of business analytics with increasingly large and diverse data sets.

Putting data in the cloud also means paying greater attention to data sovereignty and associated compliance challenges. This further highlights the need for strong data governance and metadata management.

4. DataOps and Data Observability

DataOps borrows from the lean and agile principles of DevOps, including the continuous development, testing, and deployment of data applications. It also adds statistical process control to continuously test data and prevent poor data quality from impacting the organization. According to IDC’s Stewart Bond, organizations that have implemented DataOps deliver projects faster and provide higher levels of data integrity than those who have not.

Data observability is closely related to DataOps insofar as it provides for a kind of continuous quality control for an organization’ data assets. Data observability continuously monitors and tests data pipelines to look for operational, behavioral, and measurable anomalies. That drives alerts to analytics engineers, notifying them of potential issues so they can address data issues as soon as they emerge.

5. Driving Business Results

As data takes on a more important role in driving business results, leaders are seeing the value in data integrity initiatives. Respondents in IDC’s survey said that when they have invested in higher levels of data integrity, those efforts are having positive impacts on their key business metrics. The greatest improvements were in customer satisfaction, where respondents indicated an average increase of 42%. Industry innovation, time to market, and employee retention also showed dramatic improvement. Not surprisingly, overall profitability also featured high on the list of benefits.

These are some of the trends highlighted in IDC’s spotlight report, Improving Data Integrity and Trust Through Transparency and Enrichment. To learn more, download the free report today.