Why a Streaming-First Approach to Digital Modernization Matters

Data is one of the most valuable assets in most modern organizations. Whether you’re a financial services company using data to combat financial crime, a transportation company seeking to minimize your climate impact, or a manufacturing business aiming to optimize your supply chain, digital modernization provides critical insights and visibility, unlocking the information critical to your success.

In today’s world, data comes from diverse sources, in different types and formats, and at varying speeds. Most traditional infrastructures were designed in an era when batch processing was the norm. Those systems are ill-suited to keep pace with businesses that need to ingest and analyze data in real time.

How can an organization enable flexible digital modernization that brings together information from multiple data sources, while still maintaining trust in the integrity of that data? Today’s world calls for a streaming-first approach.

The Long Road from Batch to Real-Time

Traditional “extract, transform, load” (ETL) systems were built under certain constraints, stemming from the cost of technology and implementation resources, as well as the inherent limits of computational power. It simply wasn’t practical to adopt an approach in which all of an organization’s data would be made available in one central location, for all-purpose business analytics.

Two decades ago, powerful business intelligence systems relied on on-premise data warehouses, usually fed by overnight batch ETL processes that took multiple hours to complete. To speed analytics, data scientists implemented pre-processing functions to aggregate, sort, and manage the most important elements of the data.

Nevertheless, the result was always yesterday’s news. Current transactions could only be incorporated into business analytics following another overnight ETL process. Business users could choose between powerful analytics built on large data sets or rudimentary reports that reflected what was happening on the ground in real time. The idea of having both attributes – powerful and comprehensive analytics combined with real-time visibility – was simply not realistic for most companies, and in many cases, was not possible at all.

Today, cloud data platforms like Snowflake, Databricks, Amazon Redshift, and others have changed the game. They offer an avenue to sharing information with a variety of users across the enterprise, unifying data from a wide range of sources under one roof and making it fully accessible in real time – without dragging down the performance of operational systems.

What Is a “Streaming First” Approach?

A streaming-first architecture is built to scale to any data consumption that an organization might have, making data available at whatever speed is required for the task at hand. For banks, that might mean streaming transactional data to an analytics platform for real-time anomaly detection. By promptly identifying transactions that fall outside of normal patterns, financial institutions can intervene in potentially fraudulent situations, preventing further losses. That same data may be used to notify account holders and/or trigger credit card replacement processes to ensure customers can get back up and running as quickly as possible.

Many of those financial institutions process transactions on mainframe systems, which can be costly to operate. By offloading data in real time for fraud detection analytics, they can avoid overburdening mainframe databases, while also making it easier to interface with external systems that drive mobile alerts to cardholders and perform similar functions.

That same streaming-first architecture supports a host of other use cases for the same financial institution. Compliance reporting and “know your customer” (KYC) analysis, for example, is a read-only process that should not consume mainframe resources and is far better suited to the streaming-first model. Marketing departments, likewise, can perform analytics more efficiently and effectively using cloud-based tools that offer data enrichment, location intelligence, and demographic analysis.

Benefits of a Streaming-First Approach

In addition to the operational advantages cited above, the streaming-first approach simplifies IT architectures by providing a unified framework for gathering, normalizing, and accessing data. While old-school approaches often called for bespoke point-to-point integration, the modern version offers a single solution to address virtually any use case imaginable.

With a publish-subscribe architecture, various enterprise applications can make real-time data available, and other applications and platforms can consume the information as needed. A bank’s fraud department may use its database of credit card transactions, for example, to detect potential patterns of abuse, or the marketing department may use the database to understand consumer purchasing patterns and identify opportunities to upsell customers to new financial services products. Neither scenario requires a bespoke point-to-point integration, nor do they require the tedious maintenance and testing that such architectures normally imply.

In another example, Precisely worked with a state government struggling to make information readily available to citizens. Court documents and case dockets were stored on a mainframe system, where they were inaccessible to the public at large. Precisely helped court officials to implement a streaming data pipeline to replicate that information to a cloud data platform, where it was available for web developers to publish online. As a result, citizens were able to quickly and easily access important court documents, while the system had no adverse effects whatsoever on mainframe resources or performance.

Enterprises should implement streaming-first architectures with specific business objectives in mind. Ashwin Ramachandran, Senior Director of Product Management at Precisely, notes that the company’s most successful customers are those who enter into the process with a well-defined set of value-creating objectives. Once those goals have been attained, the technology can be applied to a broader set of use cases, creating even more value for the company over the course of time.

Ramachandran also notes that a streaming-first approach is well-suited for cloud analytics deployments. A “lift and shift” approach generally misses the mark, though. To truly benefit from the cloud, you should plan on leveraging its key advantages, including scalability and elasticity.

The digital modernization and streaming-first approach has powerful implications for organizations that operate mainframe systems or on-premise business applications. To learn more, read our free ebook, Streaming Legacy Data for Real-Time Insights.