From Your Mainframe to the Cloud in Three Minutes: Here’s How

Modern cloud data platforms are gaining rapidly in popularity, and for very good reasons. Enterprises have long recognized the benefits of having a robust data warehouse for advanced analytics, but there are some distinct limitations to a traditional on-premises approach. For many organizations, it has been a challenge to build a comprehensive data analytics platform leveraging data from across the enterprise due to system disparity. Integrating data from systems such as IBM i, mainframes, relational databases, flat files,  and more is not an easy task. Effectively bringing together all these data points is  especially challenging as analytics increasingly shifts to the cloud.

Data warehouses tend to be relatively capital-intensive compared to the latest generation of cloud data platforms like Snowflake, Databricks, Amazon Redshift, and others. That problem is exacerbated by the fact that analytics workloads tend to require intensive processing power, but only for very short periods of time. When peak workloads are only in effect 10% of the time, processing power is underutilized the other 90% of the time. With a cloud data platform such as Snowflake, enterprises can pay for access to computing power only when they need it, rather than building out heavy duty on-premises data warehouses that are underutilized.

It can also be difficult to scale up traditional on-prem warehouses quickly and easily. Cloud data platforms, in contrast, allow for rapid provisioning of additional computing resources as needed. Scaling back, likewise, is a simple proposition. Companies can pay for what they use when they need it and can stop paying when they no longer require the additional capacity.

By consolidating information from multiple sources to a cloud data platform, different lines of business within an organization can run their own analytics against the same dataset, without ever getting in each other’s way. That allows for a more complete and consistent view of the organization’s data.

As companies seek to achieve greater business agility in everything they do, cloud data platforms have quickly emerged as an obvious choice over traditional on-premises data warehouses for many enterprises.

Challenges of mainframe and IBM i to the cloud

Mainframe and IBM i systems continue to serve as a critical backbone for many organizations throughout the world. Unfortunately, the data formats used by those systems are not readily consumable by data warehouses or cloud platforms. Binary data formats, REDEFINEs in data fields, and complex record structures simply don’t translate easily to newer non-mainframe systems.

Replicating data between mainframes and on-prem data warehouses can also be slow. This is especially true if the organization is relying upon batch mode integration that updates the destination data warehouse only on infrequent intervals, such as once or twice a day.

As cloud data platforms have gained in popularity, this challenge has grown even greater. In many companies, complex integration architectures have been developed over the years and were built around a set of tools and principles that don’t fully align with web services and cloud technologies.

Real-time mainframe and IBM i replication to the  cloud made simple

With Connect CDC from Precisely, replicating mainframe and IBM i data to cloud data platforms is simple, efficient, and reliable. From Connect CDC, an authorized user can connect to virtually any data source, including mainframe, IBM i, traditional ETL sources, flat files, or relational database management systems. That source data can be mapped to a destination within Snowflake and saved as a data flow that performs a one-time copy process, replicates data from the source to the destination in real time, or synchronizes data between the two data sources.

From the Connect CDC portal, the process is simple:

1. Configure your source data connection by entering relevant information such as IP address, ODBC/JDBC connection information, username(s) and password(s). Click “test” to verify that Connect CDC is able to communicate with the data source.
2. Configure your target data connection (for example, a connection to Snowflake), and test it.
3. Create a new data flow by specifying a source, a destination, and a replication type (one-time copy, one-way replication, or two-way synchronization).
4. Select the source data and destination by choosing the tables to be replicated from your source and the destination data warehouse within Snowflake.

The entire process can take as little as three minutes. Connect CDC also allows for filtering and configuring the records to be replicated based on user-defined criteria.

Connect CDC provides a complete management console to monitor data flows and highlights issues that require action from an administrator. Connect CDC supports integration to Snowflake on AWS, Google Cloud Platform, or Microsoft Azure.

Benefits of integrating IBM i and mainframe data to Snowflake

As previously noted, one of the key benefits of a robust cloud data platform such as Snowflake is that it allows multiple lines of business across an enterprise to replicate or synchronize data to the same centralized data store, resulting in a single consistent view across the entire organization. In this respect, Snowflake allows companies to eliminate silos and provide users with near real-time reporting of critical business information.

In enterprises that rely on mainframe and IBM i systems for critical business operations, this elimination of “the silo effect” is especially welcome, as business leaders in such organizations often struggle to make sense of the disjointed view of operations that inevitably results from such information silos.

Connect CDC breaks down those barriers, making some of the organization’s highest value data available within a broader data analytics context. EBCDIC encoded records are automatically transformed and made available for use in Snowflake. Connect CDC is an enterprise-grade platform, with the flexibility and scalability to reliably handle tens of thousands of messages per second.

With Connect CDC, businesses have the ability to stream data to a cloud data platform from across the enterprise. With the ability to quickly add new data sources and targets, Connect CDC provides enterprises with the agility to develop and manage analytics quickly and confidently.

To learn how easy and cost efficient it can be to get mainframe and IBM i data into Snowflake’s cloud data platform – in 3 minutes or less – watch our on-demand webcast, Get Mainframe and IBM i Data to Snowflake.