Splunk and the Mainframe: 6 Real-World Case Studies for ITOA, ITSI and SIEM

Major Insurer Achieves IT Operational Efficiency with ITOA Solution

Mainframe IT and business IT both get big data benefits

Challenge:
Organizations constantly look to extract more value from the operational data generated within their IT infrastructure, and to analyze that information to determine how their systems and applications are performing.

A primary source of operational intelligence for IBM z/OS mainframe users lies in the SMF (System Management Facility) records.

These are recorded for just about every event and activity on the system. In order to extract such valuable information, organizations typically are saddled with the time-consuming manual processes of offloading the data, extracting the relevant records and fields, and then transforming the remaining subset with expensive tools like SAS.

These are often multiple-day processes that fail to answer certain essential questions, such as: What is happening now? Is what is happening now different than what was happening this same time last week or the same time one month ago? Can I predict or even prevent issues from impacting performance or adherence to service level agreements (SLAs)

Solution:
One major insurance company had been dealing with this challenge in the same way as most other organizations. They were offloading SMF data daily, extracting the required records and fields, then doing post processing using SAS to generate reports on the desired information. As a possible alternative, however, they were intrigued by the concept of ITOA (IT operations analytics), by which their own data could be empowered to let them better understand and ultimately to improve their operations. And so they researched the leading ITOA vendors.

They then began to use Splunk® Enterprise for analytics and visualization of critical IT components. But they were still relying on those antiquated, labor-intensive processes to get z/OS SMF data loaded into Splunk Enterprise.

Discussing the problem with Precisely, and seeing a demonstration of Ironstream, they quickly realized that Ironstream would enable them to process and forward SMF data to the Splunk Enterprise analytics platform in real-time, eliminating the manual process.

Global Financial Firm Controls Costs on IT Operations with ITOA Solution

Enabled by real-time log analytics

Challenge:
Banks and financial services firms have many unique challenges in this digital-mobile-global era, and choosing which to address first is an important skill for their IT leaders. Better control over a multiplicity of costly IT operations was the goal of this multinational financial services corporation. One job, for example, was using 30% more CPU time than was normal; the cause was unclear, and the impact was unacceptable.

The lack of understanding exactly “why” certain things cost so much and took so long to resolve had to be addressed to ensure they remained a leader in this hyper-competitive sector.

More broadly, the customer was spending too much money on additional capacity on demand; the mean time to problem resolution was too long; timely alerts to many problems were lacking; and there was insufficient correlation of significant events picked up by operational and application performance monitors in various business segments. Even more critically, there was no correlation between its IBM z/OS mainframes and distributed systems.

IT operations analytics (ITOA) was the primary use case the company wanted addressed, and that included several capabilities to address their business needs:

• Getting real-time alerts from RMF III and Db2 database access threads.
• Proactively monitoring statistical anomalies relating to mainframe resource consumption.
• Real-time tracking of transaction performance as transaction data moved through multiple stages from distributed to mainframe systems.
• Real-time alerting on mainframe usage and contention issues in batch job performance.
• Real-time monitoring of Db2 Stored procedures for excessive calls.
• Identifying key areas for automation or consolidation.

The Solution:
This began with researching the IT operations analytics (ITOA) marketplace and choosing the clear leader in this big data analytics platform sector serving the high-transaction financial services firms. The customer’s overall platform selection for IT monitoring and analytics, which brings together inputs from its distributed computing environment, was Splunk Enterprise. This platform was ingesting 40 terabytes of log data per day from hundreds of applications around the world.Missing from this insight-generating stream, however, was real time performance log data, or “machine” data, from the mainframe environment. That changed when it began a pilot program for Ironstream, the chosen partner of Splunk for solutions that access mainframe log data in real time. Ironstream is the industry’s premier real time forwarder of critical SMF records and other z/OS operations and applications log data to the Splunk platform. Given its unmatched track record of success in this area, Ironstream was an easy choice.

Luxury Auto Network Opts for IT Service Intelligence to Improve Business Performance

Splunk^® IT Service Intelligence + Ironstream drives excellence

Business application performance that ensures that customer service operations deliver the value that the most discriminating luxury car buyer expects — that was the challenge that this Americas arm for a luxury European auto maker was striving to meet and overcome. Besides the company’s own IT systems, it must also effectively support continual interactions with the disparate IT systems at several hundred associated dealerships nationwide.

For a time, however, the company ’s IT staff was often becoming aware of application problems only after a dealership complained directly to the CIO. (Not good.) The thing that was missing was real time monitoring of key performance indicators (KPIs), especially for its CICS and Db2 apps.

In IT industry parlance, what this challenge called for was an IT service intelligence solution, and the premier industry solution is offered by Splunk Inc. with its revolutionary Splunk IT Service Intelligence™ premium app. So far, so good — at least as far as the various distributed systems were concerned.

Obviously, however, the vitally important CICS and Db2 mainframe performance logs were not part of the distributed systems. They were secured in “the glass house” — the company ’s z/OS mainframe environment. That inner sanctum of enterprise data is normally not easily or cost-effectively accessible from the distributed environment. For this company, getting those mainframe data feeds into the Splunk platform in real time for service-centric diagnostics was crucial.

The company demands the best for both its customers and its dealership networks worldwide. It rightfully expects IT to drive value and competitive advantage — not be the source of issues from that dealer network. They want service to be 100% focused on selling and delivering a world-class experience.

Solution:
For the Splunk sales team, the question of how to get CICS and Db2 log data into Splunk IT Service Intelligence was no question at all. It had to be Ironstream along with the Ironstream Module for Splunk IT Service Intelligence.

Precisely is a Splunk Technology Alliance Partner since 2014, and its Ironstream offering is the industry’s premier mainframe log data access solution, which for years now has been securely streaming mainframe logs into the Splunk platform for Splunk customers. Thus, Ironstream in conjunction with the Splunk IT Service Intelligence application became this customer’s solution.

Results:
Splunk IT Service Intelligence using Ironstream as the SMF data forwarder has enabled the customer to:

• Reduce MTTR (mean-time-to-resolution) of CICS, Db2, and other production issues.
• Attain end-to-end visibility into application health and performance.
• Improve satisfaction among the dealerships in getting timely business information from the distributor.

Insurance Firm is Proactive in Enterprise Security

Splunk^® IT Service Intelligence + Ironstream drives excellence

Challenge:
A large North American insurance company, a national leader in automobile and home insurance, had to eliminate a significant security and compliance exposure in its z/OS environment. The problem was in having only limited visibility into the status of customers’ sensitive personal information when an application was moved across their different production and test environments.

That personal information was well secured in applications running on their production mainframe system. However, to guard against unnecessary exposure, as well as for compliance purposes, the data needed to be scrubbed of the sensitive personal information when an application was sent for testing, etc., on the z/OS test system.

Fortunately the company was already using the industry-leading Splunk® Enterprise data-integration platform to index and analyze operational data coming from devices in its open-systems infrastructure. But it still lacked an easy, cost-effective way to get that kind of operational data into the Splunk platform from its z/OS systems.

Solution:
The company reviewed options, focusing particularly on product compatibility with Splunk and vendor expertise in both mainframe and big data. On completing the review, they chose Ironstream, a data-gathering solution developed by Precisely to supply the missing link between its z/OS systems and the Splunk platform.

Ironstream is the industry leading platform that collects a variety of operational log data from IBM z/OS systems, transforms it securely into an efficient format for operational and big-data usage, and sends it to the Splunk platform — all in real time or near real time.

Information sources in z/OS, such as Syslog, SMF, log4j, and Unix System Services (USS) logs, are easily available for analysis and visualization, and displayable through an ordinary web browser. Also important, Ironstream eliminates the user’s need for increasingly scarce z/OS expertise and costly,
specialized equipment to access operational data on the mainframe.

After satisfactory completion of a proof-of-concept (POC) exercise, the company deployed Ironstream with the Splunk platform. With that, the company became able to gather the required data movement information from SMF records across their various systems, to sift through the data with analytics and similar techniques, and to visualize the results in a variety of contexts, further enhancing security across their enterprise.

• How much data is moving from one system environment to another — i.e., from production to test.
• Which protocols are being used to move data — e.g., F TP, Direct Connect, XMIT, etc.
• How, where, and who are initiating data transfers.
• Whether the inbound data to a system is coming from a production or test environment.
• Whether the data movement is compliant, non compliant, or unknown.
• Whether approved exceptions are enabling potential unauthorized access to secure information.
• Whether data are going through the appropriate “scrubbing” as the are being moved.

In the insurance and financial services sector, enterprise-level security isn’t merely desired, it is mandated by regulation and required for customer trust. Using Splunk Enterprise + Ironstream, this leader in the insurance industry was able to efficiently and seamlessly address its security and compliance concerns.

• Failed authentication attempts.
• Attempts at accessing resources that are denied by the access control mechanism.
• Privileged user actions.
• Activities that require privilege.
• All attempted accesses of security-related resources, whether successful or not.
• Creation or deletion of users.
• Changes to user security information or access rights.
• Changes to system security configuration.
• Changes to system software.
• Attempts at escalation of privileges.
• All log-in activity.
• Password changes.

The agency is now able to audit for unusual activity at the individual user levels, helping them detect security exposures such as:

• Access from an unusual location, unusual network zone, or unusual time of day.
• Changes to user privileges and rights.
• Excessive data transmissions.
• Unusual movement of data.

Ironstream is an important step forward in the data-integration marketplace. It collects z/OS log data from IBM mainframes, transforms it to generic machine language, and forwards it securely to Splunk^® Enterprise, Splunk Enterprise Security, or Splunk Cloud™.

That way, mainframe log data can be merged on the Splunk platform with all other machine data from distributed systems across the enterprise. Users are thus able to have a true 360-degree view of the entire enterprise IT infrastructure. Companies that use Splunk platforms can see what this may mean for them by asking the Precisely rep for the free Ironstream Starter Edition.

The customer began securely forwarding ~20 gigabytes of SMF records per day through Ironstream to the Splunk platform for efficient, real-time monitoring and analysis, a volume that could eventually grow to 800 GB per day when you factor in growth and having other mainframe data sources brought in. With Ironstream’s innovative filtering, though, they can minimize the streamed data to include only those that are relevant to the use case.Among the Results:

• The monitoring of security activity on their mainframe applications in Splunk Enterprise, including log-on attempts, password changes, user access violations, etc., met the audit and compliance thresholds for SOC2 certification.
• The manual processes and related efforts and costs associated with using zSecure were eliminated.
• The SMF forwarding became automated and started being done in real-time.
• They were able to select and forward to Splunk Enterprise only those records related to security and compliance, without having to process the entire SMF record set, significantly reducing the volume of data forwarded and controlling resource consumption. With Ironstream’s industry-best file-type support, low overhead, and innovative filtering, this healthcare leader can easily expand to other use cases while keeping costs to a minimum. They are equipped to garner new insights from their data and can easily adapt to changing needs.