The Ultimate Guide to Mainframe Machine Data: SMF Data & Beyond

There are a number of different data sources that are available within the IBM z/OS® mainframe (mainframe machine data) that can be leveraged to provide insight into the operational health of the system and applications as well as providing visibility into security and compliance issues.

The primary data source is the System Management Facility (SMF) on z/OS, a component which collects and records a large amount of real-time and historical information on performance, security, and technical operations. An abundance of very useful information, providing a wealth of insights can be recorded daily. Virtually every operational event that occurs on the mainframe — from a simple log-in attempt at a particular workstation to a potential breach of system security — is captured and recorded in one or more SMF record types.

There are other data sources available which can also provide valuable insight into performance, availability, health, and security of the system and its underlying applications. With all the data sources and volume of data available on the IBM z/OS mainframe, each use case becomes an exercise in identifying the right set of information needed to meet the needs of the organization. Organizations definitely understand and see the value of incorporating mainframe logs into their analytics processes. Let’s take a look at the different data sources and how they can be used for both operational intelligence and to address security challenges.

SMF data: The mother of all data sources

What is SMF?

The System Management Facility (SMF) is a logging capability provided by the IBM z/OS mainframe operating system to capture detailed information about every activity happening within the system. This includes system-level information, application information, security information and events, transaction information, database information, and virtual anything related to the system’s operating environment. SMF data is probably the biggest single source of operational and security information on the mainframe. But it is also the most complex making it a challenge to extract information from.

What does SMF data look like?

SMF data is very complex! Every aspect of the system operating environment generates a unique record type which is self-describing and can contain thousands of unique fields. Most records typically contain multiple “parts” referred to as subtypes. Understanding how to decompose an SMF record for use typically requires a mainframe subject matter expert. However, once the record has been decomposed fields can be easily named, identified, and used by anyone familiar with an analytics platform like Splunk.

How is SMF data used?

TSMF Data is primarily used to provide an organization with IT Operational Analytics (ITOA) and address requirements related to Security Information and Event Management (SIEM).

Syslog data

What is Syslog?

Syslog on the IBM z/OS mainframe is very similar to system logging facilities on other platforms. System components, applications, and workloads write text-based messages to Syslog to record different events. These can include both normal and abnormal operational issues.

How is Syslog Used?

Analysis of Syslog messages can be used to look for issues that impact the operating environment of the system as well as to detect security issues and threats. Since Syslog is a text-based message logging facility it contains less detailed metrics than what would be provided in an SMF record, however, analysis of the messages within Syslog can still address some of the ITOA and SIEM use cases addressed with SMF data.

UNIX System Services (USS) Files

What is USS?

The UNIX® System Services element of z/OS, sometimes referred to as “z/OS UNIX”, is tightly integrated into the operating system to provide UNIX capabilities within the IBM z/OS operating environment. UNIX System Services allows UNIX applications from other platforms to run on IBM System z mainframes running z/OS. UNIX System Services is a key element of IBM’s open and distributed computing strategy. USS is a certified UNIX operating system implementation optimized for mainframe architecture, and an integral element of z/OS.

What are USS files?

Non-UNIX z/OS workloads utilize standard mainframe datasets. However, z/OS UNIX provides a hierarchical file system (HFS) familiar to UNIX users. In addition to the HFS, the IBM z/OS operating system also provides support for another UNIX filesystem — the z/OS® Distributed File Service (DFS™) zSeries® File System (zFS). Much like HFS, zFS contains files and directories that can be accessed with APIs, as well as be mounted into the z/OS UNIX hierarchy along with other local or remote file systems types such as HFS, TFS, and NFS.

How are USS files used?

USS files can contain information from Java applications, C++ programs, and other UNIX-based applications running within the USS environment on z/OS. Incorporation and analysis of USS files can address a variety of use cases centered around the applications which write and read information from USS files. USS files can be used to investigate application activity, performance, availability and virtually any operational issue related to the application. Data generated by an application into a USS files could also be used for other analytics such as understanding business performance and customer buying patterns.

Use cases based on USS files are virtually unlimited. IBM Security Key Lifecycle Manager (SKLM): Provides centralized and automated encryption key management processes on z/OS including key storage, key services, and key life-cycle management functions. IBM SKLM includes support for IBM and non-IBM storage solutions using the OASIS Key Management Interoperability Protocol (KMIP).

Log4j data

What is Log4j?

Log4j is one of several Java logging frameworks. Log4j is part of the Apache Logging Services project of the Apache Software Foundation and is used by Java applications like web-based components to record events occurring within their Java environment.

How is Log4j used?

Analysis of Log4j messages can be performed to determine if web-applications and other Java-based applications running within the IBM z/OS Mainframe are experiencing problems which are impacting the application’s ability to deliver services in a timely manner. Log4j data can also be used to analyze activity trends within the application to understand peak periods and periods of low activity.

Syslog Daemon (syslogd) data

What is syslogd?

Syslog daemon (syslogd) is a server process that runs in the z/OS UNIX (USS) environment to provide a mechanism for recording log and trace information from USS components and applications. A primary user of syslogd is the z/OS Communications Server. It’s components including TCP/IP server applications, FTP, and Network Security Services(NSS) to name a few, write messages and trace information to syslogd. syslogd is the open-systems platform version of system logging much like Syslog is for the IBM z/OS mainframe.

How is syslogd used?

Analysis of syslogd messages can be used to look for issues that impact the z/OS UNIX operating environment of the system as well as to investigate network related issues and security threats.

SYSOUT data

What is SYSOUT?

In the IBM mainframe environment there is a Job Entry System (JES) which helps to manage the initiation/start and termination/end of each workload. One of the functions of JES is to manage output messages from the z/OS system related to the executing workload. These messages are typically directed to a SYSOUT dataset which resides on a SPOOL file managed by JES. An executing workload can also direct output from an application to SYSOUT. JES uses one or more disk data sets for spooling, which is the process of reading and writing input and output streams on auxiliary storage devices, concurrently with job execution, in a format convenient for later processing or output operations. SPOOL is an acronym that stands for Simultaneous Peripheral Operations Online.

How is SYSOUT used?

SYSOUT output can be used in a variety of manners, however, the predominant use is for an application to direct some specific output to SYSOUT which can be captured and used for analytics. This can be anything from what types of functions or transactions are being performed most frequently, or what specific functions and activities are being used by specific or classes of users.

RMF data

What is RMF?

The Resource Measurement Facility (RMF) is IBM’s strategic product for z/OS performance measurement and management. It collects performance data for the z/OS environment to monitor systems’ performance behavior. RMF data is used to optimally tune and configure the z/OS system.

How is RMF data used?

RMF has different data collectors and reporting mechanisms to address different use cases and requirements. One of the most critical components is RMF Monitor III (RMF III) – a short-term data collector for problem determination, workflow delay monitoring and goal attainment supervision.

This is the information that is a critical z/OS data source for analytics platforms to be able to perform real-time problem determination and resolution for system performance issues and application delays. RMF Monitor I is a long-term data collector for all types of resources and workloads. Data collected by RMF Monitor I is logged to SMF and is mostly used for capacity planning and performance analysis. RMF Monitor I data can analyzed using the SMF type 7x records.

Summary

Splunk®, ServiceNow®, and other analytics platforms make it simple to collect, analyze and act upon the untapped value of the big data generated by technology infrastructures, security systems and business applications — providing the insights to drive operational performance and business results.

They typically collect and index log and machine data from any source and provide powerful search, analysis and visualization capabilities to empower users of all types. The problem is that most of these analytics platforms have no good connectors or mechanisms to get to mainframe logs and data sources.

Precisely Ironstream is the industry’s leading automatic forwarder of IBM z/OS mainframe log data to analytics platforms. With Ironstream, it is easy for Splunk, ServiceNow, and other analytics platforms, to provide visibility into all systems — including the mainframe — from one integrated user interface.

Ironstream enables the analytics platform to provide total visibility into the IBM z/OS mainframe and the applications it supports. With Ironstream there is no need for special knowledge and expertise to correlate mainframe data with that coming from other platforms. Simply use Ironstream to collect z/OS data sources to open your IBM mainframe and enable your organization to address operational and security issues.

To learn more, read our eBook: Understanding SMF Records and Their Value to IT Analytics & Security