System Logging on the IBM i (as400): An Introduction

As a company that works hard to protect your data, we get a lot of questions.  One topic that we frequently get asked about is system logging on the IBM i. 

System logging on the IBM i is different from logging on other platforms. In an interview with Patrick Townsend, Founder & CEO of Townsend Security, he reviewed what system logging is, and why it is unique for IBM i systems. Below is an excerpt from that conversation.

System logging has become one of the most essential compliance tasks in contemporary corporate IT. Can you give a brief explanation of what logging is and why it is so important?

Sure! All computer systems, including the IBM i (AS/400, iSeries, System i) collect lots of important information about the security state or the operational state of the system as a whole.  We call these system logs and they often include a great deal of information about what is going on in the system.  In a lot of systems, including the IBM i, these logs are created in real-time. 

To give an example, if someone tries to sign into an IBM i and for whatever reason the username or password is invalid, that event is logged in the system log.  This is an important thing to log because if you were to look at this system log and notice several invalid username and password events, you would say, “Hey, our system is being attacked. We need to take action on this now.”  In summary, system logs are just a central repository on the computer system that say what is going on within the system.  This is why they are so important from a security point of view.

Where does security log information live on the IBM i?

Security information lives in many places, which is one of the challenges that IBM i administrators have.  On the IBM i, IBM creates a central repository (QAUDJRN in the IBM i world) for a large number of security events including password and other security events. But QAUDJRN is not the only place to look for this security information.

There is also a system event log file called QHST that has important log-on and log-off information for users.  The operators console (QSYSOPR) collects and tracks important events and messages for security monitoring.  Finally, the IBM i sports a lot of new, web-type services that have their own log collection facilities including WebSphere, Apache, and SSH.  To properly look at all of the security events that are happening on an IBM i, you have to look in several places in a consolidated view to access the security and health of your machine.  This can be a challenge.

So now that you’ve identified the sources of this important system information, how do we format it for log collection and SIEM servers?

Oh, great question! Getting these collection points that I’ve talked about into a usable format is probably the biggest challenge for IBM i customers, because for the most part they are in an internal IBM format that is not recognizable by a typical collection server or a SIEM product that’s doing monitoring and alerting on log data.

IBM does provide a couple of utilities for printing log information. I’ve seen some people try to use these to get the data into a text format, but there are a lot of limitations. First off, it’s not a standard format. Log collection servers miss information when its not in a proper format. Secondly, it’s typically not done in real time. This means you’re not picking up information in a timely fashion and missing threats to your system.

Precisely has tackled this problem for customers. Their solutions can monitor all these sources of log information and generate alerts in real time or schedule customizable reports, and they can also translate the log information from IBM’s proprietary format into industry standard logging formats. Precisely supports a couple of different formats based on existing RFCs in terms of syslog formats or Common Event Format. And they provide the means to forward log information to SIEM consoles.

System logging is also important for meeting compliance regulations too, right?

Absolutely, yes.

If you look at the PCI Data Security Standard, section 10 is focused on collecting logs and analyzing them in a timely fashion to identify alerts. So, certainly in the credit card or payment industry, it’s a formal part of the security standard.

Also, if you drill down into the recommendations for HIPAA in the High Tech Act; or if you’re looking in the financial industry at FFIEC; if you talk to a Sarbanes Oxley auditor; or look at privacy notification laws, they’re all very consistent about requiring proper monitoring of system logs as part of your security strategy. This is understandable considering how threats evolve and what the typical threat looks like.

Sometimes bad software can reside within a system undetected for months at a time because nobody’s monitoring logs. That software can be phoning home through an IP connection and checking in with remote servers, and your log should be showing that activity. If you’re monitoring logs, you will catch these activities, but unfortunately in a lot of cases, that’s not happening.

That’s why you’ll find across the board in almost every compliance standard, a requirement to collect logs in a central location on a log collection server or SIEM product and have someone paying attention to those logs. When I say “someone,” that is often automated software, such as SIEM solutions that monitor these events trying to detect anomalies or threats.

So what does one need to look for when selecting and deploying a logging solution?

There are several key points when looking for a logging solution, especially on the IBM i platform.

1. You want a real-time logging solution. It just won’t cut it to have a system collecting these events once or twice a day and batching them up and sending them off to a log server. You need a real-time solution that’s collecting events as they happen so that your log collection server and your SIEM products can correlate events across servers and “see” what’s happening.
2. On the IBM i platform, performance is very important. Many users are collecting tens of millions of events a day – that’s a lot of events! We’ve seen solutions that just can’t keep up with that kind of volume, which is a risk to security and meeting compliance.
3. Make sure the logging solution is built on industry standard formatting. Industry standards are important – query, reporting and alerting tools depend on these formats.

For more information about IBM i log data, check out Precisely’s eBook: The Ultimate Guide to IBM i Machine Data Analytics