Sort Gets No Respect but Wields Speed Secrets

Quiz: Name the mainframe computing process that:

• Consumes as much as 20-25% of all CPU cycles but is generally ignored when optimizing mainframe resources
• Is one of the most-covered topics in introductory computer science classes, but tends to be ignored once school is out…and
• Includes processing types referred to as injection, histogram, merge, quick, radix, bin, bubble

It’s a computing process that, as Rodney Dangerfield less artfully but pointedly put it, “Can’t get no respect.”

Times up. We’re speaking of sort processing — one of the most widely relied-upon components in software engineering. While you’re refreshing yourself on the difference between a quick sort and an insertion sort, it’s worth considering the first of these points. While solid research on the resource utilization of sort is hard to come by, it’s self-evident that sort algorithms gobble-up resources in modern operating systems, utilities and applications.

Sort as silent partner

Sort often operates behind the scenes as a silent partner. The connection between sort and other functions, such as compression, continues to capture the interest of optimizers. For example, Matt Allington, working on a Microsoft stack with SQL Server and PowerPivot, looked at the benefits of sorting data on load.

An IBM note on improving DB2 for z/OS LOAD performance offers similar tips, e.g., “present data to LOAD in the optimal order: sort the data in cluster order to avoid needing to reorganize it after loading” and “If you are loading a single table that has, at most, one foreign key or one index key, sort the data in key sequence.” And so on.

IBM’s IMS Performance and Tuning Guide Redbook contains a section on multi-streaming batch processes (9.2.10.). The guidance invokes the sort magic dust several times:

“If you use flat files as drivers for your batch process, then ensure that you sort and split the files in the sequence of the database. This means that if you are PHIDAM, then you would sort and split the files using the Partition Selection Exit or high keys as specified in the RECON in ascending sequence. If you are using DEDBs, then you would sort and split the files in randomizer sequence” (p. 178).

Back in 2010, Willie Favero writing in Enterprise Systems Media introduced the topic of Db2 sorting by admitting that “sort invokes many thoughts and emotions. And “most in the ‘necessary evil’ category,” he quipped. Like it or not, Favero advised that in “SQL, sort is one of those things you do whatever you can to avoid, though that’s an almost impossible task.”

Favero’s focus was Db2’s relational RDS sort, but Db2 sort performance is an issue not lost on sort specialists like Precisely. In Db2, the Db2 Query feature allows Syncsort MFX‘s SORT or COPY operations to directly retrieve data from a Db2 database based on a query specified by an SQL SELECT statement.

BMC Software’s CHECK PLUS for Db2 offers this tuning tip pertaining to memory use: “To improve memory usage, consider the following information: Specifying a value greater than 0 for SORTNUM (with no SORTWK data sets specified in the JCL) allows CHECK PLUS to perform the maximum multitasking of sort processes based on available memory.”

SAS Software users may have noticed that as much as 30% of all SAS processing is consumed by sort processes. That was a level of utilization that Syncsort MFX was designed to alleviate.

Sort is so omnipresent that it’s often taken for granted. User interfaces regularly call for resorting of columnar displays and bar charts whose back end executions usually entail sorting. Compuware’s mainframe tool Topaz, for example, allows users to “filter, sort, export and display recorded events in table form.

Mainframe sort’s impact on performance

The big data era has garnered attention for new products, new startups and new applications, but that’s not the whole story. A survey conducted by Enterprise Systems Media last year suggested that big data would have an impact on mainframe processing, too. Over 89 percent of respondents anticipated that mainframes would play a significant role in their big data processing plans – through CICS, Db2 and WebSphere. Many also see mainframes as an opportunity to provide data to big data platforms such as IDAA, Netezza, Splunk, Spark, Oracle, Teradata and Hadoop.

Precisely’s incarnation of the venerable sort can also benefit from its use of IBM’s zIIP processors, which can crank processing speeds for sort tasks by as much as 90%.

Because of its central, albeit behind-the-scenes importance, research into sort techniques is ongoing. Intel Labs’ Throughput Computing group reported success at increasing radix sort performance by 2.2X on the Intel Core i7 CPU.

In-memory and load-time sort have proven critical to memory-resident business intelligence tools like QlikView that do not require data warehouse planning and deployment. As data volumes swell, mainframe hardware resources capable of both high speed I/O processes and high speed in-memory can produce similar benefits with even more vast amounts of data.

Fast sort will be a quiet, secret enabler.

Exploiting the mainframe’s powerhouse I/O

The current release of Syncsort MFX allows users to optimize on elapsed time or on CPU time. It also leverages IBM’s System z High Performance Fiber Connection (zHPF), the hardware maker’s most comprehensive set of I/O architecture and firmware improvements. As IBM Fellow Harry Yudenfriend explained:

“The zHPF open standard has been adopted by all enterprise-class vendors providing DASD storage for the mainframe. In its essence, zHPF leverages hardware assists provided by host bus adapter vendors for the fiber channel protocol transport while preserving enterprise class qualities of service supported by Fiber Connection (FICON).”

Beginning in late 2011, the zHPF standard on z/OS added its muscle to QSAM, BSAM and BPAM methods. IBM’s DFSORT and merge apps now use BSAM to exploit zHPF. Measurements made by Precisely with its release of Syncsort MFX suggest that using zHPF can cut as much as 15% off elapsed times.

Speed over respect

This explanation gets pretty technical, but the payoff is straightforward and transparent: using a zIIP to sort, with most sort steps taking place on that specialized mainframe-class iron, is typically easy to use.

Keep in mind that the mainframe sort game is far from over. In 2011, Bertasi et. al introduced a fast sort called psort in the Journal of Experimental Algorithmics. And a just-published 2016 article by Matai, Richmond, Lee et al. proposes a framework that generates sorting architectures for different computing needs.

Speed will come. Respect? That’s a concern of lightweights. You’ll garner respect for your glistening new Mainframe DevOps Chief job title instead.

For more information on how to leverage mainframe sort to maximize value, read our white paper: Getting the Most Out of Your Mainframe