This is the "Redbook" on CICS TS for z/VSE and z/VSE batch performance that I could have written while working for IBM. It is based on what I needed to learn in order to solve customer performance problems both before and during my time in CICS L3 at IBM Hursley. It can also be used with the 21CS VSEn 6.3 and OLTP products.
It is firstly a very detailed reference that covers software and hardware operation and design, and includes material that would be difficult or impossible to find elsewhere. Secondly, it explains an approach to performance analysis and tuning, mostly using tools that I developed or enhanced, including the use of CICS dump and trace analysis for the more complex problems that no z/VSE tool would be able to solve.
It explains in detail how to use and interpret that data output from the tools, especially where there are known or potential accuracy issues. It is designed to complement my CICS TS for z/VSE Health Checker tool by providing in-depth explanations for why its analysis output highlights the issues it does. It also explains how to interpret CICS Health Checker output like the tool's VSAM file analysis, LSR Pool mapping, and the option to identify potential CICS Shared Data Tables.
There are a large number of examples of performance data that show both good and bad performance. I have included MSHP Local APAR fixes to alter some of the CICS behaviour, source code for a DFHTRAP trace exit that produces a CICS System Dump when a specific transaction has a long response time so that the cause can be determined, and source code for a simple long-running CICS monitoring tool that highlights actual or potential issues by using the CICS SPI.
The main topics and first-level subtopics are as shown below.
- zSeries Performance and Capacity Planning Cheat Sheet
- zSeries CPU Design and Performance:
CPU Speed and CPU Time; CPU Operation, CPU State, the PSW and the Interrupt Mechanism; LSPR; SIIS; LCPs and CP Weight; SIE; LPAR CPU time and LPAR Management Time; LPAR Share and Entitlement; Hardware and Software Capping; z/VM and Hiperdispatch
- zSeries Storage:
Storage size; Cache Efficiency; Storage Protection
- zSeries I/O:
Hardware I/O concepts; Record, Block, CI and Buffer concepts; EXCP, CCB and CCWs; A z/VSE DASD I/O Operation; ZHPF; Parallel Access Volumes (PAV)
- The Principles of Mainframe Performance:
Instruction processing CPU time; Storage processing; I/O processing; Other processing; Summing it up
- z/VSE Performance Monitoring Tools:
CICS Monitoring tools; MONVSE; Performance Analysis using CICS Dumps and Trace; z/VSE Debug; A summary of Monitoring Tools and Their Uses; A CICS Health Check
- Performance Problems, Analysis and Tuning:
Customer Performance problems; Analysis Tools; The Importance of Profiling; Tuning; The Top-Down Analysis Approach; CICS Tuning; Batch Tuning
- z/VM Configuration Options and z/VSE Performance:
CPU Time Management; Storage Management; I/O Management; z/VSE DASD I/O queueing in z/VM; MDC; z/VM Virtual disks
- z/VSE Design:
Best Practice for z/VSE Performance; Tuning and Other issues; Workload management; Multitasking and Tasks; z/VSE APIs, SVC and PC instructions; z/VSE CPU time accuracy; The Turbo Dispatcher and Multi-CPU support; other important design concepts like the LTA; z/VSE Virtual Storage; the Address Space; Reducing the size of the Shared-24 Area; System GETVIS management; the z/VSE Partition; Partition GETVIS management; Dynamic Space GETVIS Area sizing; interpreting GETVIS command output; GETVIS fragmentation; Data Spaces; 64-bit storage; z/VSE I/O management detailed design; POWER Files
- VSAM Concepts, Design and Performance:
A VSAM performance Cheat Sheet for CICS and Batch; Important IDCAMS parameters; LISTCAT output; the VSAM API; VSAM Partition GETVIS usage; Lookaside, NSR and LSR buffering; Dataset Name Sharing; file performance issues and tuning
- CICS Design:
CICS Best Practice settings; CICS from the z/VSE design perspective; how programs run in CICS
CICS design based on the order of the Enhanced DFH0STAT output sections:
CICS System Status; CICS Monitoring Facility (CMF); Statistics; Transaction Manager; Dispatcher; Storage Manager; Loader and Programs; Temporary Storage; Transient Data; LSR Pools; Files; Shared Data Tables; CICS Dumps; ISC and IRC; Terminal Statistics; VTAM Statistics; Journals
- Sample Full Trace Analyzer (FTA EXEC) Output
- CICS Health Checker Best Practice Messages
- DMF: DMF Overview; DMF Configuration and Tuning; DMF JCL; DFHDFOU; running DMF Data Analysis Batch Programs; DMF Console Commands
- What I learned while working in a Customer Support Role
- Working with z/VSE Support on complex Performance Problems
- Sample DFHPD430 formatted CICS System Dump Output
- Sample CICS Health Checker Output
- CICS TS for z/VSE 2.2 Usermods
- Source for a DFHTRAP used for a Response Time Problem
- Source for a simple CICS Monitoring Tool
The ZIP file contains a self-evaluation questions and exercises document, plus other presentation material that is relevant.
I hope you find it useful. If you have any questions, comments, suggestions or corrections, email me.
Mike Poil January 2024
This is a simple, but practical, introduction to REXX that was given to colleagues at IBM.
The Full Trace Analyzer (FTA) is a z/VM REXX program that is used to analyze CICS TS for z/VSE FULL trace data produced by DFHPD430 and DFHTU430 , The output is a series of files, where the primary output file contains a summary of the trace activity by task number followed by a CICS system section that provides an overall summary including how busy CICS was at the time and is useful for showing a CICS loop. Other output files show performance information and a summary of API requests by task number. I wrote the code to help me convert massive amounts of trace data into a format that I had a fighting chance of understanding but that was still informative enough to be useful. Please do not ask me if I can write a z/VSE REXX version! If those of you who know REXX and CMS Pipelines find the code to be horrible and confusing, so do I whenever I go back and make changes to it!
This is the last version of the tool that I will make available before retirement.
MONVSE is a z/VSE performance monitor that looks at cpu utilisation and cpu delays at the z/VSE and Partition levels and has been used to help solve real customer z/VSE and CICS performance issues. It writes CSV format output at intervals of 1 second to 1 hour and can warn about excessive Partition cpu utilisation. While it looks superficially like CPUMON, the CSV format output is significantly more comprehensive and useful. However, MONVSE cannot be used to produce the XML format output that can be used for Capacity Planning automation.
The ZIP file contains a LNKEDT BJB job and sample JCL.
The tool was originally called VSEMON, but the name has been changed to avoid a conflict with the Velocity Software, Inc product of the same name.
A batch program to monitor VSE Lock Waits over a period of time.
The source code is available in the zip file.
This is a batch program to help you look at a VSE Lock Wait that is happening right now.
The source code is available in the zip file.
BATCHMON is designed for monitoring and tuning z/VSE batch job steps. It produces a single line of CSV output when a job steps end. By default this is written to SYSLST but the data can be written to a fixed-blocked sequential disk file. It reports on cpu usage and EXCP counts, plus it shows percentage values for active on a cpu, waiting for a cpu, NP code active and the most common task wait states such a ECB (e.g. I/O) wait, POWER wait, Lock wait, plus Partition and Getvis storage usage values. It is not a substitute for Vendor z/VSE batch monitoring software, but it could help z/VSE customers who either do not want to or cannot afford to install Vendor software. It should be easy to install and use because it runs as a normal batch program with absolutely no changes to the z/VSE Operating System and no use of key 0 etc. As always, I appreciate feedback of any kind.
The source code is available in the zip file.
The PDF contains an overview of how to operate a z/VSE system, and was originally designed to help people learn to operate one of the Hursley z/VSE systems.
The ZIP file contains a basic z/VSE JCL course that was originally designed to help z/OS people in the Hursley CICS team learn something about using z/VSE JCL.
The ZIP file contains a presentation that provides an introduction to MSHP and how it is used, in conjunction with the IUI, to apply PTFs to a z/VSE system. It includes examples of the IUI-produced JCL, console output and printed PTF job outputs.
This package contains the documentation that describes the process, plus the STAT REXX program, that I have successfully used to help me perform CICS Health Checks and solve customer CICS TS for z/VSE system performance problems. It includes a case study showing how to interpret the analysis of the included enhanced DFH0STAT output.
Ignoring its abilities to profile the whole CICS workload, individual VSAM file performance, LSR Pool mapping and modelling, CICS Shared Data Table candidate selection and the creation of CSV format data for use in a product like Excel, the tool can highlight well over 150 configuration and actual or potential run-time issues when used with the output of the Enhanced DFH0STAT program that is supplied as a separate download.
More than 10 years of experience of both customer performance analysis and problem determination as the z/VSE SME in the CICS L3 team at Hursley has been incorporated, meaning that it can identify actual or potential problems that no other documentation or ISV product is able to.
The same REXX code runs on z/VM and z/VSE.
This is an updated version of the LVC presentation material first published in 2018.
This is an updated version of the LVC presentation material first published in 2014 on the subject of VSAM performance in CICS.
This presentation provides an overview of DMF and how to configure and tune it.
This is a copy of the z/VSE LVC on CICS TS for z/VSE storage problems that was first presented in 2013.
This is a copy of the z/VSE LVC on analyzing CICS TS for z/VSE SOS problems that was first presented in 2015.
The SIRSMF REXX program uses intercepted SIR SMF commands to create a CSV format print file that contains DASD performance interval data for selected DASD cuu addresses. The minimum interval is 15 seconds and the maximum is 12 hours.
This presentation shows how to quickly extract the diagnostic information from an AP0001 or SR0001 Program Check CICS System Dump that will help you to solve the problem.
This is a copy of the WAVV 2014 presentation on how to debug CICS Application programs using formatted DFHDMPA/DFHDMPB Transaction Dumps.
This document provides an introduction to working with the CICS TS for z/VSE Trace facility and its formatted data.
The enhanced version of DFH0STAT as described by my performance presentations.
Apart from some changes to the report, the main difference is that it is now possible to produce a copy of the report as a z/VSE Sequential File by using an Extrapartition Transient Data queue and running DFH0STAT with a different transaction id.
Please read the included Readme Word document to understand how to install and use this version because there are differences between it and all previous versions.
Due to my impending retirement, I expect this to be the last version of the tool.
This presentation looks at some of the CICS design, especially that of the CICS Dispatcher. It is an updated version of what was presented at GSE Europe 2019 in Leipzig.
This is the presentation for an accompanying video about Storage Violation debugging for CICS Transaction Server for z/VSE.
The video is in two parts:
Part 1
Part 2
A sample of a modified version of DFH$MOLS that converts CICS Monitoring Task Performance data from DMF into CSV format.
Source code is provided.