Shortening z/OS Recycling Time

IBM System z server workloads are comprised of heterogeneous transaction types. DB2 queries, WAS (Websphere Application Server), Websphere MQ (Message Queueing oriented middleware), CICS TS, CTG (CICS Transaction Gateway), RDZ (Rational Developer for System z), IMS (Information Management System) transactions. Together with builtin TCP/IP stack and Unix Systems Services, all of the facilities are served to enterprise ITC infrastructure through a lot of daemons.

Workloads are organized as address spaces. Address spaces are 64-bit addressable program and data containers. Initialization of server operating system consists of initialization of all address spaces one after the other.

Standalone zOS operating system instances have a few minutes of downtime per year. Clustering as Parallel Sysplex (Systems Complex) makes them available continuously. Although continuous operations is a daily routine for System z shops, scheduled recycling is necessary for release upgrades or testing initialization procedures for disaster recovery facilities.

Large shops have automation tools with initialization, termination and recycling procedures. Recycling may be an operational burden for shops without such tools. There are two contradictory considerations in recycling live systems. First one is not to cause loss-of-data and keep data integrity and second one is to keep outage as short as possible.

Termination consists of shutting down each address space one after the other. Next question is which address space to shut down first? There are “client” address spaces and “server” address spaces. It is logical to think to close down client address spaces first. Then server address spaces become idle and it becomes possible to terminate them normally without causing data loss.

Most important single group of address spaces to terminate normally is database related address spaces. Most probably DB2 and related address spaces. So address spaces using services of DB2 should be terminated earlier. Those are so-called middleware address spaces like CICS TS, Websphere MQ, CTG, WAS. Among address spaces in a group terminated concurrently, again consider terminating in-group clients like CTG and Websphere MQ earlier than server in this case CICS TS.

DB2 not terminating normally? I am keeping the advice “Do not mention MODE(FORCE) termination option of DB2, it may have side effects and create problems..” of my colleague DB2 Expert! I am just summarizing his recommendations:

  1. Wait for “DSNYASCP ‘STO DB2’ NORMAL COMPLETION” message. Depending on number of active transactions and sizes of buffer pools, normal termination may take some minutes.
  2. If you do not receive normal completion display active threads and utilities using -DISPLAY THREAD(*) command. Wait for utilities to complete.
  3. Watch for batch, CICS protected threads, WAS and RRSAF connections. Either wait for completion or cancel using -CANCEL THREAD command.
  4. Terminate DDF using -STOP DDF command and wait for normal completion.
  5. If you sweat, keep cool and go to item number 1

For further information related with DB2, please refer to databaseguidelines.blogspot.com

Other groups of address spaces to terminate are as follows:

  1. Monitoring tools address spaces – Address spaces for monitoring tools like BMC MainView or Omegamon can be terminated after terminating DB2 address spaces.
  2. Printer related address spaces – They should be terminated before terminating communications server address spaces.
  3. Communications server address spaces – They are TCP/IP and VTAM. TCP/IP and related address spaces like TN3270E should be terminated before. Because OSA (Open system adapter) interface devices are defined to VTAM.
  4. SMS (System Management Storage) related address spaces – They are HSM (Hierarchical Storage Manager) and RMM (Removable Media Manager, cartridge management system) address spaces. They can be terminated any time provided that space management and cartridge I/O activities do not take place.

Printer signoff, inactivate, disconnect, line drain commands may flood system consoles for installations without console message automation. If this is the case, following messages can be suppressed by MPF (Message Processing Facility):

  1. $HASP097 LINExxxx IS DRAINED
  2. $HASP203 RMTxxxx DISCONNECTED FROM LNExxxx
  3. $HASP210 SESSION RMTxxxx LOGGED OFF LNExxxx
  4. $HASP879 RMT(xxxx) DEVTYPE=LUTYPE1,STATUS=INACTIVE/LNExxxx
  5. $HASP880 LINExxxx UNIT=SNA,STATUS=ACTIVE,DISCON=NO

zOS consoles are 3270 terminals. Model 2 terminals have 19 lines for messages, model 4 terminals have 39 lines for messages. Consoles are designated to keep messages on screen at least for 1/4 seconds. So in one second at most 4 * 39 = 156 messages can be displayed. Termination messages for thousands of printers flood console screens for long time not to mention several messages for one printer and related line. During flood termination of other other address spaces may not be observed.

To wrap up, following procedure may be used for termination:

z/OS Shutdown Procedure

Date: ________________

Start Time: ________________

Command Description
Start a TSO session, a monitoring tool session and an HMC (Hardware Management Console) session Display SDSF screenDisplay DB2 threads screen
STOP CTG720MODIFY CICSxxxx,CEMT PERFORM SHUTDOWN IMMEDIATESTOP CHCPROC Terminate CTGTerminate CICS regionsTerminate IBM Infosphere CDC (Change Data Capture)

Terminate other DB2 client address spaces

-STOP DB2 Wait for normal completion
DISPLAY A,L See all CICS and DB2 address spaces are terminated
Terminate monitoring tool address spaces Logoff monitoring tool
SET MPF=(00,PR) Issue MPF command to suppress printer termination messages provided that MPFLST00 is standard MPF PARMLIB member and MPFLSTPR is the member prepared to suppress printer messages
START AOPSTOPSTOP APIJPJCL Terminate Infoprint Server
STOP TN3270ESTOP TCPIP Logoff TSO (Time Sharing Option) session
STOP TSO Stop TSO address space
HALT NET,QUICK Terminate VTAM
HALT EOD Make sure that SMF records are written
$P JES2,ABEND Reply like xxEND

End Time: ________________

Initials: ________________

Command Abbreviations

D DISPLAY

F MODIFY

S START

P STOP

T SET

Z HALT

-CAN -CANCEL

-DIS -DISPLAY

-STA -START

-STO -STOP

Advertisements
This entry was posted in IBM zEnterprise Servers and tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s