SMP/E HOLDERROR Processing in IBM System z Software Management

Recent discussions with some colleagues showed me that this is an area which requires some clarification.

SMP/E (System Management Program/Extended) is the tool for performing maintenance on z/OS systems and System z components like DB/2, CICS, WAS etc. Program products are organized as FMIDs (Function Modifier IDentifier). Modifications are organized as PTFs (Program Temporary Fix). PTFs are corrective code updates for defective programs and organized as PUT (Program Update Tape). They are not delivered in cartridges anymore but name is not changed anyway. PTFs are primarily identified like PUT1302 which means belonging to year 2013 second month PUT tape.

Although PTFs are prepared to fix some erroneous code, some of them are erroneous themselves. They are called PEs (PTF in Error). PEs are marked in accompanied documentation called HOLDDATA. New superseeding PTFs are prepared and included in more recent PUTs.

As time passes some PTFs are observed as correct and included in a more reliable categorization RSU (Recommended Service Upgrade). RSU is also identified as year and month format like RSU1304 meaning RSU PTFs belonging to fourth month of 2013.

As an example, suppose that UK12344 and UK12355 are PTFs in PUT1303. In PUT1304, UK12344 is marked as PE and superseded by a new PTF UK12366. Again suppose that PTFs UK12355 and UK12366 are added to RSU1305. If you APPLY RSU PTFs instead of PUTs, it is less likely that you apply PEs.

Let me get back to HOLDDATA. I mentioned that HOLDDATA keeps information related with ERROR PTFs. HOLDDATA also keeps information related with other requirements of application of PTFs. For example a system RESTART may be required after application of a PTF, some command, parameter, message or documentation CHANGE may be introduced.

Before APPLYing PTFs, we always run an APPLY CHECK and mention BYPASS(HOLDSYSTEM, HOLDERROR)

http://www.redbooks.ibm.com/abstracts/tips0106.html?Open

With these parameters we mention “I noted SYSTEM HOLDS, go ahead and pretend apply PTFs with system holds” and “I noted ERROR PTFs, go ahead and pretend apply PEs”. Since this is a CHECK run, we observe any difficulties available in application of PTFs. Otherwise we receive GIM30206E error messages and return code 8 which may hide some other errors.

When it comes to real APPLY never use BYPASS(HOLDERROR) as mentioned in above documentation unless you have a good reason to apply a PTF in error. The reason may be PE would be related with a component you do not use – like cryptography for example – and PE may be prerequisite for another PTF you have to apply.

Majority of IBM System z customers do not apply PUT PTFs. They apply RSU PTFs, so that they would like to minimize the risk of applying error PTFs and introduce defects in their systems.

This may not be a frequently confronted problem as PEs are not included in RSUs most of the time. Do not forget that mentioning BYPASS(HOLDSYSTEM) means that BYPASS HOLDs and APPLY PTFs with system HOLDs and mentioning BYPASS(HOLDERROR) means that BYPASS ERRORS and APPLY PEs.

Advertisements
Posted in IBM zEnterprise Servers | Tagged , , , , , , , , , , , , , , , , , , | Leave a comment

Increasing Number of CHPIDs (Channel Path Identifiers) for LCUs (Logical Control Units)

The channel subsystem (CSS) and the IBM z/OS operating system need to know what hardware resources are available in the computer system and how these resources are connected. This information is called hardware configuration. Hardware Configuration Definition (HCD) provides an interactive interface that allows you to define the hardware configuration for both a processor’s channel subsystems and the operating system running on the processor (1).

A channel is the piece of hardware with logic in the CPC (Central Processing Complex) and to which you connect a cable in order to communicate with an outboard device. FICON channels are used in recent configurations. Channel path identifiers (CHPIDs) are the 2-byte identifiers for channels. You use a CHPID to identify a channel to the hardware and software in the HCD. Although the two terms are often used interchangeably, we refer to attaching a control unit to a channel and using the CHPID in a z/OS CONFIG command to identify the channel you want to bring online or offline (2).

256 devices are supported for CU (Control units). LCU definitions are used to circumvent this limitation. Multiple CUs are defined for each block of 256 devices. Most zOS shops use a set of 8 channels for all LCUs. As long as channel utilization is below 90% percent, there is no problem and these definitions can be used forever. If channel utilization closes maximum steadily, then some channel definitions should be added in hardware configuration.

Following is the view of control unit definition panel in Hardware Configuration Definition:

sekil05

As we cannot define more than 8 channels per LCU, we define channels in a circular way. Suppose we have channels A1, A2, A3, A4, A5, A6, A7 and A8. Further suppose that existing LCU definitions are as follows:

sekil10

We will add 4 more channels like B1, B2, B3 and B4. We will add CHPIDs in a circular fashion to have channels distributed evenly. New definition would be as follows:

sekil20

Since number of LCUs may not be the multiple of channels, usage of some channels would be one less then others. In our case use counts of channels A5, A6, A7, A8, B1, B2, B3 and B4 will be one less than others.

After performing modifications in a new work file, create a new production I/O definition file. You can write I/O configuration to one of IOCDS locations in the processor complex.

The problem may start when you try to activate I/O configuration dynamically. In a complex I/O configuration environment, you could receive system abend 878 with reason code 10:

IEA705I ERROR DURING GETMAIN SYS CODE = 878-10 IBMUSER IKJACCNT IKJACCNT

You increase TSO address space region to a point where you may not get more virtual storage and you may not activate I/O configuration dynamically:

IEF085I REGION UNAVAILABLE, ERROR CODE=20

IEF187I IBMUSER FAILED – SYSTEM ERROR IN INITIATOR

IEF450I IBMUSER IKJACCNT IKJACCNT – ABEND=S822 U0000 REASON=00000014

Even if you use ACTIVATE console command to test or activate configuration dynamically, you may not do so due to below messages:

sekil30

But these messages reminds you channel path and device associations not used any more. You have to VARY following paths OFFLINE using console commands similar to VARY PATH(C100-C1FF,A5),OFFLINE command:

sekil40

After varying unused paths, you test and activate new configuration successfully. Next thing is to keep an eye on RMF (Resource Management Facility) or BMC Mainview CMF Monitor reports (whichever you use) to observe a better-balanced I/O distribution.

___

Notes:

(1) z/OS Hardware Configuration Definition User’s Guide

(2) I/O Configuration Using z/OS HCD and HCM

Posted in IBM zEnterprise Servers | Tagged , , , , , , , , , , , , , , , , | Leave a comment

Suppressing CICS TD (Transient Data) Messages

Some CICS TS (Transaction Server) system messages written to TD queues tend to flood zOS JES spool. For example:

DFHSN1400 07/19/2012 17:04:32 CICSPACK Session signon for session J1 by user CTGRUS is complete.

DFHSN1500 07/19/2012 17:04:32 CICSPACK Session signoff for session J1 is complete. 1 transactions entered with 0 errors.

Each CTG (CICS Transaction Gateway) transaction causes two messages to display, first one is for signon, the other is for signoff. In a high utilized environment, thousands of messages printed on spool in minutes.
I observe that some colleagues either recycle CICS TS systems or set MSGUSR DD statement to DUMMY. Unscheduled recycles cause downtime. Setting spool listings to DUMMY may cause to miss some important messages.
CICS TS message domain exit XMEOUT may be a solution for this and similar situations. This exit allows you to suppress or reroute CICS TS and CPSM (CICSPlex System Manager) messages that use the message domain.
Product ID (“DFH” for CICS TS, “EYU” for CPSM), two-character domain name and message number parameters are passed to exit and return code 4 is used for suppressing the message.
Following code can be inserted in the sample exit in CICS TS sample library, member USREXIT1:

010500 *
010600 **********************************************************
010700 * <<<<<< SECTION TO BE MODIFIED BY THE USERS.     >>>>>> *
010800 *                        START.                          *
010900 **********************************************************
011000          L     R3,UEPCPID
011100          CLC   0(3,R3),=C’DFH’     CHECK PRODUCT ID
011200          BNE   RCNORMAL            IF NOT DFH, THEN RET CODE NORMAL
011400 **********************************************************
011500 * CTG SIGNON-SIGNOFF MESSAGE SUPPRESSION MODIFICATION
011600 **********************************************************
011700          L     R3,UEPCPDOM         LOAD DOMAIN CODE ADDRESS
011800 *
011900          CLC   0(2,R3),=C’SN’      IS IT SIGNON DOMAIN?
012000 *
012100          BNE USR_SIGNON_DOMAIN_END  NO, BRANCH TO END OF CHECK
012200 *
012300 * SIGNON DOMAIN MESSAGES
012400 *
012500          L     R3,UEPCPNUM         LOAD MESSAGE NUMBER ADDRESS
012600 *
012700 * CHECK DFHSN1400 SESSION SIGNON COMPLETE.. MESSAGE
012800 *
012900          CLC   0(4,R3),=F’1400′    IS IT SIGNON MESSAGE?
013000 *
013100          BE    RCBYPASS            YES, BRANCH TO SUPPRESS
013200 *
013300 * CHECK DFHSN1500 SESSION SIGNOFF COMPLETE.. MESSAGE
013400 *
013500          CLC   0(4,R3),=F’1500′    IS IT SIGNOFF MESSAGE?
013600 *
013700          BE    RCBYPASS            YES, BRANCH TO SUPPRESS
013800 *
013900 USR_SIGNON_DOMAIN_END DS 0H
014000 **********************************************************
014100 * CTG SIGNON-SIGNOFF MESSAGE SUPPRESSION MODIFICATION END
014200 **********************************************************
014300          L     R3,UEPCPNUM
014400          CLC   0(4,R3),=F’8320′    CHECK MESSAGE NUMBER
014500          BNE   RCNORMAL            IF NOT 8320, THEN RET CODE NORMAL
014600 *
014700          L     R3,UEPCPDOM
014800          CLC   0(2,R3),=C’DX’      CHECK DOMAIN
014900          BE    RCBYPASS            IF DX DOMAIN, THEN RET CODE BYPASS
015100 **********************************************************
015200 *                         END.                           *
015300 * <<<<<< SECTION TO BE MODIFIED BY THE USERS.     >>>>>> *
015400 **********************************************************
015500 *

Give a proper name to to-be exit, assemble and link it to a library in DFHRPL chain though I recommend you to prepare an SMPE USERMOD. Define exit program to CICS system using your conveinent resource definition method:

CEMT INQUIRE PROGRAM(‘USREXIT1’)

I PROG(USREXIT1)
STATUS:  RESULTS – OVERTYPE TO MODIFY
Prog(USREXIT1) Leng(0000000144) Ass Pro Ena Pri     Ced
Res(001) Use(0000000001) Any Uex Ful Qua Cic               Nat

You can use CICS TS ENABLE and DISABLE SPI (System Programming Interface) commands to initiate and terminate XMEOUT exit respectively. CICS TS command level interpreter transaction CECI is a convenient way to issue CICS TS API (Application Programming Interface) and SPI commands:

CECI ENABLE PROGRAM(‘pgmname’) EXIT(‘XMEOUT’) START

CECI DISABLE PROGRAM(‘pgmname’) EXIT(‘XMEOUT’) STOP

One last thing may be to start the exit when CICS TS region is started. Any command level program inserted in PLTPI (Program List Table Post Initialization) table will do that. That program will issue ENABLE command. PLTPI program will execute at startup and exit will start at startup. I include an assembler sample not to mess with language processors and LE (Language Environment) considerations:

001200 //*
001300 //TRN.SYSIN DD *
001400 *ASM XOPTS(SP)
001500          TITLE ‘USREXIT – EXIT STARTED THROUGH PLTPI’
001600          PRINT NOGEN
001700 USREXIT  AMODE 31
001800 USREXIT  RMODE ANY
001900 DFHEISTG DSECT ,
002000 *
002100 DBL1      DS   D                            DECIMAL CONV.WRK.AREA
002200 *
002300 USRRESP   DS   F
002400 USRRESP2  DS   F
002500 *
002600 * MESSAGES
002700 * ….+….1….+….2….+….3….+….4….+….
002800 * USR0001I EXIT STARTED
002900 * USR0002I EXIT NOT STARTED RESP XXX RESP2 XXX
003000 *
003100 USRWTO    DS   CL48
003200           ORG  USRWTO
003300 USRWTO1   DS   CL31
003400 WTORESP   DS   CL4
003500 USRWTO2   DS   CL6
003600 WTORESP2  DS   CL4
003700           ORG  ,
003800 *
003900 USREXIT DFHEIENT , CODEREG=REG03,EIBREG=REG11,DATAREG=REG13 DEFAULTS
004000 *
004100          EXEC CICS ENABLE PROGRAM(‘USREXIT1’) EXIT(‘XMEOUT’) START     C
004200                RESP(USRRESP) RESP2(USRRESP2)
004300 *
004400          CLC   USRRESP,DFHRESP(NORMAL)      RC = 0?
004500          BE    NORMAL_EOP                   YES, BRANCH
004600 *
004700 *        DISPLAY RESP AND RESP2
004800 *
004900          MVI   USRWTO,C’ ‘                  MOVE SPACES TO ..
005000          MVC   USRWTO+1(L’USRWTO-1),USRWTO    MSG AREA
005100 *
005200          MVC   USRWTO1,=C’USR0002I EXIT NOT STARTED RESP’
005300 *
005400          L     WRK0,USRRESP                 LOAD RESP
005500          CVD   WRK0,DBL1                    CONVERT TO DECIMAL
005600          MVC   WTORESP,=X’40202120′         MOVE IN EDIT MASK
005700          ED    WTORESP,DBL1+6               EDIT LAST 3 DIGITS
005800 *
005900          MVC   USRWTO2,=C’ RESP2′
006000 *
006100          L     WRK0,USRRESP2                LOAD RESP2
006200          CVD   WRK0,DBL1                    CONVERT TO DECIMAL
006300          MVC   WTORESP2,=X’40202120′        MOVE IN EDIT MASK
006400          ED    WTORESP2,DBL1+6              EDIT LAST 3 DIGITS
006500 *
006600          B     EOP1
006700 *
006800 NORMAL_EOP DS  0H
006900          MVI   USRWTO,C’ ‘                  MOVE SPACES TO ..
007000          MVC   USRWTO+1(L’USRWTO-1),USRWTO    MSG AREA
007100 *                             ….+….1….+….2.
007200          MVC   USRWTO1(21),=C’USR0001I EXIT STARTED’
007300 *
007400 EOP1     DS    0H                           SEND WTO
007500 *
007600          EXEC CICS WRITE OPERATOR TEXT(USRWTO)
007700 *
007800          EXEC CICS RETURN
007900 *
008000          LTORG ,
008100 *
008200 REGBAL   EQU   2
008300 REG03    EQU   3                            DEFAULT BASE REG
008400 REG11    EQU   11                           DEFAULT EIB REG
008500 REG13    EQU   13                           DEFAULT DYNAMIC STG REG
008600 WRK0     EQU   4                            USED FOR DECIMAL CONVERSION
008700 *
008800 REG05    EQU   5
008900 REG06    EQU   6
009000 REG07    EQU   7
009100 REG08    EQU   8
009200 REG09    EQU   9
009300 REG10    EQU   10
009400 REG12    EQU   12
009500          END   ,

Please note statements to test RESP and RESP2 response codes of ENABLE command and display them if nonzero.
You can test the program using CECI transaction after defining it to CICS TS system.

CEMT INQUIRE PROGRAM(‘USREXIT’)

I PROG(USREXIT)
STATUS:  RESULTS – OVERTYPE TO MODIFY
Prog(USREXIT) Leng(0000000000) Ass Pro Ena Pri     Ced
Res(000) Use(0000000000) Any Uex Ful Qua Cic

CECI LINK PROGRAM(‘SGKEXIT’)

After successful completion, you should see “USR0001I EXIT STARTED” message on zOS JES spool JOBLOG for that CICS region. If you execute more than once you see “USR0002I EXIT NOT STARTED” message with nonzero RESP and RESP2 codes mentioning it is already ENABLEd.
Finally insert the program in PLTPI table and exit will start at next CICS TS recyle.

Posted in CICS Transaction Server, IBM zEnterprise Servers | Tagged , , , , , , , , , , , , , , , , , , , , , | Leave a comment

Automatic Deletion of TS (Temporary Storage) Queues for CICS TS v4.2

CICS TS (Customer Information Control System Transaction Server) is IBM’s general purpose TP (Transaction Processing) monitor, application server and a middleware for system z servers.

During its regular operation, CICS utilizes TS queues for temporary storage requirements together with other types of resources. It is easy to utilize, just write some data, read later and delete the queue when you finish with it. But if applications are not developed by-the-book then it becomes nightmare for operations people. Local TS data set fills up, tasks either freeze or abend (Abnormally end) after the message:

DFHTS1311 cicsname Temporary storage data set is full and cannot be extended

If operations people cannot judge which queues to “purge”, the region is recycled, TS being started COLD clearing up TS queues.

After a number of incidents, “Lastusedint” parameter of TS entry is considered. This parameter displays the elapsed time in seconds since the queue was last referenced. Following master operator command could be issued or set to automation system for that region to purge TS queues older than 1 hour (3600 seconds):

CEMT Set TSqueue(*) Delete Lastusedint(3600)

Some shops utilize SPI (Systems Programming Interface) programs executed periodically to browse TS queues and delete unreferenced queues after specified interval.

For CICS TS v4.2 a new attribute EXPIRYINT is introduced for TSMODEL resource definition.

OBJECT CHARACTERISTICS CICS RELEASE = 0670
CEDC View TSmodel( USRTSM01 )
TSmodel : USRTSM01
Group : USRTSMDL
DEScription : TO DELETE TS QUEUES THRU EXPIRY
PRefix : +
XPrefix :
Location : Auxiliary Auxiliary ! Main
Expiryint : 00001 0-15000 (Hours)
RECOVERY ATTRIBUTES
RECovery : No No ! Yes
SECURITY ATTRIBUTES
Security : No No ! Yes
SHARED ATTRIBUTES
POolname :
REMOTE ATTRIBUTES
REMOTESystem :
REMOTEPrefix :
XRemotepfx :

Value of EXPIRYINT specifies expiry interval in hours and applicable nonrecoverable and local queues. Recoverable queues and queues created by CICS system are not effected.

Name prefix + (Plus sign) specifies model will match TS queues starting with any characters. Some more specific prefixes can be defined.

After installing model, it can be observed in TS models:

I TSM
STATUS: RESULTS
Tsm(DFHWEB ) Pre(DFHWEB ) Mai Exp(00000)
Tsm(DFHISLQ ) Pre(DFISLCLQ ) Aux Rec Exp(00000)
Tsm(USRTSM01) Pre(+ ) Aux Exp(00001)

If the TS queue name is matched to a TSMODEL with expiry interval, EXPIRYINT value of TS queue is set on creation.

I TSQ
STATUS: RESULTS – OVERTYPE TO MODIFY
Tsq(ESBV ) Num(00001) Len(0000000128) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00001003 ) Exp(00001)
Tsq(E346 ) Num(00001) Len(0000000128) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00002020 ) Exp(00001)
Tsq(IP31 ) Num(00005) Len(0000000640) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00002628 ) Exp(00001)
Tsq(I9IO ) Num(00001) Len(0000000128) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00002620 ) Exp(00001)
Tsq(KOJG ) Num(00002) Len(0000000256) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00004095 ) Exp(00001)
Tsq(KRBH ) Num(00001) Len(0000000128) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00002826 ) Exp(00001)
Tsq(KTAF ) Num(00006) Len(0000000768) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00003902 ) Exp(00001)
Tsq(KTBB ) Num(00001) Len(0000000128) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00001725 ) Exp(00001)
Tsq(MNBZ ) Num(00001) Len(0000000128) Aux
Tra(EKE2) Max(00128) Min(00128) Las( 00001829 ) Exp(00001)

CICS cleanup task periodically scans TS queues once in half an hour and deletes expired TS queues. Scanner issues a message similar to below message upon completion:

DFHTS1605 07/16/2012 12:12:26 CICS7248 Scan of local temporary storage queues completed. 30 local temporary storage queues were scanned and 8 were deleted.

Please note that expiry intervals of TS queues created before installing TS model with expiry interval do not change. Those queues should be deleted once and for all.

You could observe TS queues with last used interval larger than 3600 seconds (up to 5400 seconds actually) even if you set expiry interval to one hour. This is due to the execution periods of CICS scanner. It executes half an hour periods and some TS queues survive beyond 3600 seconds.

Posted in CICS Transaction Server | Tagged , , , , , , , , , , , , | Leave a comment

Shortening z/OS Recycling Time

IBM System z server workloads are comprised of heterogeneous transaction types. DB2 queries, WAS (Websphere Application Server), Websphere MQ (Message Queueing oriented middleware), CICS TS, CTG (CICS Transaction Gateway), RDZ (Rational Developer for System z), IMS (Information Management System) transactions. Together with builtin TCP/IP stack and Unix Systems Services, all of the facilities are served to enterprise ITC infrastructure through a lot of daemons.

Workloads are organized as address spaces. Address spaces are 64-bit addressable program and data containers. Initialization of server operating system consists of initialization of all address spaces one after the other.

Standalone zOS operating system instances have a few minutes of downtime per year. Clustering as Parallel Sysplex (Systems Complex) makes them available continuously. Although continuous operations is a daily routine for System z shops, scheduled recycling is necessary for release upgrades or testing initialization procedures for disaster recovery facilities.

Large shops have automation tools with initialization, termination and recycling procedures. Recycling may be an operational burden for shops without such tools. There are two contradictory considerations in recycling live systems. First one is not to cause loss-of-data and keep data integrity and second one is to keep outage as short as possible.

Termination consists of shutting down each address space one after the other. Next question is which address space to shut down first? There are “client” address spaces and “server” address spaces. It is logical to think to close down client address spaces first. Then server address spaces become idle and it becomes possible to terminate them normally without causing data loss.

Most important single group of address spaces to terminate normally is database related address spaces. Most probably DB2 and related address spaces. So address spaces using services of DB2 should be terminated earlier. Those are so-called middleware address spaces like CICS TS, Websphere MQ, CTG, WAS. Among address spaces in a group terminated concurrently, again consider terminating in-group clients like CTG and Websphere MQ earlier than server in this case CICS TS.

DB2 not terminating normally? I am keeping the advice “Do not mention MODE(FORCE) termination option of DB2, it may have side effects and create problems..” of my colleague DB2 Expert! I am just summarizing his recommendations:

  1. Wait for “DSNYASCP ‘STO DB2’ NORMAL COMPLETION” message. Depending on number of active transactions and sizes of buffer pools, normal termination may take some minutes.
  2. If you do not receive normal completion display active threads and utilities using -DISPLAY THREAD(*) command. Wait for utilities to complete.
  3. Watch for batch, CICS protected threads, WAS and RRSAF connections. Either wait for completion or cancel using -CANCEL THREAD command.
  4. Terminate DDF using -STOP DDF command and wait for normal completion.
  5. If you sweat, keep cool and go to item number 1

For further information related with DB2, please refer to databaseguidelines.blogspot.com

Other groups of address spaces to terminate are as follows:

  1. Monitoring tools address spaces – Address spaces for monitoring tools like BMC MainView or Omegamon can be terminated after terminating DB2 address spaces.
  2. Printer related address spaces – They should be terminated before terminating communications server address spaces.
  3. Communications server address spaces – They are TCP/IP and VTAM. TCP/IP and related address spaces like TN3270E should be terminated before. Because OSA (Open system adapter) interface devices are defined to VTAM.
  4. SMS (System Management Storage) related address spaces – They are HSM (Hierarchical Storage Manager) and RMM (Removable Media Manager, cartridge management system) address spaces. They can be terminated any time provided that space management and cartridge I/O activities do not take place.

Printer signoff, inactivate, disconnect, line drain commands may flood system consoles for installations without console message automation. If this is the case, following messages can be suppressed by MPF (Message Processing Facility):

  1. $HASP097 LINExxxx IS DRAINED
  2. $HASP203 RMTxxxx DISCONNECTED FROM LNExxxx
  3. $HASP210 SESSION RMTxxxx LOGGED OFF LNExxxx
  4. $HASP879 RMT(xxxx) DEVTYPE=LUTYPE1,STATUS=INACTIVE/LNExxxx
  5. $HASP880 LINExxxx UNIT=SNA,STATUS=ACTIVE,DISCON=NO

zOS consoles are 3270 terminals. Model 2 terminals have 19 lines for messages, model 4 terminals have 39 lines for messages. Consoles are designated to keep messages on screen at least for 1/4 seconds. So in one second at most 4 * 39 = 156 messages can be displayed. Termination messages for thousands of printers flood console screens for long time not to mention several messages for one printer and related line. During flood termination of other other address spaces may not be observed.

To wrap up, following procedure may be used for termination:

z/OS Shutdown Procedure

Date: ________________

Start Time: ________________

Command Description
Start a TSO session, a monitoring tool session and an HMC (Hardware Management Console) session Display SDSF screenDisplay DB2 threads screen
STOP CTG720MODIFY CICSxxxx,CEMT PERFORM SHUTDOWN IMMEDIATESTOP CHCPROC Terminate CTGTerminate CICS regionsTerminate IBM Infosphere CDC (Change Data Capture)

Terminate other DB2 client address spaces

-STOP DB2 Wait for normal completion
DISPLAY A,L See all CICS and DB2 address spaces are terminated
Terminate monitoring tool address spaces Logoff monitoring tool
SET MPF=(00,PR) Issue MPF command to suppress printer termination messages provided that MPFLST00 is standard MPF PARMLIB member and MPFLSTPR is the member prepared to suppress printer messages
START AOPSTOPSTOP APIJPJCL Terminate Infoprint Server
STOP TN3270ESTOP TCPIP Logoff TSO (Time Sharing Option) session
STOP TSO Stop TSO address space
HALT NET,QUICK Terminate VTAM
HALT EOD Make sure that SMF records are written
$P JES2,ABEND Reply like xxEND

End Time: ________________

Initials: ________________

Command Abbreviations

D DISPLAY

F MODIFY

S START

P STOP

T SET

Z HALT

-CAN -CANCEL

-DIS -DISPLAY

-STA -START

-STO -STOP

Posted in IBM zEnterprise Servers | Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , , , | Leave a comment

Daylight Saving Time Settings in IBM Mainframes – IBM Büyük Sistemlerde Yaz Saati Uygulaması

Donanım ve yazılım platformlarında saatlerin ileri ve geri alınması genellikle otomatik olarak gerçekleşir. IBM büyük sistemlerde bu işlemler otomatik olarak yapılmaz.Önce zaman kavramlarını gözden geçirelim. Kullandığımız yerel saat, bulunduğumuz boylamın Greenwich kasabasından uzaklığıyla orantılı olarak belirlenmiştir. Greenwich kasabasındaki saate GMT (Greenwich Mean Time) veya daha güncel deyimle UTC (Coordinated Universal Time, Eşgüdümlü Evrensel Zaman) diyeceğiz. Ülkemiz doğuya doğru iki saat ileridedir.

Bu durumu aşağıdaki zaman çizgisinde görüyorsunuz.

Daylight saving time settings are performed automatically in most hardware, software and middleware platforms. This is not the case for IBM System z servers.First let’s go over time concepts. Local time we use is determined proportionally by the distance of the longitude of our location to Greenwich town. The time in Greenwich is called GMT (Greenwich Mean Time) or UTC (Coordinated Universal Time) as a more recent expression. Turkey is two hours ahead of beginning longitude.

You can observe the following timeline for details.

Bu durumu işletim sistemine tanımlarız. Veri tabanı ve uygulama sunucusu gibi sunucular zaman bilgisini gerektikçe işletim sisteminden alır. Yani sistemler açılırken yerel saati belirtmemizin yanısıra bulunduğumuz zaman dilimini de belirtiriz. Zaman dilimimizi E02.00.00 gibi belirtiriz. Biz doğuya doğru iki saat ilerideyiz anlamına gelir. İşletim sürecinde bu yerel saat ve GMT (UTC) değerleri işletim sistemi tarafından güncel tutulur.Veri tabanı sistemleri güncellemeler ve geri dönüşler için kullandığı log kayıtlarını GMT (UTC) saatine göre damgalar. Uygulama sunucuları ise yerel saati kullanırlar.

Bazı kuruluşlar zaman dilimini Greenwich kasabasında imiş gibi tanımlarlar. Bu durumda yerel saat ve GMT (UTC) aynı olur. Bu iki saatin aynı olması yaz saati uygulaması prosedürleri açısından bir fark doğurmaz.

Şimdi sistem komutlarıyla saat ve zaman dilimi değiştirme özelliklerini gözden geçirelim.

SET CLOCK sistem komutu yalnızca yerel saati değiştirir. Aşağıdaki örnekte bunu görebilirsiniz. Önce ve sonra DISPLAY TIME komutları girerek SET CLOCK komutunun dökümantasyonda belirtildiği gibi yalnızca YEREL saati değiştirdiğini görürsünüz.

We define this parametrically to the operating system. Data base management servers, application servers and middleware received this information from the operating system whenever necessary. We define timezone like E.02.00.00. This means that we are ahead of two hours toward the east. In the process of operation local time and GMT (UTC) maintained current.Data base systems use GMT (UTC) time in log timestamps maintained for recovery purposes. Application servers use local time instead.

Some institutions use GMT (UTC) timezone as if they are operating in Greenwich town. In this situation local time is equal to GMT (UTC) time. This equality does not create a difference in setting operating procedures for daylight saving time.

Now let us go over changing clock and timezone parameters.

SET CLOCK system command changes local time. You can observe this at the following example. Entering display commands before and after command, you observe that command just changes local time as documented.

DISPLAY T
IEE136I LOCAL: TIME=10.09.40 DATE=2011.304 UTC: TIME=09.09.40 DATE=2011.304

SET CLOCK=11.10.00

DISPLAY T
IEE136I LOCAL: TIME=11.10.08 DATE=2011.304 UTC: TIME=09.10.44 DATE=2011.304

SET TIMEZONE komutu parametre olarak belirttiğimiz saat farkını GMT (UTC) değerine ekleyerek veya çıkararak yerel saati değiştiriyor. Yine önce ve sonra görüntüleme komutları yazarak bunu görebilirsiniz. Timezone change command takes time difference you mention and changes local time adding to or substracting from GMT (UTC) time. You can observe this by displaying before and after time settings.

DISPLAY T
IEE136I LOCAL: TIME=11.08.52 DATE=2011.304 UTC: TIME=09.08.52 DATE=2011.304

SET TIMEZONE=E.01.00

DISPLAY T
IEE136I LOCAL: TIME=10.09.40 DATE=2011.304 UTC: TIME=09.09.40 DATE=2011.304

Bu değişiklikleri yine zaman çizgilerinde görelim.Yaz saati başlangıcında sistem saatini veya zaman dilimini bir saat ileri aldığınızda yerel saatle oynamış oluyorsunuz. SET TIMEZONE komutunun 2008 yılında z/OS V1.9 sürümünde geldiğini tekrar hatırlatıyoruz. Let us review this changes in below timeline.When daylight saving time starts, you just change local time by changing system clock or timezone. Let us remember that SET TIMEZONE command is introduced at z/OS V1.9 in 2008.

Böylece sunucuyu bir anda Mogadişu kentine ışınlamış gibi oluyorsunuz. Greenwich kasabasındaki saat değişmediği için veri tabanı log kayıtlarıyla ilgili bir sorun olmuyor. Ama konumuz dışındaki yerel saatle ilgili sorunlar sizin ve değişiklik danışma kurulunun önünüzdedir.Aynı biçimde yaz saati uygulamasının sonunda da zaman dilimini bir saat azaltarak batıya alıyorsunuz. Sunucunuz Türkiye’ye geliyor ve GMT (UTC) yine değişmiyor. It is like sending your server immediately to Mogadishu. As the time in Greenwich town has not changed, there is no problem with data base log records. But local time related issues we will not mention here may be still waiting attention of you and your change control board.At the end of daylight saving time, you set timezone one hour west. Your server comes back to Turkey and GMT (UTC) does not change again.

İşletimsel Prosedürİşletimsel prosedürü hazırlarken SET TIMEZONE komutunu kullanmanızı öneririz. Çünkü saati değiştirirseniz yerel saatle GMT (UTC) arasında saniyeler düzeyinde de olsa fark oluşur. Bunu gidermek çok pratik değildir.

Adımları şöyle özetleyebiliriz.

E.02.00.00 gibi GMT (UTC) zaman diliminden iki saat ileride olduğunuz durum için yapacağız.

Yaz saati uygulamasının başlatılması

  1. Saati ileri alacağınız zaman zaman dilimini SET TIMEZONE=E.03.00 gibi ayarlayınız.
  2. PARMLIB(CLOCKxx) üyesinde TIMEZONE=E03.00.00 gibi değişiklik yaparak saatin sonraki kapanışlar, POR ve IPL işlemlerinde de doğru kalmasını sağlayınız.

Yaz saati uygulamasının bitirilmesi

  1. Saati geri alacağınız zaman zaman dilimini SET TIMEZONE=E.02.00 gibi ayarlayınız.
  2. PARMLIB(CLOCKxx) üyesinde TIMEZONE=E02.00.00 gibi değişiklik yaparak saatin sonraki kapanışlar, POR ve IPL işlemlerinde de doğru kalmasını sağlayınız.

Eğer normal operasyonda GMT (UTC) saat dilimini kullanıyorsanız yaz saati uygulamasına başlarken zaman dilimini E.01.00 olarak ayarlamanız, yaz saati uygulamasını bitirirken E.00.00 olarak ayarlamanız gerekir.

İşletimsel prosedürleri hazırladıktan sonra veri tabanı geri alma ve kurtarma işlemlerinin beklediğiniz gibi olduğunu test sistemlerinizde gördükten sonra canlı sistemlerde uygulayabilirsiniz.

Gördüğünüz gibi yapılacak işlemler iki yönlüdür ve ilkini doğru yapmazsanız ikincisinde sıkıntı oluşur.

Operational ProcedureI recommend that you use SET TIMEZONE command. If you change clock, just local time changes and there would be a small difference with GMT (UTC). It is not so easy to get rid of this difference due to typing delay.

Steps are as follows.

Let us assume you are two hours ahead of GMT (UTC) with the definition E.02.00.00.

Starting daylight saving time

  1. When you set time ahead set timezone as SET TIMEZONE=E.03.00
  2. In PARMLIB(CLOCKxx) set TIMEZONE= E03.00.00 so time services are started correctly after later shutdown, POR and IPL procedures.

Ending daylight saving time

  1. When you set time behind set timezone like SET TIMEZONE=E.02.00
  2. In PARMLIB(CLOCKxx) set TIMEZONE= E02.00.00 so time services are started correctly after later shutdown, POR and IPL procedures.

If you are using GMT (UTC) timezone, set timezone as E.01.00 as you start daylight saving time and set timezone as E.00.00 as as you end daylight saving time.

After preparing operational procedures and testing data base recovery and rollback functionalities, you should observe results as you expect and then you could deploy these procedures to live systems.

As you may observe, operations is two-fold. If you do not perform first part then you could experience trouble and need to apply temporary workarounds.

Posted in IBM zEnterprise Servers | Tagged , , , , , , | Leave a comment

Migration of System z Servers

In Memory of Levent Durdu

Perfect friend, unfailing colleague, guiding leader we lost very early
1955 – 24 April 2008
Levent Durdu
Levent Durdu Anısına
Erken yitirdiğimiz mükemmel dost, yorulmaz iş arkadaşı, yol gösterici lider
1955 – 24 Nisan 2008

Existing Environments

  1. System test (a.k.a. SYST)
  2. Development and application test system (a.k.a. TEST)
  3. Production system (a.k.a. PROD)

Existing Software

  1. z/OS V1.4
  2. CICS/TS V2.3
  3. DB/2 V7.1
  4. Content Manager V8.2
  5. Monitoring tools

Migration Targets

  1. z/OS V1.10
  2. DB/2 V 8.1
  3. Monitoring tools

Extensively Utilized Components

  1. VTS (Virtual tape server)
  2. RMM (Removable media manager, cartridge management system)
  3. HSM (Hierarchical storage manager)
  4. Infoprint server
  5. SMTP server

Discontinued Components

  1. CICS/TS V1.3
  2. ACF/NCP (Advanced communication functions/Network control program)
  3. JES328X (Job entry subsystem 328X) Print Facility

Migration Steps

  1. Apply toleration, coexistence and fallback maintenance on existing software
  2. Migrate Content Manager CICS/TS system from V1.3 to 2.3
  3. Install z/OS V1.10
  4. Customize z/OS V1.10 as SYST (System test)
  5. Install DB/2 V8.1 as SYST
  6. Clone SYST system for TEST (Development and application test)
  7. Migrate TEST system to z/OS V1.10
  8. Clone TEST system for PROD
  9. Migrate PROD system to z/OS V1.10
  10. Install DB/2 V8.1 to TEST system
  11. Install TEST system to DB/2 V8.1
  12. Install DB/2 V8.1 to PROD system
  13. Install PROD system to DB/2 V8.1

DASD (Direct Access Storage Devices) Planning

  • 3390 Model 9 device geometry
  • 2 volumes for system catalog
  • 2 volumes for distribution libraries
  • 2 volumes for target libraries
  • 3 volumes for SMF (System Management Facility) data gathering
  • 6 volumes for paging
  • 6 volumes for spool
  • 1 storage volume for TSO (Time Sharing Option) datasets [For business continuity purposes]
  • 1 public volume for temporary datasets and dynamic dumps [For business continuity purposes]
  • 2 volumes for shared subsystem CDSs (Control datasets) [RMM and HSM]
  • 1 volume for DB/2 V8.1 installation system libraries
  • 1 volume for DB/2 V8.1 installation system catalog and table spaces

SSA (System Specific Alias) Planning

  • Installation naming conventions
  • Completion of missing aliases for existing datasets [Due to past re-creations/enlargements]

Stand-alone Editor ZZSA

  • IPLable CD verification, documentation and filing next to HMC (Hardware Management Console)

Backups

  • Full dump DSS (Dataset services) backups of system disks on cartridge
  • Retention period one month
  • Perform daily after modifications or weekly
  • Verification of stand-alone DSF (Device support facility) and stand-alone DSS cartridges

Catalog Considerations

  • Connect all existing user catalogs
  • Define all existing aliases
  • Connect master catalog of existing z/OS V1.4 system
  • Define SSA of existing z/OS V1.4 system as alias
  • Define SSAs of z/OS V1.10 system datasets
  • Define alias of SSA of z/OS V1.10 system as alias to other systems
  • SSA not usable for IBMUSER and SYSADM datasets, lengths of some dataset names are close to maximum. Aliases of IBMUSER4 and SYSADM4 will be used instead
  • Store all catalog listings on workstations in case of immediate access requirement

SMP/E (System Management Program/Extended) and Open MVS (a.k.a. OMVS) Datasets for Existing Subsystems

  • CICS/TS, DB/2 V7.1 and Content Manager SMP/E and OMVS datasets had been created using z/OS system default high-level qualifiers
  • Replication planned as they should be cataloged in the master catalog of z/OS V1.10 system

z/OS V1.10 Installation and Cloning

  • Initialize disks
  • Create master catalog
  • Clone distribution and target library disks
  • Allocate JES checkpoint and spool datasets
  • Define SMF clusters
  • Allocate static dump datasets
  • Define page datasets
  • Define couple datasets
  • Allocate and copy primary and backup RACF datasets
  • Update z/OS V1.10 templates for RACF datasets
  • Catalog RACF database name table
  • Move started task users from ICHRIN03 to STARTED profile for ease of maintenance
  • Catalog a dummy ICHRIN03 started task module
  • Allocate and copy UADS dataset

Cloning Existing CICS/TS V2.3 and DB/2 V7.1 Subsystems

  • Clone existing CICS/TS V2.3 and DB/2 V7.1 subsystems if not available to z/OS V1.10

Customer Specific Initialization and Customization Libraries

  • SYS1.USR.CMDPROC
  • SYS1.USR.LINKLIB
  • SYS1.USR.OPERLIB
  • SYS1.USR.PANELIB
  • SYS1.USR.PARMLIB
  • SYS1.USR.PROCLIB
  • SYS1.USR.SAMPLIB
  • SYS1.USR.SMS
  • SYS1.USR.TCPPARMS
  • SYS1.USR.VTAMLIB
  • SYS1.USR.VTAMLST
  • One set for each SYSPLEX (Systems Complex)
  • Protect generically
  • Do not modify z/OS V1.10 system libraries

SYS1.USR.LINKLIB

  • Eligible for user modules
  • Add to LNKLST (Link list)
  • Authorize through PROG00 PARMLIB member

SYS1.USR.OPERLIB

  • Operational jobstreams
  • Start using S RDR,J=member-name

SYS1.USR.PARMLIB

  • Add to SYS1.IPLPARM(LOADxx) member
  • Modified initialization members will be stored here
  • Other PARMLIB libraries will have unmodified initialization members
  • Parameters of operational jobstreams will also be stored here
  • Load UNICODE conversion tables

SYS1.USR.PROCLIB

  • Add to master JCL PARMLIB member
  • Add to JES procedure
  • Modified procedures and installation specific procedures will be stored

SYS1.USR.SAMPLIB

  • Installation, maintenance and modification jobstreams related with z/OS V1.10
  • Only one library as opposed to other customer specific libraries
  • Accessible by all other systems through SSA like SEZOS10.SYS1.USR.SAMPLIB

SYS1.USR.TCPPARMS

  • TCP/IP, FTP, TELNET 3270 and SMTP initialization parameters
  • Performance related parameters are migrated from PROD system
  • Separate TN3270E parameters from TCP/IP startup file for z/OS V1.10

SYS1.USR.VTAMLIB

  • Define to VTAM start procedure
  • Define to TN3270E start procedure
  • Authorize through PROGxx PARMLIB member
  • Eligible for USSTAB and other VTAM modules
  • Display system name and system specific VTAM APPLIDs on USSTAB

SYS1.USR.VTAMLST

  • Define to VTAM start procedure
  • All modified VTAM start members are stored here
  • Members on other VTAMLST libraries are not modified

JES2

  • TGSPACE set to allow 6 full DASD spool space

SMF (System Management Facility)

  • Create 4 datasets for TEST system
  • Create 6 datasets for PROD system
  • Install IEFACTRT step termination exit
  • Setup RMF records collection
  • Migrate data gathering jobstreams

First IPL

  • Format JES2 spool
  • Examine SYSLOG to remove obsolete initialization statements and parameters
  • Examine SYSLOG to correct misplaced parameters
  • Compare SSI (Subsystem Interface) entries by D SSI,ALL
  • Compare authorized libraries by D PROG,APF
  • Compare Link list libraries by D LNKLST,NAME=CURRENT
  • Execute installation verification procedures (a.k.a. IVPs)

SMS

  • Define CDSs (Control datasets)
  • Enter new installation SYST objects manually
  • Migrate TEST and PROD SCDS datasets
  • Copy ACS routines to SYS1.USR.SMS library
  • Validate and activate SMS
  • Compare existence of all objects
  • OMVS and TCP/IP will be available after SMS activation

VTS

  • Connect VOLCAT to new system
  • Verify existence of cartridge volumes

RACF

  • Perform all security related modifications to clone system by batch jobstreams. It will be easier to repeat immediately before cutover

LOGGER

  • Define OPERLOG and LOGREC logstreams
  • Migrate DB/2 V7.1 RRS logstreams

WLM (Workload Manager)

  • Migrate definitions from PROD
  • Create TN3270E started task goal if not defined

TCP/IP

  • Migrate Turkish translation table
  • Migrate SMTP specific Turkish translation table
  • Perform MAKESITE
  • Migrate TN3270E printer definitions
  • Migrate TN3270E LU groups

SMP/E

  • Execute SYSMOD listings for z/OS V1.10
  • Execute SYSMOD listings for CICS/TS V2.3, DB/2 V7.1 and Content Manager V8.2
  • Prepare and apply dummy MACLIB and LOADLIB SYSMODS

OAM

  • Perform DB/2 V7.1 customization and binds

Infoprint Server

  • Export z/OS V1.4 Infoprint Server configuration
  • Allocate a large HFS dataset for Infoprint Server
  • Mount temporarily to copy existing z/OS V1.10 configuration
  • Copy /var/Printsrv
  • Add HFS dataset to BPXPRMFS PARMLIB member to mount permanently
  • Import z/OS V1.4 printer configuration
  • Verify printer configuration from ISPF/PDF Infoprint Server panels
  • Migrate JES, VTAM and TCP/IP printer definitions
  • Test printing

RMM

  • Use DASD volumes for shared subsystems
  • Define Master CDS cluster
  • Copy z/OS V1.4 master CDS using IDCAMS REPRO utility
  • Update CDS ID using EDGUTIL
  • Allocate Journal dataset
  • Copy z/OS V1.4 journal dataset
  • Start DFRMM subsystem
  • Verify existence of cartridge libraries

HSM

  • Use DASD volumes for shared subsystems
  • Define CDS clusters
  • Copy CDS clusters from z/OS V1.4 system using IDCAMS REPRO utility
  • Allocate backup datasets
  • Allocate and copy journal dataset using IEBGENER utility
  • Migrate startup PARMLIB parameters
  • Prepare and include ADDVOL statements for newly installed z/OS V1.10 DASD volumes
  • Start DFHSM subsystem
  • Compare parameter configurations

Customer Specific TSO/E logon procedures

  • Administrators
  • Group administrators
  • Developers
  • Developers with Content Manager DB/2 access
  • Operators
  • Modified TSO/E logon procedures to SYS1.USR.PROCLIB
  • Modified TSO/E CLISTs to SYS1.USR.CMDPROC
  • Modified ISPF/PDF panels to SYS1.USR.PANELIB
  • Display system name on each panel
  • Display user type header on each panel
  • Keep TSO/E and ISPF/PDF target libraries intact
  • Test all procedures

Batch and Online Program Compilation

  • Perform batch LE (Language environment) customization
  • Perform online LE customization
  • Copy z/OS V1.4 CICS/TS V2.3 and DB/2 V7.1 compilation jobs to SYS1.USR.PROCLIB
  • Convert jobstreams to Enterprise PL/I for z/OS
  • Test and document in SYS1.USR.SAMPLIB
  • Distribute to users

Compare z/OS V1.10 Clone System vs z/OS V1.4 System Members and Parameters

  • PARMLIB
  • PROCLIB
  • Logstreams
  • SMS objects
  • VTAMLST
  • VTAMLIB
  • TCPIP
  • Infoprint Server

Final Tasks from z/OS V1.4 System Before Migration

  • Copy RACF and UADS
  • JES2 spool offload
  • SMF switch and collection
  • PARMLIB CONSOL00 member
  • VTAMLST LOCAL NON-SNA terminals
  • TCPPARMS IP number
  • Final shutdown

First IPL from z/OS V1.10 System After Cutover

  • Record IPL address and LOADPARM for z/OS V1.4 system
  • Record IPL address and LOADPARM for z/OS V1.10 system
  • First IPL
  • Control IPL SYSLOG
  • JES2 restore spool
  • Migrate, validate and activate SMS
  • Check VTAM network connections
  • Check TCP/IP network connections, perform SITEINFO
  • Check Infoprint Server printers
  • Start DB/2 V7.1 subsystems
  • Start CICS/TS systems
  • Migrate and start RMM
  • Migrate and start HSM

End of Migration

  • Celebrate
Posted in IBM zEnterprise Servers, Project Management | Tagged , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , | Leave a comment