Mainframes 360
The one stop destination for System Z professionals

Thursday, February 25, 2010

IDCAMS – LISTCAT Command

Q. What is a Catalog? What is meant by cataloging or un-cataloging a file?

Data on computers is stored in Files. The files are generally scattered everywhere haphazardly in Computer Memory. Well, let's say you want to see, or read the file in which you stored your Birthday photographs. The file containing the birthday photographs could be placed anywhere in the Computer Memory. How do you know, the location of your file(where is it) in this vast galaxy of Computer Memory space? It’s like finding a needle in a haystack.

The MVS Operating System keeps a diary/register, which contains name of each file, and its location in the computer memory. This diary/register is like your map around the Mainframe's memory land. You want to find a file, just see the MVS operating system’s diary, and it will tell you the location of your file.

This diary or map, which you consult to find a file on a Mainframe computer, is called Catalog. The Catalog contains entries for each file, a file-name and a pointer to the physical memory location, where the file is present.

Not all the files present on a Mainframe may have an entry in the Catalog. If a given file’s entry is not present in the Catalog, you will have to find the file manually, by explicitly specifying the UNIT and VOLUME parameters on the DD Statements. Adding an entry for a file i.e. File name and its location to Catalog, is called cataloging a file. Deleting the entry of a file, from the Catalog is called uncataloging the file.

Actually, the Catalog itself is a VSAM File. So, you can use the ready-made IBM Supplied program IDCAMS to read the contents of the Catalog.

Q. What is the LISTCAT command in IDCAMS?

LISTCAT command stands for List Catalog. It is the control-instruction or order to IDCAMS, to display the contents of the Catalog.

LISTCAT command of IDCAMS is a very powerful tool. You can use LISTCAT command to ask a Yes/No Question. Is there a file by the name of AGY0157.DEMO.FILE on the system? If such a file exists, the answer returned is yes. If the file does not exist the answer returned is no.

The LISTCAT command has the following syntax :

LISTCAT ENTRIES(your.file.name) ALL

I have written a JCL, to test if a given file is available on the system. Take a look at it –

Image144[1] 

In the above JCL, I have asked a question, is there a file called AGY0157.DEMO.FILE available? If yes, then IDCAMS will leave behind a Condition Code = 0. If the file does not exist, then IDCAMS will sign off with a return Code > 0.

Image145[1]

Since, I do have a file called AGY0157.DEMO.FILE, IDCAMS will print a description about the file to the log. The Log shows various details – like the catalog in which the file's entry is present, when the file was created, the file’s location(On which volume the file resides?) etc.

Image146[1]

Upon scrolling down to the bottom of the log, you would find that, IDCAMS has left a trail – the condition code is set to 0.

Image147[1]

Q. Wait a minute, you are asking me to use LISTCAT command, just to know if the file’s availability. In TSO/ISPF, I would just navigate to Dataset List Menu 3.4, and perform a simple search to check availability. What’s the difference?

That’s a good question, why use LISTCAT Command, when you can just use Dataset List Menu 3.4. Let me set things straight. The Dataset List Menu 3.4 is an ISPF Screen, where you type the name of the file, and press <ENTER>, you get the list of files, and details(description) about them.

Image148[1]

Actually, Dataset List Menu 3.4 is just a GUI Front-end or screen. It presents a nice, friendly user-interface for checking the availability of a file. In background, there is a ready-made template JCL. The file-name you typed on the 3.4 screen is substituted in this template JCL. This template JCL, uses LISTCAT command to find infomation from the catalog. This template JCL is ready-made, you don’t have to write it. When you press <ENTER>, this template Job is submitted to the MVS Operating System. After the job completes, the results of this job, are populated on the ISPF Screen.

Using the TSO/ISPF Menu 3.4, is equivalent to submitting a job in background which uses LISTCAT command to query the Catalog, and check the availability of the file.

In fact, just to let you guys know, behind every ISPF Menu screen, there is a  corresponding skeleton or template JCL that runs in the background. ISPF Screen is just a presentation layer, a front-end.

Q. Well that sounds good in theory. Is the LISTCAT command used in real production environment, in projects?  

In most projects/sites, there is a complex relationship amongst systems. As an example, suppose there are two Jobs(systems) 1 and 2. Suppose, the JOB 1 produces an Output file. This Output file is fed as Input to JOB 2. In other words, the Job1 is the feeder system of Job 2. Generally in projects, they would express this fact as, Job 2 has a dependency on Job 1.

Suppose Job 2 has been scheduled to run at 12 o'clock. Remember, that Job 2 expects to pick up a file, which Job 1 will produce as Output. If this file is not available by 12 o'clock, when the Job 2 is about to start, the Job 2 will fail(abnormally end) or abend. 

Many-a-times, Job 1 gets delayed, and does not make it within the cut-off time -  12 o'clock. In this unfortunate situation, Job 2 abends(fails). How could you prevent this incident?

You could add a condition to the Job 2, to first check if the file is available. Provided the file is available, the Job 2 should process it. If the file is not yet available, the Job 2 should bypass it, avoid processing it. Take a look at the JCL, I have written below -

Image149[1]

Here, I have used LISTCAT command to check the availability of the file AGY0157.DEMO.FILE. If the file exists, the step //CHECK01 leaves behind a condition code=0. If the file is not available, I set the
MAX Condition Code = 4. The next step //PROCESS will execute if, the condition code of //CHECK01 is 0. If the condition code of //CHECK01 is 4, meaning the file is not available, the next step //PROCESS is bypassed. The job will however not fail or abend.

Friday, February 12, 2010

Submitting Job

Q. How do you submit a Job on Mainframes for execution?
On Mainframes, when you want to perform any task, you write a Job, and give it to the Mainframe Computer for processing. This is called submitting a job.

However, contrary to what you might fancy, your job doesn’t run immediately. Picture this - there are hundreds and thousands of Jobs, that are submitted on a Mainframe, minute-by-minute, every second, by different folks. How would the Mainframe computer decide which job goes first, and then which goes next and so on..

The MVS Operating System prepares a pretty time-table, a schedule, that goes something like this - JOB 1 runs at 12 o’ clock, JOB 2 runs at 1 o’ clock, JOB 3 runs at 2 o’ clock and so on.. Thus every job is allotted a time-slot(period) in the Mainframe's Calendar/time-table.

Before you are just about to submit a job on Mainframes, give your JCL a cursory-glance, to ensure it is syntactically-correct. Forgetting to put a comma, or inserting unnecessary extra-whitespaces, can lead to JCL-Errors. Be cautious about the Datasets begin used by the Job. For example, make sure you've already created the Input-Dataset AGY0157.DEMO.INPUT. Click here to get the contents of the Input-Dataset AGY0157.DEMO.INPUT, if you would like to execute the below-job. To submit a JOB on Mainframes, you must type SUBMIT command or just SUB, on the command line of the editor.  I have shown below, how you submit a job on Mainframes.

Image124[1]

When you press <Enter>, the Job AGY015A gets submitted to the Mainframe computer for processing. This is indicated by a *** message displayed at the bottom of the screen. The *** indicates, that TSO is waiting for me to read the message. As soon as you press <Enter> key again, the message goes away.

Image125[1] 

A common practice adopted by most Mainframe Programmers, is to code the NOTIFY parameter on the JOB Statement. Coding this parameter is quite useful, as it gives you an alert, a notification message, saying "The job AGY0157A has completed".

Image126[1]
Q. How do you see the Job Print Output in TSO/ISPF?
I submitted the simple 3-Step JCL, by typing the SUBMIT command. Shortly after, MVS also alerted me, by sending a message, that my job AGY0157A completed.

But, I don’t know for sure, whether my job completed successfully, or it failed(and the reason why it failed). MVS Operating System prints messages to a log, as the job is processed. These run-time messages which get recorded to the log, helps track, if the Job completed successfully or it failed. Further, they also contain error-messages that point out why the job fails.

To see the print output/log of any Batch Job, you use a software called SDSF. SDSF stands for Spooler Display and Search Facility – its a software for seeing the output in the Spool.

What is Spool? Well, generally, the log of a Job, when it runs on Mainframes, are to be sent to a printer. But prior to sending the log to the printer, the logs have to wait(buffered) in a staging area(in a queue), because hundreds of jobs complete every minute, and the logs gotta be printed. Not all logs can be printed at once. The logs have to wait in a Queue/staging area. This staging area is called Spool(Queue). The Mainframe printer will pick up the log from the spool one-by-one and print it.

Before TSO and SDSF was invented, you couldn’t see the output till the time, it got printed. Waiting for it to get printed, could take hours on end. Now-a-days, TSO makes this easy, it’s possible to view the logs in the Spool(Queue).

Image122[1] 

To view the log of a Job in the Spool(Queue), type START SDSF and press <Enter>. This shows the shows the SDSF Screen. To see the log of Job, you must type ST on the SDSF menu and press <Enter>.

Image128[1] 

A list of the logs for all jobs in Spool(Output Queue) is displayed. By default, this list will displayed only those jobs submitted via your
TSO User-id. For example, if my TSO User-id is AGY0157, it will display all jobs starting with(prefix) AGY0157*.

Image131[1] 

The screen displays a list of logs of all jobs, you have submitted. For each job that you submit, there is an entry, and it shows various details like the job-name, owner, class and priority.

If your job is complete, and its log is ready to be dispatched to the printer, the log is on the PRINT Queue. On the other hand, if the job is still running, then it is still in the EXECUTION Queue.

Mostly, all jobs that you submit, their logs would be dumped here in the Spool. To view the contents any particular job log, you can type ? against the job=name and press <Enter>. This takes you inside the log.

Image132[1]

The log(print output) of a job, contains several sections or Listings. You can type S(Show) against each Listing, and view the contents of it. 

The Input Queue and Output Queue are analogous to the run-ways on an Airport. The Input Queue is runway from where the Jobs take-off. After their flight is complete, the jobs land on the runway called the Output Queue.But who's the Air Traffic Controller(ATC) on this Airport?

The JES(Job Entry Sub-System) is a software that manages the Input Queue and Output Queue. The JES acts like Traffic Controllers(Traffic cops). Without them, two airplanes or jobs could collide. The JES is a part of the MVS Operating System that decides, what time a job can takeoff safely from the Input Queue(Takeoff runway). Thus, it monitors the traffic in the Input Queue(Runway) and prepares the time-table, a chart or a schedule for Jobs entering the system.

On the Input Queue side, the JES(Job Entry Subsystem) welcomes all Jobs that enter into the Input Queue, reads the Jobs' JCL, converts it into an internal format known to MVS, and schedules the job for takeoff at 2 o’clock or 3 o’clock, till the Job takes off, and begins its flight(execution).

Sometimes, on the Output Queue(Staging area) side contains logs(outputs) of old jobs. Such old logs(print outputs) of a job, are deleted from time-to-time periodically by JES. So, JES ensures that the Output Queue clean and tidy.

The JES Message Log would contain messages, as shown in the below Snap. You can click here to see the JES Message Log in the ordinary Text-Format.

Image133[1] 

The Job Entry Subsystem(JES) reads the Job’s JCL Statements. My Job had 24 lines of JCL Code, so JES reports this as 24 cards read.

Once the job takes off from the Input Queue, it executes, and lands at Output Queue. The log(print output) of Job, is referred as SYSOUT. This contains 125 lines of text. JES reports this as 125 SYSOUT Print records. This implies, the log(print output) of the job, contains 125 lines of text, in the Output Queue(Staging area), waiting to get printed.

Look at the snap below. In the JCL-Listing, MVS numbers the JCL Statements that you submitted as part of the job. You can click here to see the JCL-Listing in an ordinary Textual-Format.

Image134[1] 
The JESYSMSG Listing contains Memory Allocation and Cleanup Messages. ALLOC tells you which devices and how much memory was allocated for the job-step. As you know, one step runs one program. It also informs you about the CPU time required to process a Job-step. Every Job-step leaves behind a trail, a COND CODE in the range of 0000 to 4095. The below picture shows how the JESYSMSG-Listing looks. You may also click here, to see the JESYSMSG-Listing in ordinary textual-format.

Image135[1] 
But apart from this , when you want to write your own extra notes to the logs(diary), you code a SYSOUT=* parameter on the DD Statement. For the 3-step job-stream that I have written, I have set the //SYSUT2 DD statement to point to the logs. This means that, the outputs of //STEP01 and //STEP03 shall be jotted down in the logs. The first step prints/copies the contents of the Unsorted Input file to the log. The second step

These are the contents of the Unsorted Input File, written to the log -

Image136[1]

These are the contents of the Sorted Output, written to the log. Click here to see the Sorted-Output in Text-Format. 

Image137[1]

Monday, February 1, 2010

Multi-step Job Streams

Q. Can Job run just one program? What if I want the output of one program, to be fed as input to a second program?
Yes, your intuition is absolutely correct. A job can have many(multiple) steps, each step runs a program. The output of 1st step can be fed to the 2nd step, the output of 2nd step can be fed to the 3rd step and so on..

You already know the general pattern of any Job. It has a JOB statement at the beginning that describes the job, EXEC to run the program, followed by DD Statements that represent the files/datasets, from which the program accesses data. It looks like this -

//name JOB parameters
//name EXEC parameters
//name DD parameters..
//name DD parameters..
//name DD parameters..

The above was a very simple, single-step Job. However, in most production environments in real Projects, a Job has multiple steps. Each step runs a program, that reads input data, processes it, and produces some output. Thus, typically the job would be a stream, a inter-linked chain of several steps.

The best way to understand a multi-step job-stream is to think of an Indian railways train.

Image107[1] 

The JOB Statement is the locomotive engine. Each program to run is inside a train compartment, and the last bogie/compartment would be null card – indicating the end of the JOB Train.

In this job train, each step runs a program, that receives input feed from the preceding step(step before that), processes it, and gives an output, which is fed to the next succeeding step. This way, the data is processed step-by-step, and you arrive at the final output, once the last step is complete.
Q. Vow, that was fun! Tell me more about these multi-step jobs.
Well, to learn more about multi-step jobs, you need to take some existing jobs and analyse them, how they work. This would give you a far better view of how multi-step jobs work.

But, first to understand any multi-step job, you need to know how to draw JCL flow-charts.

I am going to take a 3-step job, for the sake of example. I have given the skeleton of this 3-step job below. You can download this entire three-step Toy-Job and try it at home by clicking here.
Image108[1] 
If you can graphically draw what a JCL would do(does), it helps a lot to understand the JCL. Now, each step is gonna read one/more datasets, do some processing, and produce reports and write output to one/more datasets. First let me draw a simple diagram for the above JCL Skeleton.

Image109[1] 
Now, you can enrich this basic diagram by adding some more flowchart symbols, as shown below -

Image110[1]
You must be familiar with flowcharting symbols such as decision-box and loop statements. However, understand that JCL is not a programming language, and hence you do not have such symbols out here.
Q. Often, I see a lot of these DD * and DD DUMMY Statements. Could you just give me a crash course, before telling me more about job-streams?
Before you proceed ahead, here’s a crash course on two parameters that you often code on DD Statement – * and DUMMY.

1. DD * - What if you wanted to give special commands(instructions), to IEBGENER. Like "Copy only those employee records whose name starts with A". Note, that this is not data, so you don’t supply it in a file. These are commands, or instructions(orders) to IEBGENER telling it what records to copy. Such direct commands or instructions to the EXEC PGM program, that influence(control) the way the PGM works are called Control Instructions

Usually, the custom adopted is to hard-code(or embed) the control instructions in the body of the JCL itself(instead of putting them in a separate file). When you want to directly hard-code any stuff in JCL,  you must use *(wild-card) as a parameter on the DD Statement.

//name DD *
Control instruction 1 -
Control instruction 2 -
...
/*

You indicate the end of the control instructions by putting /* statement. 

2. DD DUMMY – What if you wanted to shut off the input fed to a particular Program. As an example, you don’t want to supply any input data file to IEBGENER. Shutting off input or output isn’t that easy, because IEBGENER would expect you to write //SYSUT1 DD Statement. Omitting //SYSUT1 isn’t an option(that would be a JCL Error). You can’t do away with it, just like that. So, you code a DUMMY parameter on the DD Statement. DUMMY implies, that this is just a DUMMY input or output file. In other words, that input/output feed is shut off(you have just sealed it off!).

DD * and DD DUMMY should come in handy while understanding a complete job-stream.
Q. Can you put together piece-by-piece, how the flow of a 3-step job looks like?
Alright friends, we shall go about it like this. We are first going to construct (pieces)blocks one-by-one, and them put them all together, assemble them to build a complete job.

The toy job shown here does the following
1. STEP01 – Read EMPLOYEE Data stored in a disk file(AGY0157.EMPLOYEE.INPUT) and print a list of all employees to the log for information purposes.
2. STEP02 - Arrange the records of the EMPLOYEE Data file in alphabetical order i.e. sort the EMPLOYEEs data
3. STEP03 – Print a report of the sorted output EMPLOYEE Data to the logs.

The first Step01 of the toy job, runs IEBGENER.
a) IEBGENER expects to read the input file at //SYSUT1 DD Statement.
b) IEBGENER copies the contents, and prints it to the dataset represented by //SYSUT2 DD Statement. We would like the input file’s contents are printed to the log just for informative purposes(so it helps tracking), //SYSUT2 file must point to the log(code the parameter SYSOUT=*).
c) IEBGENER expects to receive control instructions, at the //SYSIN DD Statement. Since, we are not gonna issue any control instruction, and want this functionality turned off, you can code a DUMMY parameter on the //SYSIN DD Statement.
d) Also, MVS Operating System, expects to write messages about the execution of the program at the dataset represented by //SYSPRINT DD Statement. Usually, you would want these messages to be printed to the log, so I would code SYSOUT=* as a parameter on the //SYSPRINT DD Card.

Okay, with that background, let me show you the JCL for the first step STEP01.

Image111[1] 
The second step STEP02 of the toy job, runs SORT. Essentially, SORT utility takes the input file records, and arranges them in order(in sequence), ascending or descending.

Image112[1]

1. SORT pgm. expects to reads records from the //SORTIN DD Statement. The //SORTIN DD Statement points to the File – AGY0157.INPUT.DATA.
2. SORT Pgm. sorts the input records in this file, as per the special control instructions(commands) that are supplied in the //SYSIN DD statement. I have instructed/commanded SORT Pgm, to apply sort on the field starting from columns 1 to 4, the data must be treated as a textual string(Character), and it should be sorted in ascending order(A).
SORT FIELDS=(1,4,CH,A)

For your reference, the input file’s records are shown below.

Image114[1] 

3. SORT Pgm. writes the sorted(ordered) output to the file represented by the //SORTOUT DD Statement. Here, I’ve created a new file, to which the sorted output records are written. This is one of the most difficult category of DD Statements to compose. Instead of specifying an actual output file-name, the &&OUTPUT on the SYSOUT DD Statement indicates a temporary file. This temporary output file, will be deleted, after the job completes.

4. Once again, the MVS Operating System, generates a report at how the SORT PGM ran, at //SYSOUT DD Statement. We would write this messages to log, by coding SYSOUT=* Parameter.

The third step STEP03 of the toy job, runs IEBGENER, a second time.
1. IEBGENER reads the input from the //SYSUT1 file – this is temporary output file &&OUTPUT containing the Sorted Output Records from the preceding Step STEP02. Thus, temp output file of the previous step is picked up as input for the next step.

Image115[1]

2. IEBGENER copies these sorted records from the temporary file, to the //SYSUT2 DD File. The //SYSUT2 DD Statement points to the log - SYSOUT=*. Thus, the list of sorted records are written to the log.

3. Since, I have not given any special commands or instructions to IEBGENER, I have turned off the //SYSIN DD functionality, by coding DUMMY Parameter on it.

4. The report generated MVS about the execution of IEBGENER, is written to the file represented by //SYSPRINT DD Statement. I have set //SYSPRINT DD to point to the log(SYSOUT=*). As a consequence, the Status Report of MVS, of how the IEBGENER PGM ran, is printed to the log.

The crux of this entire job-stream is -
- It picks up unsorted,un-arranged input records from the file AGY0157.INPUT.DATA.
- It sorts, arranges the data in ascending order, on the key field from
cols 1-4.
- It writes the sorted, ordered output records to the Log.

Putting it all together, and assembling, the below flow-diagram gives a gist of the entire job-stream.

Image116[1] 

To many people who are thrown to work at a mainframe computer on their first job, they feel lost. Mainframe people seem to speak a completely different language and that doesn't make life easy. What's more, the books and manuals are incredibly hard to comprehend.

"What on earth is a Mainframe?" is an absolute beginner's guide to mainframe computers. We'll introduce you to the hardware and peripherals. We'll talk about the operating system, the software installed on a mainframe. We'll also talk about the different people who work on a mainframe. In a nutshell, we'll de-mystify the mainframe.

Readers based in India, can buy the e-book for Rs. 50 only or the print book. International readers based in the US and other countries can click here to purchase the e-book.