Mainframes 360
The one stop destination for System Z professionals

Monday, February 1, 2010

Multi-step Job Streams

Q. Can Job run just one program? What if I want the output of one program, to be fed as input to a second program?
Yes, your intuition is absolutely correct. A job can have many(multiple) steps, each step runs a program. The output of 1st step can be fed to the 2nd step, the output of 2nd step can be fed to the 3rd step and so on..

You already know the general pattern of any Job. It has a JOB statement at the beginning that describes the job, EXEC to run the program, followed by DD Statements that represent the files/datasets, from which the program accesses data. It looks like this -

//name JOB parameters
//name EXEC parameters
//name DD parameters..
//name DD parameters..
//name DD parameters..

The above was a very simple, single-step Job. However, in most production environments in real Projects, a Job has multiple steps. Each step runs a program, that reads input data, processes it, and produces some output. Thus, typically the job would be a stream, a inter-linked chain of several steps.

The best way to understand a multi-step job-stream is to think of an Indian railways train.

Image107[1] 

The JOB Statement is the locomotive engine. Each program to run is inside a train compartment, and the last bogie/compartment would be null card – indicating the end of the JOB Train.

In this job train, each step runs a program, that receives input feed from the preceding step(step before that), processes it, and gives an output, which is fed to the next succeeding step. This way, the data is processed step-by-step, and you arrive at the final output, once the last step is complete.
Q. Vow, that was fun! Tell me more about these multi-step jobs.
Well, to learn more about multi-step jobs, you need to take some existing jobs and analyse them, how they work. This would give you a far better view of how multi-step jobs work.

But, first to understand any multi-step job, you need to know how to draw JCL flow-charts.

I am going to take a 3-step job, for the sake of example. I have given the skeleton of this 3-step job below. You can download this entire three-step Toy-Job and try it at home by clicking here.
Image108[1] 
If you can graphically draw what a JCL would do(does), it helps a lot to understand the JCL. Now, each step is gonna read one/more datasets, do some processing, and produce reports and write output to one/more datasets. First let me draw a simple diagram for the above JCL Skeleton.

Image109[1] 
Now, you can enrich this basic diagram by adding some more flowchart symbols, as shown below -

Image110[1]
You must be familiar with flowcharting symbols such as decision-box and loop statements. However, understand that JCL is not a programming language, and hence you do not have such symbols out here.
Q. Often, I see a lot of these DD * and DD DUMMY Statements. Could you just give me a crash course, before telling me more about job-streams?
Before you proceed ahead, here’s a crash course on two parameters that you often code on DD Statement – * and DUMMY.

1. DD * - What if you wanted to give special commands(instructions), to IEBGENER. Like "Copy only those employee records whose name starts with A". Note, that this is not data, so you don’t supply it in a file. These are commands, or instructions(orders) to IEBGENER telling it what records to copy. Such direct commands or instructions to the EXEC PGM program, that influence(control) the way the PGM works are called Control Instructions

Usually, the custom adopted is to hard-code(or embed) the control instructions in the body of the JCL itself(instead of putting them in a separate file). When you want to directly hard-code any stuff in JCL,  you must use *(wild-card) as a parameter on the DD Statement.

//name DD *
Control instruction 1 -
Control instruction 2 -
...
/*

You indicate the end of the control instructions by putting /* statement. 

2. DD DUMMY – What if you wanted to shut off the input fed to a particular Program. As an example, you don’t want to supply any input data file to IEBGENER. Shutting off input or output isn’t that easy, because IEBGENER would expect you to write //SYSUT1 DD Statement. Omitting //SYSUT1 isn’t an option(that would be a JCL Error). You can’t do away with it, just like that. So, you code a DUMMY parameter on the DD Statement. DUMMY implies, that this is just a DUMMY input or output file. In other words, that input/output feed is shut off(you have just sealed it off!).

DD * and DD DUMMY should come in handy while understanding a complete job-stream.
Q. Can you put together piece-by-piece, how the flow of a 3-step job looks like?
Alright friends, we shall go about it like this. We are first going to construct (pieces)blocks one-by-one, and them put them all together, assemble them to build a complete job.

The toy job shown here does the following
1. STEP01 – Read EMPLOYEE Data stored in a disk file(AGY0157.EMPLOYEE.INPUT) and print a list of all employees to the log for information purposes.
2. STEP02 - Arrange the records of the EMPLOYEE Data file in alphabetical order i.e. sort the EMPLOYEEs data
3. STEP03 – Print a report of the sorted output EMPLOYEE Data to the logs.

The first Step01 of the toy job, runs IEBGENER.
a) IEBGENER expects to read the input file at //SYSUT1 DD Statement.
b) IEBGENER copies the contents, and prints it to the dataset represented by //SYSUT2 DD Statement. We would like the input file’s contents are printed to the log just for informative purposes(so it helps tracking), //SYSUT2 file must point to the log(code the parameter SYSOUT=*).
c) IEBGENER expects to receive control instructions, at the //SYSIN DD Statement. Since, we are not gonna issue any control instruction, and want this functionality turned off, you can code a DUMMY parameter on the //SYSIN DD Statement.
d) Also, MVS Operating System, expects to write messages about the execution of the program at the dataset represented by //SYSPRINT DD Statement. Usually, you would want these messages to be printed to the log, so I would code SYSOUT=* as a parameter on the //SYSPRINT DD Card.

Okay, with that background, let me show you the JCL for the first step STEP01.

Image111[1] 
The second step STEP02 of the toy job, runs SORT. Essentially, SORT utility takes the input file records, and arranges them in order(in sequence), ascending or descending.

Image112[1]

1. SORT pgm. expects to reads records from the //SORTIN DD Statement. The //SORTIN DD Statement points to the File – AGY0157.INPUT.DATA.
2. SORT Pgm. sorts the input records in this file, as per the special control instructions(commands) that are supplied in the //SYSIN DD statement. I have instructed/commanded SORT Pgm, to apply sort on the field starting from columns 1 to 4, the data must be treated as a textual string(Character), and it should be sorted in ascending order(A).
SORT FIELDS=(1,4,CH,A)

For your reference, the input file’s records are shown below.

Image114[1] 

3. SORT Pgm. writes the sorted(ordered) output to the file represented by the //SORTOUT DD Statement. Here, I’ve created a new file, to which the sorted output records are written. This is one of the most difficult category of DD Statements to compose. Instead of specifying an actual output file-name, the &&OUTPUT on the SYSOUT DD Statement indicates a temporary file. This temporary output file, will be deleted, after the job completes.

4. Once again, the MVS Operating System, generates a report at how the SORT PGM ran, at //SYSOUT DD Statement. We would write this messages to log, by coding SYSOUT=* Parameter.

The third step STEP03 of the toy job, runs IEBGENER, a second time.
1. IEBGENER reads the input from the //SYSUT1 file – this is temporary output file &&OUTPUT containing the Sorted Output Records from the preceding Step STEP02. Thus, temp output file of the previous step is picked up as input for the next step.

Image115[1]

2. IEBGENER copies these sorted records from the temporary file, to the //SYSUT2 DD File. The //SYSUT2 DD Statement points to the log - SYSOUT=*. Thus, the list of sorted records are written to the log.

3. Since, I have not given any special commands or instructions to IEBGENER, I have turned off the //SYSIN DD functionality, by coding DUMMY Parameter on it.

4. The report generated MVS about the execution of IEBGENER, is written to the file represented by //SYSPRINT DD Statement. I have set //SYSPRINT DD to point to the log(SYSOUT=*). As a consequence, the Status Report of MVS, of how the IEBGENER PGM ran, is printed to the log.

The crux of this entire job-stream is -
- It picks up unsorted,un-arranged input records from the file AGY0157.INPUT.DATA.
- It sorts, arranges the data in ascending order, on the key field from
cols 1-4.
- It writes the sorted, ordered output records to the Log.

Putting it all together, and assembling, the below flow-diagram gives a gist of the entire job-stream.

Image116[1] 

To many people who are thrown to work at a mainframe computer on their first job, they feel lost. Mainframe people seem to speak a completely different language and that doesn't make life easy. What's more, the books and manuals are incredibly hard to comprehend.

"What on earth is a Mainframe?" is an absolute beginner's guide to mainframe computers. We'll introduce you to the hardware and peripherals. We'll talk about the operating system, the software installed on a mainframe. We'll also talk about the different people who work on a mainframe. In a nutshell, we'll de-mystify the mainframe.

Readers based in India, can buy the e-book for Rs. 50 only or the print book. International readers based in the US and other countries can click here to purchase the e-book.