Waiting for a file to arrive before a job can start processing it is the most common and quintessential requirement for any workload automation tool.  Historically, this was done by creating a File dependency using OPENS and Unix tricks to manage wild cards and multiple matching files.  Then Event Driven Workload Automation was introduced where an Event Rule could monitor a file with wild cards and when the condition was satisfied the dependent Job Stream was submitted.  With the advent of the Start Condition feature introduced in version 9.4 Fix Pack 1, the event rule file monitor capability is integrated into a Job Stream using the utility, filemonitor.  Another important aspect of this requirement is to pass the name of the matching file to another job in the workflow to process the data contained in the file.

This article explores this feature and also uses an internal variable, ListOfFilesFound, to pass the name of the matching file to a successive job in the same Job Stream.

How to use the Start Condition feature with File Created

Follow the steps below to create a Start Condition for a Job Stream

  1. Login to the Dynamic Workload Console
  2. Navigate to Design à Manage Workload Definitions
  3. Select Create New à Job Stream
  4. In the General tab, fill in the Job Stream name and Workstation
  5. In the Start condition tab, fill in the values in the fields as shown below:
    • Condition: File created              An existing file with a matching name will not satisfy the condition, a new file with a matching name must be created.
    • Workstation: EEL
    • File: /tmp/start.cond.*.txt
    • Output file: /tmp/start.cond.out.txt
    • Job name: FILE_CREATED       A file monitor job with this name is auto generated. This job iteratively keeps monitoring the file and when satisfied, submits a new                                                        instance of the Job Stream.

 

 

6. Select Create New à Job Definition à UNIX

7. In the General tab, fill in the Job name, Workstation, and Login

8. In the Task tab, enter the following

Script name: /bin/echo “File Monitored: ${job:FILE_CREATED.ListOfFilesFound}”

9. Click on Save

10. Select Create New à Job Definition à UNIX

11. In the General tab, fill in the Job name, Workstation, and Login

12. In the Task tab, enter the following

Script name: cat /tmp/start.cond.out.txt

13. Click on Save

14. Add the two jobs created earlier to the Job Stream

15. Click on Save

16. The following is the Job Stream definition is composer format.

SCHEDULE EEL#START_COND_JS

STARTCOND FILECREATED EEL#”/tmp/start.cond.*.txt” INTERVAL 60

( ALIAS FILE_CREATED RERUN OUTFILE “/tmp/start.cond.out.txt” )

:

EEL#FILE_NAME_FROM_VARIABLE

SCRIPTNAME “/bin/echo \”File Monitored: ${job:FILE_CREATED.ListOfFilesFound}\””

STREAMLOGON iwadmin

DESCRIPTION “Retrieve file name in Start Cond from an internal variable, ListOfFilesFound”

TASKTYPE UNIX

RECOVERY STOP

 

EEL#FILE_NAME_FROM_OUTPUT_FILE

SCRIPTNAME “cat /tmp/start.cond.out.txt

STREAMLOGON iwadmin

DESCRIPTION “Retrieve file name in Start Cond from the Output file field”

TASKTYPE UNIX

RECOVERY STOP

 FOLLOWS FILE_NAME_FROM_VARIABLE

 

END

17. Select the Job Stream Name and choose Select an Action à Submit Job Stream into Current Plan

Monitor the Status of Job Stream with the Start Condition and File Created

Follow the steps below to monitor the status of the Job Stream

  1. Login to the Dynamic Workload Console
  2. Navigate to Monitoring and Reporting à Monitor Workload
  3. Choose Job Stream for Object Type and click on Run
  4. In the Filter window, enter Start_cond and press Enter
  5. In the resulting view, Right Click on the Job Stream, START_COND_JS, and choose Jobs
  6. Notice that there are 4 jobs in the Job Stream

FILE_CREATED                                 Auto generated job to keep monitoring the file

RESTART_STARTCOND                     Auto generated job to submit the Job Stream

FILE_NAME_FROM_VARIABLE         Job to retrieve the matching file from a variable

FILE_NAME_FROM_OUTPUT_FILE    Job to retrieve the matching file from an output file

 

 

7. Left click on the Job, FILE_CREATED, to view its properties. Click on Extra Properties tab to view the internal variable that stores the name of the matching file.

8. Right click on the Job, FILE_NAME_FROM_VARIABLE, and choose Job Log. Notice that the name of the matching file is retrieved from the internal variable, ListOfFilesFound, set by the job,

FILE_CREATED, ${job:FILE_CREATED.ListOfFilesFound}.

File Monitored: /tmp/start.cond.092920201352.txt

 

9. Right click on the Job, FILE_NAME_FROM_OUTPUT_FILE, and choose Job Log. Notice that the name of the matching file is retrieved from the file name specified in the Output file field,

cat /tmp/start.cond.out.txt

/tmp/start.cond.092920201352.txt

 

 

 

How to use the Start Condition feature with File Modified

Follow the steps below to create a Start Condition for a Job Stream

  1. Login to the Dynamic Workload Console
  2. Navigate to Design à Manage Workload Definitions
  3. Select Create New à Job Stream
  4. In the General tab, fill in the Job Stream name and Workstation
  5. In the Start condition tab, fill in the values in the fields as shown below:
    • Condition: File modified                                                                An existing file with a matching name will not satisfy the condition, the file with a matching name                                                                                                             must be modified.
    • Workstation: EEL
    • File: /tmp/start.cond.*.txt
    • Output file: /tmp/startcond.out                                                     Must have a different pattern than the file being monitored.
    • Additional Parameters: -modificationCompletedTime 120         When a file is modified, the event is not sent immediately, but only after the interval of time                                                                                                                     specified by -modificationCompletedTime <seconds> has elapsed and during which no additional                                                                                                           changes were made to the file.  For other parameters, refer to Chapter 16, Using utility commands,                                                                                                         in the IBM Workload Scheduler version 9.5 User’s Guide and Reference.
    • Job name: FILE_MODIFIED                                                          A file monitor job with this name is auto generated.  This job iteratively keeps monitoring the file                                                                                                             and when satisfied, submits a new instance of the Job Stream

 

6. Select Create New à Job Definition à UNIX

7. In the General tab, fill in the Job name, Workstation, and Login

8. In the Task tab, enter the following

Script name: /bin/echo “File Monitored: ${job:FILE_MODIFIED.ListOfFilesFound}”

 

9. Click on Save

10. Select Create New à Job Definition à UNIX

11. In the General tab, fill in the Job name, Workstation, and Login

12. In the Task tab, enter the following

Script name: cat /tmp/startcond.out

13. Click on Save

14. Add the two jobs created earlier to the Job Stream

15. Click on Save

16. The following is the Job Stream definition is composer format.

SCHEDULE EEL#START_COND_JS

STARTCOND FILEMODIFIED EEL#”/tmp/start.cond.*.txt” INTERVAL 60

( ALIAS FILE_MODIFIED RERUN PARAMS “-modificationCompletedTime 120” OUTFILE “/tmp/startcond.out” )

:

EEL#FILE_NAME_FROM_VARIABLE

SCRIPTNAME “/bin/echo \”File Monitored: ${job:FILE_MODIFIED.ListOfFilesFound}\””

STREAMLOGON iwadmin

DESCRIPTION “Retrieve file name in Start Cond from an internal variable, ListOfFilesFound”

TASKTYPE UNIX

RECOVERY STOP

 

EEL#FILE_NAME_FROM_OUTPUT_FILE

SCRIPTNAME “cat /tmp/startcond.out

STREAMLOGON iwadmin

DESCRIPTION “Retrieve file name in Start Cond from the Output file field”

TASKTYPE UNIX

RECOVERY STOP

FOLLOWS FILE_NAME_FROM_VARIABLE

 

END

17. Select the Job Stream Name and choose Select an Action à Submit Job Stream into Current Plan

Monitor the Status of Job Stream with the Start Condition with File Modified

Follow the steps below to monitor the status of the Job Stream

  1. Login to the Dynamic Workload Console
  2. Navigate to Monitoring and Reporting à Monitor Workload
  3. Choose Job Stream for Object Type and click on Run
  4. In the Filter window, enter Start_cond and press Enter
  5. In the resulting view, Right Click on the Job Stream, START_COND_JS, and choose Jobs
  6. Notice that there are 4 jobs in the Job Stream

FILE_MODIFIED                               Auto generated job to keep monitoring the file

RESTART_STARTCOND                     Auto generated job to submit the Job Stream

FILE_NAME_FROM_VARIABLE         Job to retrieve the matching file from a variable

FILE_NAME_FROM_OUTPUT_FILE    Job to retrieve the matching file from an output file

7. Left click on the Job, FILE_CREATED, to view its properties. Click on Extra Properties tab to view the internal variable that stores the name of the matching file.

8. Right click on the Job, FILE_NAME_FROM_VARIABLE, and choose Job Log. Notice that the name of the matching file is retrieved from the internal variable, ListOfFilesFound, set by the job,

FILE_CREATED, ${job:FILE_MODIFIED.ListOfFilesFound}.

File Monitored: /tmp/start.cond.092920201352.txt

 

 

9. Right click on the Job, FILE_NAME_FROM_OUTPUT_FILE, and choose Job Log. Notice that the name of the matching file is retrieved from the file name specified in the Output file field,

cat /tmp/start.cond.out.txt

/tmp/start.cond.092920201352.txt

 

Want to learn more? Write me at Sajjad.Kabir@hcl.com or schedule a demo.

Author’s BIO

Sajjad Kabir, Solutions Architect, HCL Software

Sajjad Kabir is an Information Technology professional with over 25 years of diverse industry experience with primary emphasis in IT Architecture and Service Management. Extensive experiences in management, architecture, design, development, implementation, and systems integration solutions in multiple industries, platforms, and network environments.

View my profile on LinkedIn

 

 

Comment wrap
Further Reading
article-img
Automation | June 20, 2022
The Dynamic Workload Console is the one-stop automation platform for users across the business
The Dynamic Workload Console (DWC) has become a core platform for workload automation, providing visibility into everything all in one place.“The designing of a job stream is a key operation for schedulers and application developers to or interconnect business applications and achieve governance and control,” Zaccone said. “Our idea with the new Workload Designer is to empower what we had and push it to an advanced level to provide everything is needed to our customers.” 
article-img
Automation | May 24, 2022
Ensuring Passwordless Job Schedulation with CyberArk Integration
CyberArk is an identity and access manager offering a wide set of identity security capabilities. You can use it to submit Workload Automation jobs without specifying any passwords, which are retrieved from the CyberArk vault.
article-img
Automation | May 19, 2022
Continuous Security and Quality Updates on Dynamic Workload Console 10.1
After the biggest launch of Workload Automation 10.0.1 release in 1Q of 2022 (see the Workload automation original Video), what can we expect in 2022? Big news! Our first release refreshing for Dynamic Workload Console 10.0.1 is ready. Let’s answer the 5 WH questions.
Close
Filters result by
Sort:
|