UNICORE Workflow service manual

This chapter covers basic installation of the Workflow engine and the integration of the workflow services into an existing UNICORE system.

As a general note, the Workflow engine is based on a UNICORE/X instance. General UNICORE configuration concepts (such as user authentication, gateway integration, shared registry, attribute sources) fully apply, and you should refer to the UNICORE/X manual for such details.

If you update from 7.x, please note that it is a major update, and we suggest installing from scratch based on the template config files. The required changes are very similar to the UNICORE/X 7.x to 8.x update.

The workflow system is available either as part of the UNICORE Server Bundle, or as separate Linux packages (deb or rpm).

The basic installation procedure is completely analogous to the installation of a UNICORE/X server.

If you downloaded the UNICORE server bundle, untar the tar.gz, edit configure.properties and run configure.py

If using the Linux packages, simply install using the package manager of your system and review the config files.

After installation, there are some manual steps needed to integrate the new server into your UNICORE installation.

Using the UNICORE commandline client, you can check whether the new server is available and accessible:

should include output such as

The "authenticated as:" line should list you as "user".

Some more info about the server can be obtained via

Using UCC again, you can submit workflows

and get the ID of your new workflow back, e.g.

Since version 8.0, the Workflow engine exclusively uses a RESTful API for all operation including job submission.

You can find an API reference and usage examples on the UNICORE Wiki at https://sourceforge.net/p/unicore/wiki/REST_API

This chapter covers configuration options for the Workflow server. Since the Workflow server is running in the same underlying environment (UNICORE Services Environment, USE), a lot of the basic configuration options are documented in the UNICORE/X manual.

NOTE

Depending on how you installed the server, the files are located ion

Some details of the workflow engine’s behaviour can be configured. All these settings are made in uas.config.

To avoid too many tasks submitted (possibly erroneously) from a workflow, various limits can be set.

The workflow engine will (in some cases) resubmit failed tasks to the service orchestrator. To completely switch off the resubmission,

To change the maximum number of resubmissions from the default "3",

This controls the behaviour when a workflow is removed (automatically or by the user). By default, the workflow engine will remove all child jobs, but will keep the storage where the files are. This can be controlled using two properties

The workflow engine uses the XNJS library for processing workflows. Some settings for modifying the behaviour are available, and are usually found in the workflow server’s container.properties file

An important characteristic is the number of threads used by the workflow engine for processing.Note, this does not control the number of concurrent activities etc, since all XNJS processing is asynchronous. The default number (4) is usually fine.

This properties is set via

A complete reference of the properties for configuring the Workflow server is given in the following table.

This chapter provides an overview of the JSON workflow description that is supported by the Workflow engine. It will allow you to write workflows "by hand", i.e. without using tools such as the Java or Python APIs.

After presenting all the constructs individually, several complete examples are given in [wf_examples].

The overall workflow document has the following form:

Two special elements are

Both of these are analogous to their conterparts for single jobs in UNICORE.

In the next sections the elements of the workflow description will be discussed in detail.

Activity elements have the following form

The id element must be UNIQUE within the workflow. There are different types of activity, which are distinguished by the "type" element.

The workflow description allows nested sub workflows, which have the same formal structure as the main workflow (without the "tags"). There is an additional "type" element that is used to distinguish the different control structure types.

Job activities are the basic executable pieces of a workflow. The embedded JSON job definition will be sent to an execution site (UNICORE/X) for processing.

The execution site is specified by the optional "Site name" element in the job

NOTE that there is currently no form of "brokering" in place, it is up to the user to select an execution site

The job description is covered in detail here: https://sourceforge.net/p/unicore/wiki/Job_Description

The processing of the job can be influenced using the (optional) "option" sub-element. Currently the following options (key-value) can be used

For example

If you need to pass on user preferences to the site, e.g. for selecting your primary group, or choosing between multiple user IDs, you can specify this in the "job" element like this

where the allowed field names are "role", "uid", "group" and "supplementaryGroups".

Data management is one of the most important aspects of workflow definition and execution.

The Workflow service offers an additional tool for referencing data that is produced and consumed during workflow execution, which is workflow files.

A "workflow file" is a named reference to a physical file location, that can be generated at at any time during a workflow’s lifetime.

Such a reference is an URI of the form wf:… which you can use in a job’s "Imports" section or in For-Each file sets (see below).

In a Job you can use them as if they were a normal remote file:

The Workflow engine will resolve these at runtime, and the job will thus import the file at the registered physical location.

There are several ways to define these workflow files.

You can define inputs as part of your workflow

(see the UCC manual for an example how a client might make use of this feature)

If your job / workflow step produces output, you can register these output files as workflow files, simply by "exporting" them.

The Workflow service will not move the file, just register its location under the "wf:…" name, so you can reference it later.

The workflow file registration and resolution mechanisms support wildcards.

For example

The basic flow of control in a workflow is handled using transition elements. These reference from and to activities or subflows, and may have conditions attached. If no condition is present, the transition is followed unconditionally, otherwise the condition is evaluated and the transition is followed only if the condition matches (i.e. evaluates to true).

The syntax for a Transition is as follows.

The from and to elements denote activity or subworkflow id’s.

An activity can have outgoing (and incoming) transitions. In general, all outgoing transitions (where the condition is fulfilled) will be followed. The exception is the "Branch" activity, where only the first matching transition will be followed.

The optional condition element is a string-valued expression. The workflow engine offers some pre-defined functions that can be used in these expressions. For example you can use the exit code of a job, or check for the existence of a file within these expressions.

Workflow variables need to be declared using an entry in the variables array before they can be used.

Currently variables of type "STRING", "INTEGER" , "FLOAT" and "BOOLEAN" are supported.

Variables can be modified using an activity of type MODIFY_VARIABLE.

For example, to increment the value of the "COUNTER" variable, the following Activity is used

The "expression" contains an expression in Groovy syntax (which is very close to Java).

The workflow engine will replace variables in job data staging sections and environment definitions, allowing to inject variables into jobs. Examples for this mechanism will be given in the examples section.

Apart from graphs constructed using activity and transition elements, the workflow system supports special looping constructs, for-each, while and repeat-until, which allow to build complex workflows.

These allow to loop a certain part of the workflow while (or until) a condition is met. A while loop looks like this

The necessary ingredients are that the loop’s "body" modifies the loop variable ("C" in the example), and the exit condition eventually terminates the loop.

Completely analogously, a repeat-until loop is constructed, the only syntactic difference is that the subworkflow now has a different type element:

Semantically, the repeat-loop will always execute the body at least once, since the condition is checked after executing the body, while in the "while" case, the condition will be checked before executing the body.

The for-each loop is a complex and powerful feature of the workflow system, since it allows parallel execution of the loop body, and different ways of building the different iterations. Put briefly, one can loop over variables (as in the "while" and "repeat-until" case), but one can also loop over enumerated values and (most importantly) over file sets.

The basic syntax is

The iterator_name element allows to control how the "loop iterator variable" is to be called, by default it is named "IT".

Using value, iteration over a fixed set of strings can be defined. The main use for this is parameter sweeps, i.e. executing the same job multiple times with different arguments or environment variables.

In each iteration, the workflow variables "CURRENT_ITERATOR_VALUE" and "CURRENT_ITERATOR_INDEX" will be set to the current value and index.

The variables element allows to define the iteration range using one or more variables, similar to a for-loop in a programming language.

The sub-elements should be self-explanatory.

Note that you can use more than one variable range, allowing you to quickly create things like parameter studies,

This variation of the for-each loop, allows to loop over a set of files, optionally chunking together several files in a single iteration.

The basic structure of a file set definition is this

The base element defines a base of the filenames, which will be resolved at runtime, and complemented according to the include and/or exclude elements. The recurse attribute allows to control whether the resolution should be done recursively into any subdirectories. The indirection attribute is explained below.

For example to recursively collect all PDF files (except two files named "unused*.pdf") in a certain directory on a storage:

Of course you can use workflow files in the file sets as well. For example:

The following variables are set where ITERATOR_NAME is the loop iterator_name defined in the for group as shown above.

Sometimes the list of files that should be looped over is not known at workflow design time, but will be computed at runtime. Or, you wish simply to list the files in a file, and not put them all in your workflow description. The indirection attribute on a FileSet allows to do just that. If indirection is set to true, the workflow engine will load the given file(s) in the fileset at runtime, and read the actual list of files to iterate over from them. As an example, you might have a file filelist.txt containing a list of UNICORE file URLs:

and the fileset

You can have more than one file list.

Chunking allows to group sets of files into a single iteration, for example for efficiency reasons.

A chunk is either a certain number of files, or a set of files with a certain total size.

The chunksize element is either the number of files in a chunk, or (if type is set to "SIZE") the total size of a chunk in kbytes.

For example:

To process 10 files per iteration:

To process 2000 kBytes of data per iteration:

The chunksize can also be computed at runtime using the expression given in the optional expression element. In the expression, two special variables may be used. The TOTAL_NUMBER variable holds the total number of files iterated over, while the TOTAL_SIZE variable holds the aggregated size of all files in kbytes. The script must return an integer-valued result. The type element is used to choose whether the chunk size is interpreted as number of files or data size.

For example:

To choose a larger chunksize if a certain total file size is exceeded:

The optional filename_format allows to control how the individual files (which are staged into the job directory) should be named. By default, the index is prepended, i.e. with

"1_inputfile.txt" to "N_inputfile.txt" in each chunk. The pattern uses the without extension and the extension respectively.

For example, if you have a set of PDF files, and you want them to be named "file_1.pdf" to "file_N.pdf", you could use the pattern

which would ignore the name in the "To" field completely.

Or, if you prefer to keep the existing name, but append an index to the name,

which would result in filenames like below:

You can also keep the original filenames by setting:

This section collects a few simple example workflows. They are intended to be submitted using UCC.

This example shows how to use transitions for building simple workflow graphs. It consists of four "Date" jobs arranged in a diamond shape, i.e. "date2a" and "date2b" are executed roughly in parallel. A "Split" activity is inserted to divide the control flow into two parallel branches.

All "stdout" files are staged out to the workflow storage.

Transitions from one activity to another may be conditional, which allows all sorts of if-else constructs. Here is a simple example

Here we use the "BRANCH" activity which will only follow the first matching transition.

The next example shows some uses of workflow variables in a while loop. The loop variable "C" is copied into the job’s environment. Another possible use is to use workflow variables in data staging sections, for example to name files.

The next example shows how to use the for-each loop to loop over a set of files. The jobs will stage-in the current file. Also, the name of the current file is placed into the job environment.

This chapter covers the steps required to update an existing workflow installation (v8.x).

Stop the workflow server.

You should make a backup of your existing data, and, if necessary, your config files.

The Java libraries have to be replaced with the new versions.

Compare the config files from the new version to your existing one. Check the changelog for new features that might require updates to config files.

Restart the workflow server, and check the logs for any suspicious error messages!