HiLA User Guide

Highly dynamic features and security concerns often make it difficult for programmers to easily develop clients accessing Grids. There is a lot of programming overhead involved for each call to a Grid resource, although many of these tasks are structured similarly. A universal, high-level API can alleviate the need to perform the tedious and repetitive tasks of accessing Grid resources and let the developer focus on the purely functional aspects of programming. The extra amount of work necessary to access a site is hidden behind a concise interface.

Another advantage of a cleanly defined API is the opportunity to implement for multiple backend Grid environments. The HiLA API currently supports two implementations, one for UNICORE 5 and more recently another one for the Web services based UNICORE 6. An implementation for OGSA-BES is in a development state. Additional backend implementations are conceivable to support other Grid middlewares in the same uniform way.

Rather than designing a task focused API, HiLA takes a resource oriented approach to the definition of the Java interfaces. The linkages between the resources on the Grid are approached in a consistent manner. A resource is Locatable and possesses a Location. As such, there is a generic mechanism in place for navigating the linkages. For example, a Site has a number Storages, a Storage contains a number of Files and a Site references its running Tasks. So different underlying Grid infrastructures become browsable in a uniform way. Analysing and using Grid resources can then be as easy as browsing a UNIX file system.

The API uses a Factory mechanism to dynamically load particular implementations. Multiple implementations of the API can co-exist within the lifetime of a program execution, and it encourages late binding to a specific implementation, selectable at runtime. Location of the HiLA resources are based on URI’s, where the scheme is used to select a particular implementation.

This document describes the HiLA API from a user’s point of view. Of course, users of APIs are developers. There is a separate developer guide available for those, who want to develop additional implementations of the abstract interfaces.

Originally, HiLA was named Roctopus, which is why you will find references to this name as well [Hagemeier2007] [roctopus_cgw05].

HiLA is a convenient toolkit for building web interfaces or command line environments for Grids, and is used by the DESHL suite of tools developed in the JRA7 activity of the DEISA project. In the A-WARE project, we are using HiLA as the basis for a Grid agent component to be plugged into a service bus, which is then used as for building higher-level orchestration services for Grids.

HiLA is an API for accessing Grid resources of different middleware in a consistent manner. It has been implemented for UNICORE versions 5 and 6, and an OGSA-BES implementation is also available. HiLA is in use in various places in different Grid access libraries. The DEISA Services for Heterogeneous management Layer (DESHL) has been implemented using HiLA right from the start. This severely eased the shift to UNICORE 6, as HiLA provides the same interface for both middlewares. In the German AeroGrid project [aerogrid], a JavaGAT adaptor has been implemented on top of HiLA, thus enabling the use of UNICORE 5 and UNICORE 6 through GAT and this adaptor.

The key concept in HiLA is that every Resource can be referenced by a URI.

The generic URI structure in HiLA (for Grids) looks like this:

The <scheme> placeholder is the implementation specific part of the URI. In real scenarios, it will be something like unicore6, unicore5 or ogsa.

The actual URI is pattern matched against the patterns of registered ResourceTypes.

Staying with the Grid example, <site-name> will be the name of the site, as it has been discovered by the configuration of HiLA Grid API. <task-id> is the id of the task, which is generated by the middleware. This enables us to located particular tasks only from this reference. The same holds for <storage-name>, where all storages exposed by the middleware at the site under consideration will be valid values for <storage-name>.

This follows a regular pattern, where collections and their contents alternate in the URI structure. Collections are sites, tasks, storages, and files. While all of these do have equivelents in the Java interfaces, they are less important, because they do not offer mor functionality than any other Locatable artifact.

The concept of artifacts being locatable is generic and extensible. Which artifacts in the Grid can be referenced by HiLA, will be subject of the following sections.

As you can see, a Location has a locate() method, with which you can locate the Resource behind this Location. Of course, it is only this simple when dealing with generic Resources. Most of the time, however, you will want to deal with specific Resources, on which you can call certain actions. Have a look at the following example:

Another option would be (assuming loc is still the same):

A base set of Exceptions have been defined to represent certain error or exceptional conditions. All of them are derived from HiLAException.

The HiLA API with its notions of Location and Resource is very generic and merely allows to create a navigable hierarchy of Resources that have Locations.

The basic artifacts, which can be accessed by HiLA and referenced via HiLA locations, are:

The HiLA Architecture figure shows the interrelationships between the various artifacts. A Grid is made up of a number of sites, which in turn contain tasks and storages. Storages contain files, which can be imported to, exported from or transferred among storages. Tasks also have a storage, which is their working directory. One can thus access the working directory of a task at any time, if the underlying implementation supports it.

The UML class diagram shows the main classes within the abstract HiLA API. An implementation of HiLA has to implement the interfaces as appropriate and support the given operations. Operations on the interfaces will be described in the following.

Grid locations are structured like <scheme>:/. For example, a grid location for the UNICORE 6 implementation of HiLA would look like unicore6:/. As shown in the above figure and more specifically in the UML diagram, a Grid consists of an arbitrary number of Sites. You can query the Sites belonging to a Grid by calling the getAllSites() method, which returns a List of Sites. If you know the Location of an entity, to which you want to gain access, you can locate() it. The locate() method returns a Locatable, which you may have to cast to the appropriate type.

Site locations are structured like <scheme>:/sites/<site-name> A Site location for the UNICORE 6 implemenation looks like unicore6:/sites/Alpha-Site. Sites implement both the Locatable and Submitable interfaces. While Locatable has been explained before, the Submitable interface is used for anything that one can submit jobs to. Hence, it implements the methods submit(), getTasks(), and getTask().

Storage locations are structured like

<scheme>:/sites/<site-name>/storages/<storage-name>.

There is a special twist to Storage Locations, as a Task also has a Storage, where its execution takes place. Thus, a Storage Location can also look like

<scheme>:/sites/<site-name>/tasks/<task-name>/wd

where wd means working directory. Storages belonging to Sites can be retrieved by calling getStorages() on a Site object. From a Task, one can directly jump into the working directory by calling getWorkingDirectory() on the Task, which actually returns a File, but that is what you want anyway.

Examples of Storage Locations for the UNICORE 6 implementation would be

unicore6:/sites/Alpha-Site/storages/home and

unicore6:/sites/Alpha-Site/tasks/ \
79258434-f295-11dd-b58a-001f160cb28c/wd.

A Task is something that can run concurrently with the rest of the code, e.g. a file transfer or remote Job. Tasks can be monitored.

Task locations are structured like <scheme>:/sites/<site-name>/tasks/<task-name>. An example Task Location for the UNICORE 6 implementation looks like this unicore6:/sites/ \
Alpha-Site/tasks/79258434-f295-11dd-b58a-001f160cb28c.

One can register listeners on Tasks that listen for changes of its status.

A specialization of the generic Task is a Job, which has an exit code, outcome files, a working directory etc. Unlike ordinary Tasks, Jobs have a Storage representing the Jobs' working directory, which can be browsed like any other Storage.

Starting with version 2.2, Sites implement the Reservable interface, which allows for creating reservations based on ResourceDescriptions. Since not all Sites or even Grid middleware support this, it is very likely that calling the reserve method throws a HiLANotImplementedException or HiLANotSupportedException.

Jobs can be run inside Reservations.

File locations are structured like

<scheme>:/sites/<site-name>/storages/<storage-name>/files/<file-name>.

Again, as Tasks have associated Storages as well, and these Storages also contain Files, an alternative structure of a File’s location is

<scheme>:/sites/<site-name>/tasks/<task-name>/wd/files/<file-name>.

Example File Locations for the UNICORE 6 implementation look like the following:

unicore6:/sites/Alpha-Site/storages/home/files/.bashrc and

unicore6:/sites/Alpha-Site/tasks/\
79258434-f295-11dd-b58a-001f160cb28c/wd/files/stdout.

If you would like to develop using the HiLA API, you will most likely start with the Location of a Resource serving as an entry point into the Resource hierarchy of your problem domain. It could be something as simple as a pure scheme name appended with »:/«, such as unicore6:/.

In code, this would look something like this:

As you can see, you can start from a location and have it locate() the resource. Generally, the locate() method returns objects of type eu.unicore.hila.Resource or one of its decendants. You can already query the Location, if it points to a certain resource type. That way, it is not necessary to locate (and instantiate) the actual resource, if you already know that the Location points to the wrong resource type.

If you want to be even more sure about the type of Resource returned by locate(), you can check it using the instanceof operator.

However, this should be obsolete, if the implementation of the API does not misbehave.

Starting with version 2.3.0, HiLA supports a more sophisticated JobModel that is supposed to better support the multitude of job description languages that are used by different Grid middlewares. Before that version, the Submitable.submit() merely consumed an object that the implementation had to know what to do with.

The benefit of consuming a real abstraction of the job description, the job model, it is fully up to the implementation to map it to the correct description language. Before, it was partly up to the user (developer) to know what job description could be consumed by a particular implementation and thus to also be aware against which implementation the code was written.

While the old method still remains, but is deprecated, the new method should be used from now on. It will be described in the following.

The job model is centered around the JobModel class, which is an abstraction of common job description languages. The model itself is agnostic of any particular job description languages, i.e. JSDL or EMI ADL.

All Grid middlewares support the concept of running an explicit executable on the Grid site. This can be set in the job model as an executable. Some middlewares support the concept of applications that are mapped down to an executable with possibly additional arguments. These two concepts are treated as mutually exclusive by the job model implementation. Thus, setting an executable unsets an application that has previously been set and vice-versa.

Arguments are treated as a list of commandline arguments. The existing list can be extended by adding arguments. Another way to set arguments it to set the list in a single operation.

The environment is a list of environment variables and their corresponding values. Being a list, this is treated in the same way as arguments, one can either add additional variables or set the entire list in one operation.

Most job description languages only know about third-party transfers (as opposed to client-server transfers) for job submission. There’s a good reason for this, as the job description can be passed on to the server side as is and processed there.

On the other hand, it can be very handy to be able to refer to local files for copying them into the job’s working directory before the job starts and retrieving output files from the same directory after the job has finished.

The job model captures this situation by distinguishing these two scenarios. It provides StageIn and StageOut classes for the third-party scenario and Import and Export classes for client to server transfers.

…

The following table shows the relationships among the different implementations of HiLA APIs. At the base, there is hila-api, which is generic and does not know anything about Grid or other specific resources. It merely provides the basis for implementing resource types and giving them structured locations.

On the next level, there are hila-grid-api and hila-other-api, where the latter just represents any other API that is based on hila-api.

The UNICORE 6 implementation supports the service orchestrator, which is used as a broker to submit jobs to Sites matching the resource requirements given in the job description.

Configuration for this implementation is expected in the user’s home directory under
~/.hila2/unicore6.properties. The format of this file is the following

This configuration should be used whenever you can use your credentials directly, i.e. you have the keystore with your private key at hand. The next section will describe what you have to do if you configure HiLA to be used as an agent, e.g. inside a portal, and users have issued trust delegations, which allow the agent to act on their behalf. Another use case for the advanced configuration below is when you have multiple identities.

The configuration has changed a bit from version 2.3.0 to 2.4.0. Starting with 2.4.0, a new security library is in use, which provides new configuration options. Detailed information can be found e.g. in the UNICORE/X manual.

In order to stick with tradition, we kept the prefix of "hila.unicore6" with an optional profile part. Following this will be the new keywords from now on.

A standard configuration (without profiles) should look something like the following.

with the following meanings of the elements.

hila.unicore6.truststore has been replaced by the respective hierarchy of configuration options.

This configuration can be used for multiple users, either in case you have multiple identities and need to distinguish among them, or when you are an agent that acts on other user’s behalf. In the latter case, users will have issued trust delegations.

In comparison with versions prior to 2.4.0, we have gotten rid of the profile keyword here that had to precede the actual profile name, e.g. hila.unicore6.profile.demo2.password. This has been done for no other reason than shortening the configuration.

The multi-user configuration depends on the existance of an implementation of the Config interface. This can be configured via the hila.unicore6.config property. The default value for this property is

eu.unicore.hila.grid.unicore6.security.SimpleProfileConfig, which can use profiles as configured above. If you want to use any of the configured profiles, then the profile name needs to be added to the HiLA Location of the resources, e.g. unicore6:/demo1@sites or unicore6:/demo2@sites will use the keystores as configured above. Note that this does not have anything to do with trust delegation, each of the profile configurations will be used as primary identities to access Grid resources.

In order to use trust delegation, you need to use a different implementation of the Config interface. The unicore.properties file should contain the following line:

TDConfig requires SAML assertions in the directory ~/.hila2/saml-assertions. SAML assertions can be generated e.g. using the UNICORE Commandline Client (UCC) and its issue-delegation command. HiLA as an API does not offer this functionality at the moment. Trust delegation in HiLA is usually used in locations where the user credentials needed for creating the delegation are not available. Thus, HiLA as the receiver of such SAML assertions does not require this functionality.

The EMI Execution Services (EMI-ES) specification [emies] is an agreement among European middleware providers Arc, gLite, and UNICORE, to offer a common interface to their compute elements (CEs), which is the equivalent of a HiLA Site.

Starting with version 2.3.0, HiLA supports this specification and can send jobs to EMI-ES endpoints.

Analogous to the UNICORE 6 configuration file, which is expected to reside in ${user.home}/.hila2/unicore6.properties, the EMI ES configuration file is expected in the same directory as ${user.home}/.hila2/emi-es.properties.

Configuration properties have the following meanings.

All properties can be found in the securityLibrary documentation.

The EMI ES specification is only about compute services. It allows for submitting jobs and monitoring their status. Additionally, it allows for retrieving information about Sites.

The GridFTP implementation allows for accessing GridFTP services using ordinary gsiftp:/ URLs.

GridFTP does not currently use any additional configuration. It consumes direct gsiftp:/ URLs like any ordinary GridFTP client. Therefore, you will need a local Grid proxy certificate initialized with grid-proxy-init. Future versions of this implementation will be capable of using the user’s certificate directly, as is the case for the other implementations.