RUS Accounting for UNICORE

This user manual covers UNICORE accounting system developed in ICM. The system is using JMS messaging as its internal transport layer and the OGF Usage Record (UR) specification as a data representation format.

Data about jobs is collected from a grid component (XNJS, which is a part of Unicore/X server) and from a batch system server (BSS). These pieces of information are merged together to form a full job record.

Please note that this accounting system is designed for production installations. Therefore it is fairly useless if you don’t use a real BSS, but a Java or NOBATCH TSI.

The main modules of the system are:

Additionally the ActiveMQ JMS broker must be installed, however this component is a generic, 3rd party server and it is not a part of the UNICORE RUS Accounting System distribution.

Accounting architecture: dotted lines: JMS communication, solid lines: inter-component communication.

The RUS accounting system collects information about individual jobs and stores it in the database. At the same time the daily summaries are stored in the database, using the information from the completed records.

The jobs table, providing the accurate and detailed information can quickly grow to an insane size. Therefore the records from this table are automatically moved to a historical table, after a configurable period of time (e.g. 6 months). The historical table is not used by the system and its size does not influence the performance.

The aggregated reporting table holds only a subset of information of the summed amount of resources consumed by jobs performed on a particular site, by a particular user per day (to be precise: there are also other grouping variables). This table is subject to the same process of moving records to the historical table as we have in the case of individual jobs table. However as reporting table grows much slower then the jobs table, this operation can be done after a much longer period of time (e.g. 3 years).

The accounting system, since the 2.0 version uses the reporting data whenever possible, as its analyzes is significantly faster and at the same it provides majority of the data interesting for the stakeholders.

It is important to know that there are some (minor) limitations of the reporting data:

Version 1.2.0 introduced new architecture so you can’t use rus-service in version earlier than 1.2.0 with other components in version 1.2.0+.

If installing into an existing UNICORE service, as an add-on:

For UNICORE 6.4.x use rus-service 1.5.0

For UNICORE 6.5.0 use rus-service 1.6.0

For UNICORE 6.3 use rus-job-processor in version 1.2.0

For UNICORE 6.4 use rus-job-processor in version 1.3.0+.

UNICORE 6.5 has rus-job-processor installed (version 1.5.0+)

If you want to use rus-usage-logger-feeder you need to have logs generated by Unicore 6.4.0+. Due to bug in Unicore/X 6.4.0 you can’t collect information about user’s VOs. This problem was fixed in Unicore 6.4.1.

Logs from earlier versions of UNICORE will be also correctly parsed, but they don’t contain any useful information (we have information about a job, but we can’t connect that information with its corresponding part from rus-bss-adapter).

For UCC 6.5.0 use rus-ucc-plugin version 1.5.1

For UCC 6.4.2 use rus-ucc plugin version 1.5.0

We currently provide support for Torque and Sun Grid Engine. In order to choose BSS engine alter rus.service.engine property in ${bss-adapter}/conf/rus_bssadapter.conf

The accounting system is JMS based. It is good to be aware about the possible advanced settings like socket timeouts or low level logging. This can be achieved using the advanced settings of ActiveMQ clients which are used by the solution. The documentation is available available on ActiveMQ web pages. In particular the configuration of the popular TCP transport is here: http://activemq.apache.org/tcp-transport-reference.html

Currently the RUS accounting system is available only as a packed tar.gz archive. However it can be deployed on UNICORE installed both form the quickstart bundle (tar.gz or zip) and from the RPM package. The following table summarizes locations of important files, which are dependent on the UNICORE installation method.

rus-job-processor is installed on each Unicore/X server which accepts user jobs. Similarly rus-bssadapter should be installed on a machine executing the jobs, i.e. the BSS server (usually the same where TSI is installed).

Location of the rus-service is up to the administrator. It can be deployed into the same Unicore/X server where the rus-job-processor is installed or even to a UNICORE server with other services like Registry. However a good practice is to deploy it on a separate UNICORE container. Such deployment introduces an increased maintenance effort as an additional server must be run, but allows for a better stability, ensures that problems in the rus-service won’t influence a crucial Unicore/X server and splits the load.

Resource requirements of the rus-service are rather low. Tests have shown that on a commodity hardware, a database containing 1,000,000 of entries (jobs) behaves correctly and doesn’t overload the machine. Around 2,000,000 of entries the commodity server can start to have problems. Note, however, that rus-service provides mechanism to solve this problem automatically by rolling older records - see the configuration section.

The client side, rus-webui and rus-ucc-plugin can be left as the last components (both are fully optional). The UCC plugin can be installed with any UCC installation, assuming the version matches (see the Compatibility section).

Installation should be started from deploying the ActiveMQ broker. Then the rus-service should be deployed. Finally all other components can be installed. Such installation order ensures that each installed component can be started and at least minimally tested without waiting for an installation of other components.

The central point of data exchange is ActiveMQ broker. You can just download and extract it from ActiveMQ official web site. There is no need to perform any extra configuration. However you can add distinct users with privileges to write and read from queues.

Example:

Let’s suppose that we have the following installation:

We need one queue. Both grid_bss and grid_process will be producers, hence they need write access to queue. grid_service will be consumer.

And we configure queue access:

We use following credentials:

Of course it is also possible to set up SSL for the JMS messaging layer - see ActiveMQ documentation for details.

In this section the rus-service is described in more details. RUS-service collects and stores accounting data - it is the heart of the whole system. RUS-service may also export accounting data to external systems, it can be used in a hierarchy and also can produce summary or aggregated reports.

You should follow one of the instructions below to install a standalone rus-service instance or to add it to an existing UNICORE server.

The 2.0.x version of the RUS service was tested on the 6.6.0 release of the base UNICORE distribution. Any other 6.6.x release should be equally fine. When installing into the existing container or when referring to the documentation of the base UNICORE container it is suggested to use this version.

The easiest way to install the standalone version is to use the provided RPM package: unicore-rus-service. It is enough to download the package and install it normally using the rpm -i <rpmfilename> command.

You have to configure keystore/truststore (for instructions please refer to the UNICORE documentation of any UNICORE distribution as Unicore/X - the container’s options are the same) Then go to point 6. under <<rus-service - installing into existing server>, which describes how to configure the rus-service.

The following instruction is valid for installation of tar.gz bundle into an existing UNICORE container e.g. UNICORE Registry or UNICORE/X server.

The following options can be specified in the rus/rus.service configuration file:

The JMS options are as follows. The same options are used to configure the JMS producers in the rus-job-processor and in the rus-bssadapter, however with other prefixes (RUS.PROCESSOR.jms. and rus.bssadapter.jms. respectively). Also the same options are used to configure JMS producers in the case of export plugins - this time with a simple prefix jms. (see below for examples).

In order to provide a possibility to extend rus-service module, data export plugins mechanism is available. Plugins allows you to invoke arbitrary action after a record is inserted or updated. Plugin configuration is in the ${CONF}/rus/rus_extensions.xml file. Plugin configuration can be changed on-the-fly. Module will automatically detect this event and reload configuration.

Sample configuration is shown below:

Class attribute inside <extension> element defines Java class name. Attribute Id allows you to use few plugins which are implemented by the same class. Id distinguishes different entries.

Config section settings are passed to a plugin as Java Properties.

In order to generate notification by notification manager, condition defined in <condition> element must be evaluated to true. See below for description how to create conditions.

We provide two plugins: UrEventListener (shipped inside rus-service) and BATEventListener (inside rus-export-bat module).

UrEventListener allows for simple passing of UsageRecords to another broker. It can be used to sent records to rus-ur-site module. Alternative usage is ability to create hierarchical structure of a grid. Records can be then gathered at several, arbitrary defined levels. Please keep in mind, that <condition> element allows you to filter records.

The second plugin (BATEventListener) converts record from UsageRecord format to BAT format, which is used inside the PL-Grid project.

Parameters responsible for broker communication are the same in case of the two plugins. BATEventListener allows to use one additional configuration property: bat.site.name, which overrides the site for all records exported with this plugin.

Element <condition> have to contain a valid Spring Expression Language (SpEL) expression. The simple rules are quite intuitive for a detailed discussion of the SpEL syntax check the Spring documentation: http://static.springsource.org/spring/docs/3.0.x/reference/expressions.html#expressions-language-ref.

Sample SpEL expressions:

Matches all records that have exitStatus and globalUserId fields set. Such expression will match only complete records (with data from BSS and Unicore/X) of finished jobs.

Matches all records as above, but the security related attributes of the job must possess the urn:SAML:voprofile:group attribute with the value starting with /someVO. Note that the check is constructed in a way that correctly handles all situations when the attribute is not set at all.

Available elements in SpEL context:

Programming custom plugins

If you want to create your own extension, you have to implement class which inherits from AbstractDataEventListener, which defines the following methods:

and contains a default constructor.

This feature was mainly developed to allow for generating Aggregated Usage Records, but can be extended to create other report types.

Configuration file is located in: ${CONF}/rus/reporting.xml.

The configuration will be explained using a representative example:

We have one reportItem in this sample configuration file. Each of reportItem must have a unique id. Report items are used to distinguish between different report configurations.

The process of reporting consists of two steps: report preparation and report postprocessing. In the first step internal rus-service report data is formatted to the desired form. In the latter step the previously generated report for instance can be sent over network or saved to a file.

The resultTransformerClass entry defines what implementation will handle a translation from an internal format to the final report format. In the example the pl.edu.icm.unicore.accounting.service.reporting.AURReportingTransformer class performs a translation to the Aggregated Usage Record (AUR).

The postProcessorClass defines a class which performs the record postprocessing. The pl.edu.icm.unicore.accounting.service.reporting.JMSSendPostProcessor class sends generated reports via JMS.

The list of available implementations is:

The config element contains arbitrary properties which are used to configure the chosen result transformer and postprocessor implementations. In the example the JMS configuration is set: it is possible to specify jms url, queue, username or password, and keystore/truststore information.

There are several advanced properties which can be set for all implementations. There is one of them which can be especially useful: use.current.period. If this property is defined and its value is true, then reporting data is collected from the current time period (day, week, month or year). Note that in this case subsequent invocations of the reporting in the same period (e.g. in the same week) can produce different results, as new records can be added to the database in the meantime. When this option is not set, then reporting uses data from the last completed period of time. Example: Assuming that the report is generated on 25.02.2010, timePeriod is set to monthly, by default the report will contain all jobs accounted between 01.01.2010 and 31.01.2010. If additionally use.current.period is set to true, then the report will include the jobs accounted between 01.02.2010 and 25.02.2010.

The cronExpression element allows for setting up a schedule of the report generation. For information about the cron expression syntax refer to the page: http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger.

The timePeriod element allows for selecting time range over which the report is generated. Available values: daily, weekly, monthly, yearly.

The optional whereAttributes element allows for filtering which records are used for generating report. Each subelement of the whereAttributes element must be one of the attributes enumerated in possible reporting xml values table. If you specify <any /> element as child of whereAttribute, than all values will be included in report. However, if you enumerate values, than only those enumerated values are used for generating the report.

Additionally each of the elements in whereAttributes can have the groupBy="true" XML attribute. If so, then the behaviour of the report generator is changed: a separate report is created for each and every distinct value of such an attribute (or for each distinct combination of values if more then one attribute has the groupBy set).

Example: We have grid system which is used by users: CN=user1,OU=Grid, CN=user2,OU=Grid, and some others. The following configuration generates two reports: one for user CN=user1,OU=Grid, and second for CN=user2,OU=Grid while for other users no report is generated.

But when we set groupBy to false (or simply skip it), then only one report will be generated, only based on jobs, which are submitted either by CN=user1,OU=Grid or CN=user2,OU=Grid.

Every child of whereAttributes element can have the groupBy attribute.

Possible where attributes. You can use <any /> or enumerate values:

Additionally each report transformer as its input receives the actual reported values, which are summed up for each entry (controlled by the groupBy attribute). The aggregated data consist of: number of jobs, walltime, CPU time, virtual memory and physical memory.

With this plugin (which supersedes the older test version called rus-export-apel) one can export records from the RUS service using EMI Messaging Protocol for Accounting (defined: https://wiki.egi.eu/wiki/APEL/MessagingProtocol). Records are encoded in EMI Compute Accounting Record (CAR) format. It is possible to use this system either send individual job records or summary (aggregated) records.

In order to install rus-export-car, copy required jars from distribution into rus-service lib folder.

To configure the export of aggregated records, create new entry in reporting.xml:

You have to set:

You may want to additionally configure a consumer certificate to encrypt records and to enable SSL on the broker connection (truststore is needed then). See below for the complete reference of configuration options.

Finally you can provide arbitrary filtering of records to be sent, but ensure to have groupBy turned on for globalUserId, site and vo - this is required by CAR specification. Also timePeriod must be set to monthly according to standard.

For general information about reporting refer to Configuring rus-service reporting.

To configure the export of individual job records, create new entry in rus_extensions.xml:

The configuration of the broker, truststore, keystore and factors is the same as for the reporting configuration.

The table below contains a reference documentation of all allowed options.

In order to install the UCC plugin:

This UCC extension adds new commands in the (new) Accounting category. Invocation of UCC with help option will provide a full list.

Resource Usage Service client allows you to query records, which meet specified criteria. The service path is services/RUS, what is important when using the precise address, without the + -s SITE-NAME+ parameter.

Examples:

UCC client can be used to schedule plugin notifications or report from given time period. The service path is services/PluginsExecutor.

Examples:

RUS-Service gets data from two sources per each job (or from one if one of the components is not configured): rus-job-processor with Grid data and bss-adapter with BSS data. Typically for each job, each of the data collectors produces multiple records, when job’s state is changed. Rus-Service merges all records corresponding to a single job into a single record, which should contain the most recent job’s status. Merging is done in different ways depending on record’s element.

Records are matched together first by the globalJob id and then by localJobId and MachineName.

The following table shows all supported record elements along with components which can produce them and merging algorith being applied. Note that in many cases particular piece of data is provided only in some records comming from a particular source. For instance consumed memory is provided by bss-adapter but only in the record signaling job’s end. More details (e.g. syntax of fields) can be found in https://www.ogf.org/documents/GFD.98.pdf

The merging algorithms used in the table:

Since UNICORE 6.5.0, rus-job-processor is distributed together with Unicore/X server, therefore only its configuration is necessary. If you use older UNICORE distribution, then you should also use older release of this accounting software and check its documentation about installation instructions.

Installation and configuration is relatively simple:

The detailed configuration options reference follows:

The detailed JMS properties reference is available in the rus-service configuration section.

Ur-site requires own database. We copy data from rus-site via JMS using pl.edu.icm.unicore.accounting.service.jms.UrEventListener plugin. We can specify condition, to filter outgoing traffic.

Look at architecture overview for better understaning data flow:

rus-service → UrEventListener → (JMS) → broker → (JMS) → ur-site.

Following informations describes installation on Apache Tomcat. For other containers some of paths may be different.

Additionally we assume that we deploy application under /ur-site context.

Logging of the ur-site web application is controlled by a file WEB-INF/classes/log4j.properties. By default it logs to a separate file: CATALINA_HOME/logs/ur-site.log. You can reconfigure logging settings there.

When infrastructure on which we installing accounting system works for some period of time, then the question arrives: Is there any possibility to import archival accounting data to rus-accounting service? The answer is true, but we need archival log files from Unicore/X in version 6.4.1+. Those logs contains UsageLogger entries. Based on those entries we can generate accounting data, which are identical to those which are generated by rus-job-processor.

Rus-usage-logger-feeder is a standalone application. It is configured with ActiveMQ broker parameters. As input parameters we supply time range of imported data and path to the Unicore/X server logs. Application search throught input files in order to find UsageLogger entries. Every item found is parsed and UsageRecord is generated, and sent to the broker. After the import operation is completed, the application displays information about amount of records found per each day.

Sample execution:

This sections describes problems caused by broker interaction.

Log:

Solution:

Enable debug mode in logging.properties for more info about cause. Than check your connection with broker.

For ActiveMQ problems please refer to common problems section.

When scheduled notification fails for 25000 times, then following message is displayed:

and notification is removed.

When plugins configuration is changed but there are remaining notifications for non-existent plugin, than following message is shown:

When notification fails for 3 times in single notification executor interation, there is high probablity that subsequent calls will also fail. So NotificationExecutor doesn’t process notification for this plugin until the next iteration. Following message is shown in log:

There was some records to sent in JMS queue, but unicore was unclean closed or freezed:

Job processor uses in memory storage for created records. Capacity of records queue is 5000. If you stop unicore/X or unicore/X hangs while queue isn’t empty then those records are lost. You can use rus-ussage-logger-feeder to parse logs and generate lost records. If connection with broker works fine, than records are sent immediately.

When queue size exceed 4000 records following warning is shown:

When queue is full than following warning is shown:

Queue has limited capacity to avoid out of memory exceptions.

Refer to common problems. When broker connection is down, current record pointer doesn’t advance. This approach gives you guarantee that any of record isn’t lost. After connection is again established historical records are sent to broker.

When record line in BSS log isn’t correctly parsed, then following error is shown:

This line is simply ignored, since parsing proceduce is deterministic, and it could cause poison pill for bss-adapter.

Refer to common problems.

Log:

Solution:

Verify your installation: Refer to point 10 in rus-service installation procedure.

Refer to common problems.

-Renamed to rus-webui from ur-site