A maintenance task is a process which runs once or in regular time intervals. It is defined by a core plugin of type
maintenance-tasks. Usually a maintenance task can only run on AS or DSS but not in both environments.
The following properties are common for all maintenance tasks:
|class||The fully-qualified Java class name of the maintenance task. The class has to implement |
|execute-only-once||A flag which has to be set to |
A time interval (in seconds) which defines the pace of execution of the maintenance task. Can be specified with one of the following time units:
|start||A time at which the task should be executed the first time. Format: |
Generic Maintenance Tasks
Description: Triggers archiving of unarchived data sets.
Data set type of the data sets to be archived. You can only specify one data set type; second plugin should be created for different data-set-type(~/servers/core-plugins/multi-dataset-archiver/2/dss/maintenance-tasks/auto-archiver2) . If undefined all data set of all types might be archived
Minimum number of days a data set to be archived hasn't been accessed. Default: 30
Discoverer of candidates to be archived:
|policy.class||A policy specifies which data set candidates should be archived. If undefined all candidates will be archived. Has to be a fully-qualified name of a Java class implementing |
Properties specific for the policy specified by
BlastDatabaseCreationMaintenanceTask (since 13.04.11)
Description: Creates BLAST databases from FASTA and FASTQ files of data sets and/or properties of experiments, samples, and data sets.
The title of all entries of the FASTA and FASTQ files will be extended by the string
[Data set: <data set code>, File: <path>]. Sequences provide by an entity property will have identifiers of the form
<entity kind>+<perm id>+<property type>+<time stamp>. This allows to determine where the matching sequences are stored in openBIS. A sequence can be a nucleic acid sequence or an amino acid sequence.
For each data set a BLAST nucl and prot databases will be created (if not empty) by the tool
makeblastdb. For all entities of a specified kind and type one BLAST database (one for nucleic sequences and one for amino acid sequences) will be created from the plain sequences stored in the specified property (white spaces will be removed). In addition an index is created by the tool
makembindex if the sequence file of the database (file type
.nsq) is larger than 1MB. The name of the databases are
<data set code>-nucl/prot and
<entity kind>+<entity type code>+<property type code>+<time stamp>-nucl/prot. These databases are referred in the virtual database
If a data set is deleted the corresponding BLAST nucl and prot databases will be automatically removed the next time this maintenance task runs. If an entity of specified type has been modified the BLAST databases will be recalculated the next time this maintenance task runs.
Works only if BLAST+ tool suite has been installed. BLAST+ can be downloaded from ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/
Comma-separated list of regular expressions of data set types. All FASTA and FASTQ files from those data sets are handled. All data sets of types not matching at least one of the regular expression are not handled.
Comma-separated list of descriptions of entity properties with sequences. A description is of the form
|file-types||Space separated list of file types. Data set files of those file types have to be FASTA or FASTQ files. Default: |
|blast-tools-directory||Path in the file system where all BLAST tools are located. If it is not specified or empty the tools directory has to be in the PATH environment variable.|
|blast-databases-folder||Path to the folder where all BLAST databases are stored. Default: |
|blast-temp-folder||Path to the folder where temporary FASTA files are stored. Default: |
|last-seen-data-set-file||Path to the file which stores the id of the last seen data set. Default: |
Description: Checks that the file information in pathinfo database is consistent with the information the file system provides. This is done for all recently registered data sets.
|checking-time-interval||Time interval in the past which defines the range of data sets to be checked. That is, all data sets with registration date between now minus |
Description: Sends a data set summary report to a list of e-mail recipients in regular time intervals. The report contains all new data sets registered since the last report. Selected properties can be included into the report. The data sets are grouped by the data set type.
In order to be able to send an e-mail the following properties in
service.properties have to be defined:
mail.from = openbis@<host> mail.smtp.host = <SMTP host> mail.smtp.user = <can be empty> mail.smtp.password = <can be empty>
Interval (in seconds) between regular checks whether to create a report or not. This value should be set to 86400 (1 day). Otherwise the same report might be sent twice or no report will be sent.
Time the report will be created. A good values for this parameter is some early time in the morning like in the example below.
Comma-separated list of numbers denoting days of week (Sunday=1, Monday=2, etc.). This parameter should be used if reports should be sent weekly or more often.
Comma-separated list of numbers denoting days of month. Default value of this parameter is 1.
Comma-separated list of e-mail addresses.
Optional comma-separated list of data set properties to be included into the report.
|Restrict the report to the specified comma-separated data set types.|
|Use the specified content as the body of the email.|
A report is sent at each day which is either a specified day of week or day of month. If only weekly reports are needed the parameter
days-of-month should be set to an empty string.
This means that on the 1st day of every month at 1:00 AM openBIS sends to the specified e-mail recipients a report about the data sets of types RAW_DATA and MZXML_DATA that have been uploaded in the previous month.
Description: Deletes data sets which have been deleted on AS.
If this task isn't configured neither in service.properties nor as a core plugin it will be established automatically by using default configuration and running every 5 minutes.
|last-seen-data-set-file||Path to a file which will store the code of the last data set handled. Default: |
|Maximum number of retries in case of currently not available filesystem of the share containing the data set. Default:11|
|timing-parameters.failure-interval||Waiting time (in seconds) between retries. Default: 10|
Description: Deletes archived data sets which have been deleted on AS. This tasks needs the archive plugin to be configured in
|status-filename||Path to a file which will store the technical ID of the last data set deletion event on AS.|
Description: Deletes database entries which are related to data sets deleted in AS. The database is can be any relational database accessible by DSS.
|data-source||Key of a data source configured in |
|synchronization-table||Name of the table which stores the technical ID of the last data set deletion event on AS. This is ID is used to ask AS for all new data set deletion events. Default value: |
|last-seen-event-id-column||Name of the column in the database table defined by property |
|data-set-table-name||Comma-separated list of table names which contain stuff related to data sets to be deleted. In case of cascading deletion only the tables at the beginning of the cascade should be mentioned. Default value: |
|data-set-perm-id||Name of the column in all tables defined by |
Description: Re-evaluates dynamic properties of all entities
DynamicPropertyEvaluationTriggeredByMaterialChangeMaintenanceTask (since 13.04.5)
Description: Re-evaluates dynamic properties of all samples which refer via properties of type MATERIAL directly or indirectly to materials changed since the last re-evaluation.
|timestamp-file||Path to a file which will store the timestamp of the last evaluation. Default value: |
|initial-timestamp||Initial timestamp of the form |
Description: Archives all data sets of experiments which fulfill some criteria. This tasks needs the archive plugin to be configured in
|excluded-data-set-types||Comma-separated list of data set types. Data sets of such types are not archived. Default: No data set type is excluded.|
|estimated-data-set-size-in-KB.<data set type>||Specifies for the data set type <data set type> the average size in KB. If <data set type> is |
|free-space-provider.class||Fully qualified class name of the free space provider (implementing |
|monitored-dir||Path to the directory to be monitored by the free space provider.|
|minimum-free-space-in-MB||Minimum free space in MB. If the free space is below this limit the task archives data sets. Default: 1 GB|
If there is not enough free space the task archives all data sets experiment by experiment until free space is above the specified limit. The oldest experiments are archived first. The age of an experiment is determined by the youngest modification/registration time stamp of all its data sets which are not excluded by data set type or archiving status.
The free space is only calculated once when the task starts to figure out whether archiving is necessary or not. This value is than used together with estimated data set sizes to get an estimated free space which is used for the stopping criteria. Why not calculating the free space again with the free space provider after the data sets of an experiment have been archived? The reason is that providing the free space might be an expensive operation. This is the case when archiving means removing data from a database which have been fed by data from data sets of certain type. In this case archiving (i.e. deleting) those data in the database do not automatically frees disk space because freeing disk space is for databases often an expensive operation.
The DSS admin will be informed by an e-mail about which experiments have been archived.
Description: Queries openBIS database to find data sets without a size filled in, then queries the pathinfo DB to see if the size info is available there; if it is available, it fills in the size from the pathinfo information. If it is not available, it does nothing. Data sets from openBIS database are fetched in chunks (see data-set-chunk-size property). After each chunk the maintenance tasks checks whether a time limit has been reached (see time-limit property). If so, it stops processing. A code of the last processed data set is stored in a file (see last-seen-data-set-file property). The next run of the maintenance task will process data sets with a code greater than the one saved in the "last-seen-data-set-file". This file is deleted periodically (see delete-last-seen-data-set-file-interval) to handle a situation where codes of new data sets are lexicographically smaller than the codes of the old datasets. Deleting the file is also needed when pathinfo database entries are added after a data set has been already processed by the maintenance task.
|last-seen-data-set-file||Path to a file that will store a code of the last handled data set. Default value: "fillUnknownDataSetSizeTaskLastSeen"|
|A time interval (in seconds) which defines how often the "last-seen-data-set-file" file should be deleted. The parameter can be specified with one of the following time units: |
Number of data sets requested from AS in one chunk. Default: 100
|time-limit||Limit of execution time of this task. The task is stopped before reading next chunk if the time has been used up. This parameter can be specified with one of the following time units: |
Description: Creates/updates a mirrot of the data store. Data set are organized hierachical in accordance to their experiment and samples
|storeroot-dir-link-path||Path to the root directory of the store as to be used for creating symbolic links. This should be used if the path to the store as seen by clients is different than seen by DSS. (Since 13.04.15)|
|storeroot-dir||Path to the root directory of the store. Used if |
|hierarchy-root-dir||Path to the root directory of mirrored store.|
|link-naming-strategy.class||Fully qualified class name of the strategy to generate the hierarchy (implementing |
|link-source-subpath.<data set type>||Link source subpath for the specified data set type. Only files and folder in this relative path inside a data set will be mirrored. Default: The complete data set folder will be mirroed.|
|link-from-first-child.<data set type>||Flag which specifies whether only the first child of or the complete folder (either the data set or the one specified by link-source-subpath.<data set type>). Default: False|
Description: Feeds a report database with recently added or modified materials.
|database-driver||Fully qualified name of the JDBC driver class.|
|database-url||URL to access the database server.|
User name of the database. Default: User who started openBIS AS.
|database-password||Optional password of the database user.|
Path to the file containing configuration information of mapping material types and material properties to tables and columns in the report database
The SQL select statement which returns one column of type time stamp for the time stamp of the last report. If the result set is empty the time stamp is assumed to be 1970-01-01. If the result set has more than one row the first row is used.
The SQL statement which updates or adds a time stamp. The statement has to contain a '?' symbol as the placeholder of the actual time stamp.
The SQL statement to add a time stamp the first time. The statement has to contain a '?' symbol as the placeholder of the actual time stamp. Default: same as
The mapping file is a text file describing the mapping of the data (i.e. material codes and material properties) onto the report database. It makes several assumptions on the database schema:
- One table per material type. There are only table of materials to be reported.
- Each table has a column which contains the material code.
- The entries are unique.
- The material code is a string not longer than 60 characters.
- Each table has one column for each property type. Again, there are only column for properties to be reported.
- The data type of the column should match the data type of the properties:
- MATERIAL: only the material code (string) will be reported. Maximum length: 60
- CONTROLLEDVOCABULARY: the label (if defined) or the code will be reported. Maximum length: 128
- TIMESTAMP: timestamp
- INTEGER: integer of any number of bits (maximum 64).
- REAL: fixed or floating point number
- all other data types are mapped to text.
The general format of the mapping file is as follows:
[<Material Type Code>: <table Name>, <code column name>]
<Property Type Code>: <column name>
<Property Type Code>: <column name>
[<Material Type Code>: <table Name>, <code column name>]
<Property Type Code>: <column name>
<Property Type Code>: <column name>
- Empty lines and lines starting with '#' will be ignored.
- Table and column names can be upper or lower case or mixed.
- Material type codes and property type codes have to be in upper case.
If you put a foreign key constraint on the material code of one of the material properties, you need to define the constraint checking as DEFERRED in order to not get a constraint violation. The reason is that this task will not order the
INSERT statements by its dependencies, but in alphabetical order.
MultiDataSetUnarchivingMaintenanceTask (since 13.04.12)
Description: Triggers unarchiving of multi data set archives. Is only needed if the configuration property
delay-unarchiving of the Multi Data Set Archiving (since 13.04.11) is set
Configuration: No specific properties.
Description: Feeds the pathinfo database with file paths of all data sets in the store. It can be used as a maintenance task as well as a post registration task. As a maintenance task it is needed to run only once if a PostRegistrationMaintenanceTask is configured. This task assumes a data source with for 'path-info-db'.
If used as a maintenance task the data sets are processed in the order they are registered. The registration time stamp of the last processed data set is the starting point when the task is executed next time.
Number of data sets requested from AS in one chunk if it is used as a maintenance task. Default: 1000
|max-number-of-chunks||Maximum number of chunks of size |
|time-limit||Limit of execution time of this task if it is used as a maintenance task. The task is stopped before reading next chunk if the time has been used up. If it is specified it is an alternative way to limit the number of data sets to be processed instead of specifying |
Description: Calculates the CRC32 checksum of all files in the pathinfo database with unknown checksum. This task is needed to run only once. It assumes a data source for key 'path-info-db'.
Configuration: No additional configuration.
Description: A tasks which runs a sequence of so-called post-registration tasks for each freshly registered data set.
|ignore-data-sets-before-date||Defines a registration date. All data sets registered before this date are ignored. Format: |
|last-seen-data-set-file||Path to a file which stores the code of the last data set successfully post-registered. Default value: |
|cleanup-tasks-folder||Path to a folder which stores serialized clean-up tasks always created before a post-registration task is executed. These clean-up tasks are executed on start up of DSS after a server crash. Default value: |
|post-registration-tasks||Comma-separated list of keys of post-registration task configuration. Each key defines (together with a '.') the prefix of all property keys defining the post-registration task. They are executed in the order their key appear in the list.|
Description: Removes unofficial unused vocabulary terms. For more details about unofficial vocabulary terms see Ad Hoc Vocabulary Terms.
|Unofficial terms are only deleted if they have been registered more than the specified number of days ago. Default: 7 days.|
Screening Maintenance Tasks
Configuration: See DeleteFromExternalDBMaintenanceTask