Child pages
  • Maintenance Tasks
Skip to end of metadata
Go to start of metadata


A maintenance task is a process which runs once or in regular time intervals. It is defined by a core plugin of type maintenance-tasks. Usually a maintenance task can only run on AS or DSS but not in both environments.

The following properties are common for all maintenance tasks:

Property KeyDescription
classThe fully-qualified Java class name of the maintenance task. The class has to implement IMaintenanceTask.
execute-only-onceA flag which has to be set to true if the task should be executed only once. Default value: false

A time interval (in seconds) which defines the pace of execution of the maintenance task. Can be specified with one of the following time units: ms, msec, s, sec, m, min, h, hours, d, days. Default time unit is sec. Default value: one day.

startA time at which the task should be executed the first time. Format: HH:mm. where HH is a two-digit hour (in 24h notation) and mm is a two-digit minute. By default the task is execute at server startup.

Generic Maintenance Tasks


Environment: DSS

Description: Triggers archiving of unarchived data sets.


Property KeyDescription
remove-datasets-from-storeIf true the archived data set will be removed from the store. Default: false

Data set type of the data sets to be archived. You can only specify one data set type; second plugin should be created for different data-set-type(~/servers/core-plugins/multi-dataset-archiver/2/dss/maintenance-tasks/auto-archiver2) . If undefined all data set of all types might be archived


Minimum number of days a data set to be archived hasn't been accessed. Default: 30


Discoverer of candidates to be archived:

  • ch.systemsx.cisd.etlserver.plugins.AgeArchiveCandidateDiscoverer: All data sets with an access time stamp older than specified by property older-than are candidates. This is the default discoverer.
  • ch.systemsx.cisd.etlserver.plugins.TagArchiveCandidateDiscoverer: All data sets which are marked by one of the tags specified by the property archive-candidate-discoverer.tags are candidates.
policy.classA policy specifies which data set candidates should be archived. If undefined all candidates will be archived. Has to be a fully-qualified name of a Java class implementing ch.systemsx.cisd.etlserver.IAutoArchiverPolicy.

Properties specific for the policy specified by policy.class. More about policies can be found here.

class = ch.systemsx.cisd.etlserver.plugins.AutoArchiverTask
interval = 10 days
archive-candidate-discoverer.class = ch.systemsx.cisd.etlserver.plugins.TagArchiveCandidateDiscoverer
archive-candidate-discoverer.tags = /admin-user/archive
policy.class = ch.systemsx.cisd.etlserver.plugins.GroupingPolicy
policy.minimal-archive-size = 1500000
policy.maximal-archive-size = 3000000
policy.grouping-keys = Space#DataSetType, Space#Experiment:merge

BlastDatabaseCreationMaintenanceTask (since 13.04.11)

Environment: DSS

Description: Creates BLAST databases from FASTA and FASTQ files of data sets and/or properties of experiments, samples, and data sets.

The title of all entries of the FASTA and FASTQ files will be extended by the string [Data set: <data set code>, File: <path>]. Sequences provide by an entity property will have identifiers of the form <entity kind>+<perm id>+<property type>+<time stamp>. This allows to determine where the matching sequences are stored in openBIS. A sequence can be a nucleic acid sequence or an amino acid sequence.

For each data set a BLAST nucl and prot databases will be created (if not empty) by the tool makeblastdb. For all entities of a specified kind and type one BLAST database (one for nucleic sequences and one for amino acid sequences) will be created from the plain sequences stored in the specified property (white spaces will be removed). In addition an index is created by the tool makembindex if the sequence file of the database (file type .nsq) is larger than 1MB. The name of the databases are <data set code>-nucl/prot and <entity kind>+<entity type code>+<property type code>+<time stamp>-nucl/prot. These databases are referred in the virtual database all-nucl (file: all-nucl.nal) and all-prot (file: all-prot.pal).

If a data set is deleted the corresponding BLAST nucl and prot databases will be automatically removed the next time this maintenance task runs. If an entity of specified type has been modified the BLAST databases will be recalculated the next time this maintenance task runs.

Works only if BLAST+ tool suite has been installed. BLAST+ can be downloaded from



Property KeyDescription

Comma-separated list of regular expressions of data set types. All FASTA and FASTQ files from those data sets are handled. All data sets of types not matching at least one of the regular expression are not handled.


Comma-separated list of descriptions of entity properties with sequences. A description is of the form

<entity kind>+<entity type code>+<property type code>

where <entity kind> is either EXPERIMENTSAMPLE or DATA_SET (Materials are not supported).

file-typesSpace separated list of file types. Data set files of those file types have to be FASTA or FASTQ files. Default: .fasta .fa .fsa .fastq
blast-tools-directoryPath in the file system where all BLAST tools are located. If it is not specified or empty the tools directory has to be in the PATH environment variable.
blast-databases-folderPath to the folder where all BLAST databases are stored. Default: <data store root>/blast-databases
blast-temp-folderPath to the folder where temporary FASTA files are stored. Default: <blast-databases-folder>/tmp
last-seen-data-set-filePath to the file which stores the id of the last seen data set. Default: <data store root>/last-seen-data-set-for-BLAST-database-creation

class = ch.systemsx.cisd.etlserver.plugins.BlastDatabaseCreationMaintenanceTask
interval = 1 h
dataset-types = BLAST-.+
blast-tools-directory = /usr/local/ncbi/blast/bin


Environment: DSS

Description: Checks that the file information in pathinfo database is consistent with the information the file system provides. This is done for all recently registered data sets.


Property KeyDescription
checking-time-intervalTime interval in the past which defines the range of data sets to be checked. That is, all data sets with registration date between now minus checking-time-interval and now will be checked. Can be specified with one of the following time units: ms, msec, s, sec, m, min, h, hours, d, days. Default time unit is sec. Default value: one day.

class = ch.systemsx.cisd.etlserver.path.DataSetAndPathInfoDBConsistencyCheckTask
interval = 7 days
checking-time-interval = 8 days


Environment: AS

Description: Sends a data set summary report to a list of e-mail recipients in regular time intervals. The report contains all new data sets registered since the last report. Selected properties can be included into the report. The data sets are grouped by the data set type.

In order to be able to send an e-mail the following properties in have to be defined:

mail.from = openbis@<host> = <SMTP host>
mail.smtp.user = <can be empty>
mail.smtp.password = <can be empty>


Property Key



Interval (in seconds) between regular checks whether to create a report or not. This value should be set to 86400 (1 day). Otherwise the same report might be sent twice or no report will be sent.


Time the report will be created. A good values for this parameter is some early time in the morning like in the example below.


Comma-separated list of numbers denoting days of week (Sunday=1, Monday=2, etc.). This parameter should be used if reports should be sent weekly or more often.


Comma-separated list of numbers denoting days of month. Default value of this parameter is 1.


Comma-separated list of e-mail addresses.


Optional comma-separated list of data set properties to be included into the report.


Restrict the report to the specified comma-separated data set types.


Use the specified content as the body of the email.

A report is sent at each day which is either a specified day of week or day of month. If only weekly reports are needed the parameter days-of-month should be set to an empty string.

Example: of AS
<task id>.class = ch.systemsx.cisd.openbis.generic.server.task.DataSetRegistrationSummaryTask
<task id>.interval = 86400
<task id>.start = 1:00
<task id>.data-set-types = RAW_DATA, MZXML_DATA
<task id>.email-addresses =, 

This means that on the 1st day of every month at 1:00 AM openBIS sends to the specified e-mail recipients a report about the data sets of types RAW_DATA and MZXML_DATA that have been uploaded in the previous month.


Environment: DSS

Description: Deletes data sets which have been deleted on AS.

If this task isn't configured neither in nor as a core plugin it will be established automatically by using default configuration and running every 5 minutes.


Property KeyDescription
last-seen-data-set-filePath to a file which will store the code of the last data set handled. Default: deleteDatasetsAlreadyDeletedFromApplicationServerTaskLastSeen


Maximum number of retries in case of currently not available filesystem of the share containing the data set. Default:11
timing-parameters.failure-intervalWaiting time (in seconds) between retries. Default: 10

class = ch.systemsx.cisd.etlserver.plugins.DeleteDataSetsAlreadyDeletedInApplicationServerMaintenanceTask
interval = 60
last-seen-data-set-file = lastSeenDataSetForDeletion.txt


Environment: DSS

Description: Deletes archived data sets which have been deleted on AS. This tasks needs the archive plugin to be configured in


Property KeyDescription
status-filenamePath to a file which will store the technical ID of the last data set deletion event on AS.

class = ch.systemsx.cisd.etlserver.plugins.DeleteFromArchiveMaintenanceTask
interval = 3600
status-filename = ../archive-cleanup-status.txt


Environment: DSS

Description: Deletes database entries which are related to data sets deleted in AS. The database is can be any relational database accessible by DSS.


Property KeyDescription
data-sourceKey of a data source configured in or in a core plugin of type 'data-sources'. A data source defines the credentials to access the database.
synchronization-tableName of the table which stores the technical ID of the last data set deletion event on AS. This is ID is used to ask AS for all new data set deletion events. Default value: EVENTS
last-seen-event-id-columnName of the column in the database table defined by property synchronization-table which stores the ID of the last data set deletion event. Default value: LAST_SEEN_DELETION_EVENT_ID
data-set-table-nameComma-separated list of table names which contain stuff related to data sets to be deleted. In case of cascading deletion only the tables at the beginning of the cascade should be mentioned. Default value: image_data_sets, analysis_data_sets.
data-set-perm-idName of the column in all tables defined by data-set-table-name which stores the data set code. Default value: PERM_ID

class = ch.systemsx.cisd.etlserver.plugins.DeleteFromExternalDBMaintenanceTask
interval = 300
data-source = proteomics-db
data-set-table-name = data_sets 


Environment: AS

Description: Re-evaluates dynamic properties of all entities


Property KeyDescription

class = ch.systemsx.cisd.openbis.generic.server.task.DynamicPropertyEvaluationMaintenanceTask
interval = 3600

DynamicPropertyEvaluationTriggeredByMaterialChangeMaintenanceTask (since 13.04.5)

Environment: AS

Description: Re-evaluates dynamic properties of all samples which refer via properties of type MATERIAL directly or indirectly to materials changed since the last re-evaluation.


Property KeyDescription
timestamp-filePath to a file which will store the timestamp of the last evaluation. Default value: ../../../data/DynamicPropertyEvaluationTriggeredByMaterialChangeMaintenanceTask-timestamp.txt.
initial-timestampInitial timestamp of the form YYYY-MM-DD (e.g. 2013-09-15) which will be used the first time when the timestamp file doesn't exist or has an invalid value. This is a mandatory property.

class = ch.systemsx.cisd.openbis.generic.server.task.DynamicPropertyEvaluationTriggeredByMaterialChangeMaintenanceTask
interval = 7 days
initial-timestamp = 2012-12-31


Environment: DSS

Description: Archives all data sets of experiments which fulfill some criteria. This tasks needs the archive plugin to be configured in


Property KeyDescription
excluded-data-set-typesComma-separated list of data set types. Data sets of such types are not archived. Default: No data set type is excluded.
estimated-data-set-size-in-KB.<data set type>Specifies for the data set type <data set type> the average size in KB. If <data set type> is DEFAULT it will be used for all data set types with unspecified estimated size.
free-space-provider.classFully qualified class name of the free space provider (implementing ch.systemsx.cisd.common.filesystem.IFreeSpaceProvider). Depending on the free space provider additional properties, all starting with prefix free-space-provider.,  might be needed. Default: ch.systemsx.cisd.common.filesystem.SimpleFreeSpaceProvider
monitored-dirPath to the directory to be monitored by the free space provider.
minimum-free-space-in-MBMinimum free space in MB. If the free space is below this limit the task archives data sets. Default: 1 GB

class = ch.systemsx.cisd.etlserver.plugins.ExperimentBasedArchivingTask
interval = 86400
minimum-free-space-in-MB = 2048
monitored-dir = /my-data/
estimated-data-set-size-in-KB.RAW_DATA = 12000
estimated-data-set-size-in-KB.DEFAULT = 35000

If there is not enough free space the task archives all data sets experiment by experiment until free space is above the specified limit. The oldest experiments are archived first. The age of an experiment is determined by the youngest modification/registration time stamp of all its data sets which are not excluded by data set type or archiving status.

The free space is only calculated once when the task starts to figure out whether archiving is necessary or not. This value is than used together with estimated data set sizes to get an estimated free space which is used for the stopping criteria. Why not calculating the free space again with the free space provider after the data sets of an experiment have been archived? The reason is that providing the free space might be an expensive operation. This is the case when archiving means removing data from a database which have been fed by data from data sets of certain type. In this case archiving (i.e. deleting) those data in the database do not automatically frees disk space because freeing disk space is for databases often an expensive operation.

The DSS admin will be informed by an e-mail about which experiments have been archived.


Environment: DSS

Description: Queries openBIS database to find data sets without a size filled in, then queries the pathinfo DB to see if the size info is available there; if it is available, it fills in the size from the pathinfo information. If it is not available, it does nothing. Data sets from openBIS database are fetched in chunks (see data-set-chunk-size property). After each chunk the maintenance tasks checks whether a time limit has been reached (see time-limit property). If so, it stops processing. A code of the last processed data set is stored in a file (see last-seen-data-set-file property). The next run of the maintenance task will process data sets with a code greater than the one saved in the "last-seen-data-set-file". This file is deleted periodically (see delete-last-seen-data-set-file-interval) to handle a situation where codes of new data sets are lexicographically smaller than the codes of the old datasets. Deleting the file is also needed when pathinfo database entries are added after a data set has been already processed by the maintenance task. 


Property KeyDescription
last-seen-data-set-filePath to a file that will store a code of the last handled data set. Default value: "fillUnknownDataSetSizeTaskLastSeen"


A time interval (in seconds) which defines how often the "last-seen-data-set-file" file should be deleted. The parameter can be specified with one of the following time units: ms, msec, s, sec, m, min, h, hours, d, days . Default time unit is sec . Default value: 7 days.

Number of data sets requested from AS in one chunk. Default: 100

time-limitLimit of execution time of this task. The task is stopped before reading next chunk if the time has been used up. This parameter can be specified with one of the following time units: ms, msec, s, sec, m, min, h, hours, d, days. Default time unit is sec.

<task id>.class = ch.systemsx.cisd.etlserver.plugins.FillUnknownDataSetSizeInOpenbisDBFromPathInfoDBMaintenanceTask
<task id>.interval = 86400
<task id>.data-set-chunk-size = 1000
<task id>.time-limit = 1h


Environment: DSS

Description: Creates/updates a mirrot of the data store. Data set are organized hierachical in accordance to their experiment and samples


Property KeyDescription
storeroot-dir-link-pathPath to the root directory of the store as to be used for creating symbolic links. This should be used if the path to the store as seen by clients is different than seen by DSS. (Since 13.04.15)
storeroot-dirPath to the root directory of the store. Used if storeroot-dir-link-path is not specified.
hierarchy-root-dirPath to the root directory of mirrored store.
link-naming-strategy.classFully qualified class name of the strategy to generate the hierarchy (implementing ch.systemsx.cisd.etlserver.plugins.IHierarchicalStorageLinkNamingStrategy). Depending on the actual strategy additional properties, all starting with prefix link-naming-strategy.,  mighty be needed. Default: ch.systemsx.cisd.etlserver.plugins.TemplateBasedLinkNamingStrategy
link-source-subpath.<data set type>Link source subpath for the specified data set type. Only files and folder in this relative path inside a data set will be mirrored. Default: The complete data set folder will be mirroed.
link-from-first-child.<data set type>Flag which specifies whether only the first child of or the complete folder (either the data set or the one specified by link-source-subpath.<data set type>). Default: False

class = ch.systemsx.cisd.etlserver.plugins.HierarchicalStorageUpdater
storeroot-dir = ${root-dir}
hierarchy-root-dir = ../../mirror


Environment: AS

Description: Feeds a report database with recently added or modified materials.


Property KeyDescription
database-driverFully qualified name of the JDBC driver class.
database-urlURL to access the database server.

User name of the database. Default: User who started openBIS AS.

database-passwordOptional password of the database user.

Path to the file containing configuration information of mapping material types and material properties to tables and columns in the report database.


The SQL select statement which returns one column of type time stamp for the time stamp of the last report. If the result set is empty the time stamp is assumed to be 1970-01-01. If the result set has more than one row the first row is used.


The SQL statement which updates or adds a time stamp. The statement has to contain a '?' symbol as the placeholder of the actual time stamp.


The SQL statement to add a time stamp the first time. The statement has to contain a '?' symbol as the placeholder of the actual time stamp. Default: same as update-timestamp-sql.

Example: of AS
<task id>.class = ch.systemsx.cisd.openbis.generic.server.task.MaterialExternalDBSyncTask
<task id>.interval = 120
<task id>.read-timestamp-sql = select timestamp from timestamp
<task id>.update-timestamp-sql = update timestamp set timestamp = ?
<task id>.insert-timestamp-sql = insert into timestamp values(?)
<task id>.mapping-file = ../report-mapping.txt
<task id>.database-driver = org.postgresql.Driver
<task id>.database-url = jdbc:postgresql://localhost/material_reporting

Mapping File

The mapping file is a text file describing the mapping of the data (i.e. material codes and material properties) onto the report database. It makes several assumptions on the database schema:

  • One table per material type. There are only table of materials to be reported.
  • Each table has a column which contains the material code.
    • The entries are unique.
    • The material code is a string not longer than 60 characters.
  • Each table has one column for each property type. Again, there are only column for properties to be reported.
  • The data type of the column should match the data type of the properties:
    • MATERIAL:  only the material code (string) will be reported. Maximum length: 60
    • CONTROLLEDVOCABULARY: the label (if defined) or the code will be reported. Maximum length: 128
    • TIMESTAMP: timestamp
    • INTEGER: integer of any number of bits (maximum 64).
    • REAL: fixed or floating point number 
    • all other data types are mapped to text.

The general format of the mapping file is as follows:

[<Material Type Code>: <table Name>, <code column name>]

<Property Type Code>: <column name>

<Property Type Code>: <column name>


[<Material Type Code>: <table Name>, <code column name>]

<Property Type Code>: <column name>

<Property Type Code>: <column name>



# Some comments

[SIRNA: si_rna, code]
INHIBITOR_OF: suppressed_gene
SEQUENCE: Nucleotide_sequence

Some rules:

  • Empty lines and lines starting with '#' will be ignored.
  • Table and column names can be upper or lower case or mixed.
  • Material type codes and property type codes have to be in upper case.

If you put a foreign key constraint on the material code of one of the material properties, you need to define the constraint checking as DEFERRED in order to not get a constraint violation. The reason is that this task will not order the INSERT statements by its dependencies, but in alphabetical order.

MultiDataSetUnarchivingMaintenanceTask (since 13.04.12)

Environment: DSS

Description: Triggers unarchiving of multi data set archives. Is only needed if the configuration property delay-unarchiving of the Multi Data Set Archiving (since 13.04.11) is set true.

Configuration: No specific properties.

class = ch.systemsx.cisd.openbis.dss.generic.server.plugins.standard.archiver.MultiDataSetUnarchivingMaintenanceTask
interval = 1 d
start = 01:00


Environment: DSS

Description: Feeds the pathinfo database with file paths of all data sets in the store. It can be used as a maintenance task as well as a post registration task. As a maintenance task it is needed to run only once if a PostRegistrationMaintenanceTask is configured. This task assumes a data source with for 'path-info-db'.

If used as a maintenance task the data sets are processed in the order they are registered. The registration time stamp of the last processed data set is the starting point when the task is executed next time.


Property KeyDescription
compute-checksumIf true the CRC32 checksum of all files will be calculated and stored in pathinfo database. Default value: false

Number of data sets requested from AS in one chunk if it is used as a maintenance task. Default: 1000

max-number-of-chunksMaximum number of chunks of size data-set-chunk-size are processed if it is used as a maintenance task. If it is <= 0 and time-limit isn't defined all data sets are processed. Default: 0
time-limitLimit of execution time of this task if it is used as a maintenance task. The task is stopped before reading next chunk if the time has been used up. If it is specified it is an alternative way to limit the number of data sets to be processed instead of specifying  max-number-of-chunks. This parameter can be specified with one of the following time units: ms, msec, s, sec, m, min, h, hours, d, days. Default time unit is sec.

class = ch.systemsx.cisd.etlserver.path.PathInfoDatabaseFeedingTask
execute-only-once = true
compute-checksum = true  


Environment: DSS

Description: Calculates the CRC32 checksum of all files in the pathinfo database with unknown checksum. This task is needed to run only once. It assumes a data source for key 'path-info-db'. 

Configuration: No additional configuration.

class = ch.systemsx.cisd.etlserver.path.PathInfoDatabaseChecksumCalculationTask
execute-only-once = true


Environment: DSS

Description: A tasks which runs a sequence of so-called post-registration tasks for each freshly registered data set.


Property KeyDescription
ignore-data-sets-before-dateDefines a registration date. All data sets registered before this date are ignored. Format: yyyy-MM-dd, where yyyy is a four-digit year, MM is a two-digit month, and dd is a two-digit day. Default value: no restriction.
last-seen-data-set-filePath to a file which stores the code of the last data set successfully post-registered. Default value: last-seen-data-set.txt
cleanup-tasks-folderPath to a folder which stores serialized clean-up tasks always created before a post-registration task is executed. These clean-up tasks are executed on start up of DSS after a server crash. Default value: clean-up-tasks
post-registration-tasksComma-separated list of keys of post-registration task configuration. Each key defines (together with a '.') the prefix of all property keys defining the post-registration task. They are executed in the order their key appear in the list.

class = ch.systemsx.cisd.etlserver.postregistration.PostRegistrationMaintenanceTask
interval = 60
cleanup-tasks-folder = ../cleanup-tasks
ignore-data-sets-before-date = 2011-01-27
last-seen-data-set-file = ../last-seen-data-set
post-registration-tasks = eager-shuffling, eager-archiving
eager-shuffling.class = ch.systemsx.cisd.etlserver.postregistration.EagerShufflingTask
eager-shuffling.share-finder.class = ch.systemsx.cisd.openbis.dss.generic.shared.ExperimentBasedShareFinder
eager-archiving.class = ch.systemsx.cisd.etlserver.postregistration.ArchivingPostRegistrationTask


Environment: AS

Description: Removes unofficial unused vocabulary terms. For more details about unofficial vocabulary terms see Ad Hoc Vocabulary Terms.


Property Key



Unofficial terms are only deleted if they have been registered more than the specified number of days ago. Default: 7 days.

Example: of AS
<task id>.class = ch.systemsx.cisd.openbis.generic.server.task.RemoveUnusedUnofficialTermsMaintenanceTask
<task id>.interval = 86400
<task id>.older-than-days = 30

Screening Maintenance Tasks


Environment: DSS

class = ch.systemsx.cisd.openbis.dss.etl.ComputeIntensityLevelTransformationsMaintenanceTask
data-source = imaging-db

# interval in seconds beetween executions
interval = 60

# the code of the transformation
transformation-code = INTENSITY_LEVEL

# optional. default is "Fixed rescaling"
transformation-label = Intensity level

# optional. The default value is a descriptive information about the transformation.
transformation-description = The informative description

# whether the given transformation should be default
is-default-transformation = false

# specify whether the min/max levels should be calculated or fixed. Default is true (calculate)
compute-min-max-levels = false

# calculation parameter. Mandatory only if compute-min-max-levels = true
intensity-threshold = 0.5

# fixed min level. Mandatory if compute-min-max-levels = false
min-level = 18

# fixed max level. Mandatory if compute-min-max-levels = false
max-level = 103

# the number of datasets processed in one run of the task
batch-size = 5
# the local file where the plugin can store information about already processed data 
status-filename = /a/path/to/the/local/file


Environment: DSS

Configuration: See DeleteFromExternalDBMaintenanceTask

class = ch.systemsx.cisd.openbis.dss.etl.DeleteFromImagingDBMaintenanceTask
data-source = imaging-db

Proteomics Maintenance Tasks

  • No labels