Child pages
  • Importing images
Skip to end of metadata
Go to start of metadata

Adding Microscopy support to your existing "openBIS for HCS" server

To activate browsing Microscopy data, you have to:

  • Application Server
    • add a line in web-client.properties:

      data-set-types-with-image-overview = MICROSCOPY_IMAGE
      
    • create MICROSCOPY_IMAGE data set type using the Web Browser
  • DSS
    • add following lines to service.properties:

      overview-plugins = microscopy-image-overview
      
      microscopy-image-overview.class = ch.systemsx.cisd.openbis.dss.generic.server.MergingImagesDownloadServlet
      microscopy-image-overview.dataset-types = MICROSCOPY_IMAGE
      

      or as a core plugin of type imaging-overview-plugins.

Python Dropbox Configuration

Configuration is similar to screening.

Datastore Server Configuration

Create a core plugin of type drop-boxes. The plugin.propertiesreads:

plugin.properties
incoming-dir = ${incoming-root-dir}/incoming-microscopy
incoming-data-completeness-condition = auto-detection
top-level-data-set-handler = ch.systemsx.cisd.openbis.dss.etl.jython.v2.JythonPlateDataSetHandlerV2
script-path = dropbox.py
storage-processor = ch.systemsx.cisd.openbis.dss.etl.MicroscopyStorageProcessor
storage-processor.data-source = imaging-db

It assumes

  • incoming-root-dir is defined in service.properties,
  • the Jython drop box script file dropbox.py.

If the folder incoming-microscopy doesn't exist it will be created on DSS start up.

Jython Script Configuration

The main difference between microscopy drop box and screening drop box is that images are not connected to wells anymore.
This has to be specified by invoking in the script imageDatasetConfig.setMicroscopyData(True) method.

There is a convenience class SimpleImageContainerDataConfig which maps image identifier attributes to image meta data properties.

Here is an example:

import os
from ch.systemsx.cisd.openbis.dss.etl.dto.api.v1 import SimpleImageContainerDataConfig, ChannelColor

class MicroscopyConfig(SimpleImageContainerDataConfig):
    def getChannelColor(self, channelCode):
        dict = { "CHANNEL-1" : ChannelColor.GREEN, "CHANNEL-2" : ChannelColor.RED }
        if channelCode in dict:
            return dict[channelCode]
        else:
            return None

def process(transaction): 
    incoming = transaction.getIncoming()
    if incoming.isDirectory(): 
        imageDatasetConfig = MicroscopyConfig()
        imageDatasetConfig.setRecognizedImageExtensions(['lif', 'dv', 'tif', 'tiff', 'nd2', 'lsm'])
        imageDatasetConfig.setDataSetType("IMG_MY_DATASET_TYPE")
        imageDatasetConfig.setMicroscopyData(True)
        imageDatasetConfig.setImageLibrary("BioFormats")
        dataset = transaction.createNewImageDataSet(imageDatasetConfig, incoming)
        dataset.setSample(transaction.getSample("/DEMO/M3"))
        transaction.moveFile(incoming.getPath(), dataset)

Registration of many data sets for one image container file

An image container file might contain images which should be assigned to different data sets. Instead of making a copy of the image container file (which can be 100 MB or more) there is a way to have this file only once in the data store. Here is a snipped of Jython code which does the trick:

	image_data_set = None
	for i in range(num_series):
		singleDatasetConfig = MicroscopySingleDatasetConfig(bioFormatsProcessor, self._logger, i)
		if image_data_set is None:
			dataset = self._transaction.createNewImageDataSet(singleDatasetConfig, java.io.File(fileName))
			image_data_set = dataset
			self._transaction.moveFile(fileName, image_data_set)
		else:
			dataset = self._transaction.createNewImageDataSetFromDataSet(singleDatasetConfig, image_data_set)
		dataset.setExperiment(experiment)

This code assumes num_series image series in the image container file fileName. There is a class MicroscopySingleDatasetConfig extending SimpleImageContainerDataConfig declared somewhere above the code snippet which knows how to get configuration data from the image container file.

From the first series a normal image data set is created with createNewImageDataSet. Also the image container file is moved to this image data set. Effectively an image data set container is created which has a physical data set for the image container file. Depending on the configuration also thumbnail data sets are created and part of the image data set container.

For all other series of the image container file the method createNewImageDataSetFromDataSet is used to create the corresponding image data set. Instead of a file it gets the image data set created first as an argument. It takes physical data set with the big image container file and adds it to the newly created data set. Thus, the physical data set with the image container file will be a component in num_series image data set containers.

Deprecated - preconfigured dropboxes

  • to import images with tiles/timepoints/depth scans/channels one has to configure a new drop box using a new MicroscopyStorageProcessor storage processor, e.g.:

    storage-processor = ch.systemsx.cisd.openbis.dss.etl.MicroscopyStorageProcessor
    storage-processor.file-extractor = ch.systemsx.cisd.openbis.dss.etl.MicroscopyImageFileExtractor
    storage-processor.data-source = imaging-db
    storage-processor.channel-names = RED, GREEN, BLUE
    storage-processor.extract-single-image-channels = RED, GREEN, BLUE
    storage-processor.well_geometry = 2x3
    storage-processor.tile_mapping = 1,2,3;4,5,6
    

    Note that type-extractor and data-set-info-extractor configuration is the same as for any other drop box.
    The default MicroscopyImageFileExtractor supports following images naming convention:

    s<TILE-NUMBER>_z<DEPTH>_t<TIMEPOINT>_c<CHANNEL-CODE>.<any-file-extension>
    

    So valid file names are e.g.: s1_z3_t5_cDAPI.png, s4_cGFP.gif, s3_z19.23_cGFP.gif
    To support different naming conventions a new plugin has to be written.

  • to import images with any naming convention use the following drop box configuration:

    incoming-dir = <PATH-TO-THE-INCOMING-DIRECTORY>
    incoming-data-completeness-condition = auto-detection
    
    # The extractor class to use for code extraction
    data-set-info-extractor = ch.systemsx.cisd.etlserver.DefaultDataSetInfoExtractor
    data-set-info-extractor.entity-separator = .
    data-set-info-extractor.index-of-sample-code = 0
    data-set-info-extractor.space-code = <SPACE-CODE, e.g. DEMO>
    
    type-extractor = ch.systemsx.cisd.etlserver.SimpleTypeExtractor
    type-extractor.file-format-type = <FILE-FORMAT-TYPE-CODE, e.g. TIFF>
    type-extractor.locator-type = RELATIVE_LOCATION
    type-extractor.data-set-type = MICROSCOPY_IMAGE
    type-extractor.is-measured = true
    
    storage-processor = ch.systemsx.cisd.openbis.dss.etl.MicroscopyBlackboxSeriesStorageProcessor
    storage-processor.data-source = imaging-db
    
  • No labels