(warning)  This guide is not meant to substitute the official Docker documentation. Standard Docker commands are present in sections that are not necessarily related with them.

Overview

SIS provides a Docker image for the installation of a JupyterHub server, available at https://hub.docker.com/r/openbis/.

This guide explains how to modify JupyterHub docker images and save them. It is aimed at users who are not familiar with Docker, but it should not be considered a substitute of the official Docker documentation.

(warning) We advise non expert users, to first test the instructions provided in this guide on their local machine, to familiarise themselves with the process, before making changes on the JupyterHub server.

Docker images are stateless, which means that after rebooting all changes made will not be saved. This guarantees a stable environment, which is particularly desirable to run services.

If a user wants to introduce changes, the docker image needs to be updated. There are two possibilities for this: 

Nomenclature

Docker: A computer program that performs operating-system-level virtualisation also known as containerisation. The official website can be found here https://www.docker.com/.

Docker image: Docker images describe the environment to virtualise. Docker images are stateless.

Docker container: Docker containers provide the environment to execute the images.

Prerequisites for testing in a local environment

  1. Docker environment. All examples shown below require a working docker environment. Please visit https://www.docker.com to download the Docker Community Edition for your OS.
  2. JupyterHub Docker image. The jupyterhub-openbis images can be found at https://hub.docker.com/r/openbis/. They can be installed locally like any other Docker Hub image.
  3. openBIS installation (optional).


How to run the official JupyterHub for openBIS image in your local machine


1. After downloading the jupyterhub-openbis, find the id of your image.

$ docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
openbis/jupyterhub-openbis-sis-20180405                 latest              585a9adf333b        23 hours ago        4.75GB


2. Run the image with one of the two following commands:

a. if you want to connect to your productive openBIS instance (e.g. https://openbis-elnlims.ch), use the following command:

docker run -e OPENBIS_URL=https://openbis-elnlims.ch -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p 8002:8002 585a9adf333b ./vagrant/initialize/start_jupyterhub.sh

b. if you have a local openBIS installation for testing, you can run the following command:

docker run -v /Users/juanf/jupyterhub-local/home:/home -v /Users/juanf/jupyterhub-local/config/certificates:/vagrant/config/certificates -e OPENBIS_URL=https://129.132.228.42:8443 -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p 8002:8002 585a9adf333b ./vagrant/initialize/start_jupyterhub.sh


(warning)  Please note the following configuration options:

  1. -v /Users/juanf/jupyterhub-local/home:/home
    This option is only required if you want to store the changes you are making. You need to have a home directory for this. It is not necessary for testing, as the image will provide a default one. This directory should contain a "vagrant" sub directory.

  2. -v /Users/juanf/jupyterhub-local/config/certificates:/vagrant/config/certificates
    This option is only required in production environments where you need valid certificates. It is not necssary for testing, as the image will provide a default one.

  3. OPENBIS_URL=https://129.132.228.42:8443
    By defaut docker is in bridge mode, which means that your docker container accesses your local machine network directly through it. If you have a local openBIS installation please use your IP address; if you use a server installation use the typical address you use to access it.


To stop a running docker container, run "docker kill container_ID". 

The container_ID can be found by running the command "docker ps".

How to extend the official JupyterHub for openBIS image

Modify a currently running container (for testing)

1. Find the container id of the image currently running.

$ docker ps
CONTAINER ID        IMAGE                				COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
a2b76d1dd204        jupyterhub-openbis-sis-20180405   "./vagrant/initial..."   4 seconds ago       Up 2 seconds        0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8081->8081/tcp   nervous_leakey

2. Log into the container.

$ docker exec -it a2b76d1dd204 bash

Add Python Library

Add a new library to Python 3

# First we should move to the environment used by JupyterHub
[root@a2b76d1dd204 /]# export PATH=/vagrant_installation/miniconda3/bin:$PATH
[root@a2b76d1dd204 /]# export LC_ALL=en_US.utf8
[root@a2b76d1dd204 /]# export LANG=en_US.utf8
# Install a new python lib using pip
[root@a2b76d1dd204 /]# python --version
Python 3.6.4 :: Anaconda, Inc.
[root@a2b76d1dd204 /]# pip install prettytable

This type of changes can be validated straightaway in JupyterHub, by just starting a Python 3 notebook. Other changes could require to reboot JupyterHub.

Please note that this approach should only be used for testing. To preserve the changes, the running container should be saved as a new image, otherwise when the container is shutdown these changes will be lost.

Add R Library

Add a new library to R

# First we should move to the environment used by JupyterHub
[root@a2b76d1dd204 /]# export PATH=/vagrant_installation/miniconda3/bin:$PATH
[root@a2b76d1dd204 /]# export LC_ALL=en_US.utf8
[root@a2b76d1dd204 /]# export LANG=en_US.utf8
# Install a new r lib using conda
[root@a2b76d1dd204 /]# sudo conda list r-
[root@a2b76d1dd204 /]# sudo conda install -c r -y r-base64enc

This type of changes can be validated straightaway in JupyterHub, by just starting a R notebook. Other changes could require to reboot JupyterHub.

Save the state of a running container as a new image 

 

If you know that you have made significant changes that you want to keep until you build a new docker recipe, you have the option to save the running container as a new image. 

 

bs-mbpr28:jupyterhub_reference_installation juanf$ docker ps
CONTAINER ID        IMAGE                             COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
a2b76d1dd204        jupyterhub-openbis-sis-20180405   "./vagrant/initial..."   37 minutes ago      Up 37 minutes       0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8081->8081/tcp   lucid_stonebraker

$ docker commit a2b76d1dd204 jupyterhub-openbis-sis-juanextensions-20180406
sha256:5dd0036664c75a21d6a62b80bf5780e70fcad345bb12a7ad248d01e29a3caa99
$ docker images
REPOSITORY                                       TAG                 IMAGE ID            CREATED             SIZE
jupyterhub-openbis-sis-juanextensions-20180406   latest              5dd0036664c7        4 seconds ago       4.75GB
jupyterhub-openbis-sis-20180405                  latest              585a9adf333b        23 hours ago        4.75GB


Extend a docker image using a docker recipe (for maintenance)

The recommended approach for maintenance purposes is to extend the latest official docker image distributed by SIS.

Using our last example, let's create a file called "Dockerfile" and with the content shown below.

# vim:set ft=dockerfile:
FROM openbis/jupyterhub-openbis-sis-20180405
## Adding Python 3 library
RUN export PATH=/vagrant_installation/miniconda3/bin:$PATH && \
	export LC_ALL=en_US.utf8 && \
	export LANG=en_US.utf8 && \
	pip install prettytable

Please change the name of the image in the file to the one you are using.

Now we can create a new image using as a starting point the latest from the official repository.

(warning) It is best practice to include both the name of the user and the creation date in the image name. This will help when dealing with many versions created by different users at different times.

$ docker build -t jupyterhub-openbis-sis-juanextensions-recipe-20180406 .
Sending build context to Docker daemon  4.957GB
Step 1/2 : FROM openbis/jupyterhub-openbis-sis-20180405
....
Step 2/2 : RUN export PATH=/vagrant_installation/miniconda3/bin:$PATH && 	export LC_ALL=en_US.utf8 && 	export LANG=en_US.utf8 && 	pip install prettytable
....
Successfully tagged jupyterhub-openbis-sis-juanextensions-recipe-20180406:latest
 

The new image is now available and can be started as described above.

$ docker images
REPOSITORY                                              TAG                 IMAGE ID            CREATED             SIZE
jupyterhub-openbis-sis-juanextensions-recipe-20180406   latest              a0106501b223        3 minutes ago       4.75GB
openbis/jupyterhub-openbis-sis-20180405                 latest              585a9adf333b        23 hours ago        4.75GB

How to start a jupyterhub-openbis docker image on a productive JupyterHub server


(warning) You can only have ONE jupyterhub-openbis image running on a server at one given time, since JupyterHub makes use of certain ports on the machine that are also configured in openBIS.

1. Find the jupyterhub-openbis-start.sh file in your server (please ask your admin).

2. Find the container id of the image that is currently running.

$ docker ps
CONTAINER ID        IMAGE                				COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
a2b76d1dd204        jupyterhub-openbis-sis-20180405   "./vagrant/initial..."   4 seconds ago       Up 2 seconds        0.0.0.0:8000-8002->8000-8002/tcp, 0.0.0.0:8081->8081/tcp   nervous_leakey

3. Stop the current container.

$ docker kill a2b76d1dd204
a2b76d1dd204

4. Edit the  jupyterhub-openbis-start.sh file in your server and update the name of the image it runs to the one of your choice

docker run -v /Users/juanf/Documents/programming/git/jupyter-openbis-integration/jupyterhub_reference_installation/home:/home -v /Users/juanf/Documents/programming/git/jupyter-openbis-integration/jupyterhub_reference_installation/vagrant/config/certificates:/vagrant/config/certificates -e OPENBIS_URL=https://129.132.229.37:8443 -e JUPYTERHUB_INTEGRATION_SERVICE_PORT=8002 -e JUPYTERHUB_PORT=8000 -e CERTIFICATE_KEY=/vagrant/config/certificates/default.key -e CERTIFICATE_CRT=/vagrant/config/certificates/default.crt -p 8000:8000 -p 8081:8081 -p 8001:8001 -p 8002:8002 jupyterhub-openbis-sis-20180405 ./vagrant/initialize/start_jupyterhub.sh

5. Start the new image.

$ ./jupyterhub-openbis-start.sh

Other useful Docker commands

Save an image as a tar file to share it

(warning) It is best practice to include both the name of the user and the creation date in the image name. This will help when dealing with many versions created by different users at different times.

$ docker save jupyterhub-openbis-sis-20180405 > jupyterhub-openbis-sis-20180405.tar
$ ls -lah
total 9681080
-rw-r--r--  1 juanf  1029   4.6G Apr  5 15:38 jupyterhub-openbis-sis-20180405.tar

Load an image from a tar file

$ docker load < jupyterhub-openbis-sis-20180405.tar
8feeda13d3ce: Loading layer [==================================================>]  27.65kB/27.65kB
622cd2c170f3: Loading layer [==================================================>]    152MB/152MB
633fa40a6caa: Loading layer [==================================================>]  2.048kB/2.048kB
7219a9159e4f: Loading layer [==================================================>]  223.9MB/223.9MB
678b55e862c7: Loading layer [==================================================>]  4.377GB/4.377GB
Loaded image: jupyterhub-openbis-sis-20180405:latest
$ docker images
REPOSITORY                        TAG                 IMAGE ID            CREATED             SIZE
jupyterhub-openbis-sis-20180405   latest              585a9adf333b        24 hours ago        4.75GB

Remove an image

$ docker rmi jupyterhub-openbis-sis-juanextensions-recipe-20180406

Remove all stopped containers

$ docker rm $(docker ps -aq)