Conflicting EasyBuild dependencies

Dear colleagues,

I have an application (geographical spreading epidemic model) compiled against two libraries: CGAL (for computational geometry) and GDAL (for GIS routines).

Both modules exist in Baobab’s EasyBuild, but when I try to load them simultaneously, they seem to use different version of gcc and OpenMPI.

E.g. module spider GDAL/3.0.0-Python-3.7.2 shows these dependencies:

GCC/8.2.0-2.31.1 OpenMPI/3.1.3

and module spider CGAL/4.14.1-Python-3.7.4 shows these:

GCC/8.3.0 OpenMPI/3.1.4

now, it seems to be impossible to load all of these. If I try, the dependency modules are reloaded, and then it exits with the following error message:

Lmod has detected the following error: These module(s) or extension(s) exist but cannot
be loaded as requested: “GDAL/3.0.0-Python-3.7.2”
Try: “module spider GDAL/3.0.0-Python-3.7.2” to see how to load the module(s).

Is there a way to load both modules simultaneously? Or any workaround?

All the best,
Alexander

Hi,
Other than having the admin solve the compatibility problem or installing the easy build in your home directory (cf [HOWTO] compile a software in your home directory ), there is a third solution of using singularity.
Using singularity is more involved but allows to solve complex dependency problem on the cluster.
In short using singularity is done in two steps. First building a docker image on your own machine. Second convert the docker image to singularity image and run on the cluster.

To create a docker image we first choose a base docker image. For demonstration purpose I will take Docker . Which contain an image of docker operating system. In practice you can use another image that contain more of the library you need.

Now in a new empty folder create a file named Dockerfile.
The file has the following content:

from ubuntu:bionic
RUN apt-get update -y
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y libcgal-dev libgdal-dev

This file give the instruction on how to build the image starting from a basic ubuntu bionic image. It start by updating the package list and then install the ubuntu package libcgal-dev and libgdal-dev.

Now we need to build this image and push it to baobab. First create an account on https://hub.docker.com/ . Then create a repository. Finally open a terminal in the same folder where the Dockerfile is located. Login with:

docker login

Build the docker image with:

docker build . -t pablostrasser/demodocker

Where pablostrasser is my account and demodocker is the name of the repository.
Finally push the image with:

docker push pablostrasser/demodocker

Now that the image is pushed on the docker registry, we can download it from the cluster.
On baobab load the singularity module.

module load GCCcore/8.2.0 Singularity/3.4.0-Go-1.12

Now we pull and convert the docker image.

singularity pull demoDocker.simg docker://pablostrasser/demodocker

This command pull our docker image pablostrasser/demodocker and create an image file demoDocker.simg which can be used to execute the image.

You can now execute bash inside the container with:

singularity run demoDocker.simg

By default, your home directory will be mounted and the rest of the container image is read only except /tmp which is mounted to /tmp . You can now compile the file you need in your home directory.
And can execute them with srun and singularity.

I hope this short tutorial help to understand how you can install software on baobab without being root.

Some important remark on singularity:
When building the container with docker build. The user is root which allows to install software without using sudo. However, when executing with singularity the user is a normal user. It is very important to ensure that a normal user can execute the binary you install.

In short docker with singularity allows you to install an arbitrary software by beeing root and execute it on the cluster as normal user.

Dear @Alexander.Temerev

Please see here: New software installed GDAL/3.0.2-Python-3.7.4

Best

Yann

Thank you! It was extremely helpful.