[tutorial] use conda in a container

HPC Technical
1

Dear users, many HPC users use Conda to install the required software on the cluster and/or on their laptop.
The advantage of using Conda for the end user is great flexibility and easy software installation.
The downside is that each user installs their own copy of the software in their home or scratch directory, and a typical Conda installation creates thousands of files and puts a lot of pressure on our shared storage.

The good news: it is quite easy to create a container, in our case with [Apptainer] (hpc:applications_and_libraries [eResearch Doc]), that contains all the software you need. The big advantage is that there is only one file, easier to maintain and co use on all the clusters.

In this tutorial, we’ll use a project called cotainr for this, because one of the use cases of this script is to create a container from a yaml environment file that you would use to install your Conda stuff.

  1. Download cotainr:
wget https://github.com/DeiC-HPC/cotainr/archive/refs/tags/2023.11.0.tar.gz
  1. Extract the files tar xf 2023.11.0.tar.gz
  2. Make a symlink to your ~/bin folder so you can execute it without specifying the full path
mkdir ~/bin
cd ~/bin
ln -s ../cotainr/cotainr-2023.11.0/bin/cotainr .
  1. To use cotainr, you need Python 3.8.0 or higher, so let’s load this Python version using module: ml GCCcore/13.2.0 Python/3.11.5.
  2. You will need to create an environment file: my_conda_env.yml which describe what you want to be installed in the container
channels:
  - conda-forge
Dependencies:
  - python=3.11.0
  - numpy=1.23.5

Now you are ready to build your container:

cotainr build my_conda_env_container.sif --baseimage=docker://ubuntu:22.04 --accept-licences --conda-env=my_conda_env.yml
  1. You can now use the container:
apptainer exec my_conda_env_container.sif python3 -c "import numpy; print(numpy.__version__)"
1.23.5`

Even if cotainr doesn’t directly support building an image from a pip requirements file, you can add the pip packages you need to your conda env file.

channels:
  - conda-forge
dependencies:
  - python=3.11.0
  - numpy=1.23.5
  - pip
  - pip:
    - scipy==1.9.3

If you convert an existing Conda environment using cotainr, do not forget to erase the .conda files not needed anymore.

Your feedback is greatly appreciated!