How to run Python script in Singularity container

HPC Technical
1

Hello! I apologise for this basic question, but what is the proper way to run a Python script in a Singularity container and have the results of that script saved in $HOME?

I am trying to run a machine learning Python training script in a Singularity container. The Singularity image is in $HOME, and the script is in $HOME/folder1/folder2/script.py, in the same directory as the input files needed for training.

My bash script has

ml load GCC/9.3.0 Singularity/3.7.3-Go-1.14
singularity run img_name.simg

I have also tried running singularity exec img_name.simg python $HOME/folder1/folder2/script.py in lieu of singularity run, both of which returned

FATAL:   "python": executable file not found in $PATH

My Dockerfile used in building the Singularity image has

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-py38_4.10.3-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-py38_4.10.3-Linux-x86_64.sh -b \
    && rm -f Miniconda3-py38_4.10.3-Linux-x86_64.sh
.
.
.
CMD ["python", "$HOME/folder1/folder2/script.py"]

I would greatly appreciate any help or links to relevant resources.

2

Hi,

To test, could you give the path to the singumarity img.

Could you try to load the Python module :

ml load GCC/9.3.0 Singularity/3.7.3-Go-1.14 Python

I allow myself to copy and test your sbatch:

(yggdrasil)-[gaositi2@login1 ~]$ sbatch lolcow.sh.test
Submitted batch job 13523117

With python module:

(yggdrasil)-[gaositi2@login1 ~]$ cat lolcow-out.test.o13523117
 _______
< hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||
3

Hi, thank you for your reply!

The image is at $HOME/dd3c_pyrun.simg. I have modified my bash script train_w_CelebA.sh to load Python as suggested:

ml load GCC/9.3.0 Singularity/3.7.3-Go-1.14 Python                             
singularity run dd3c_pyrun.simg
singularity exec dd3c_pyrun.simg python $HOME/DD3C_test/DeepDeformable3DCaricatures/train_with_CelebA.py

Unfortunately, it still returns

FATAL:   "python": executable file not found in $PATH

twice in train_w_CelebA-error.e13523597.

4

Hi,

Could give me more information:

How did you create the singularity img ? by yourself or something downloaded (give the url source)

Considering DD3C_test/DeepDeformable3DCaricatures/dockerDockerfile is the Dockerfile to create your image, could you try to add “apt-get install python36” before your wget command

5

Hello, my apologies for the late reply.

I built the Singularity image from a Docker image following the guide here: hpc:applications_and_libraries [eResearch Doc]

In the end I managed to resolve my issue by making a conda environment and downloading all the necessary libraries in the bash script instead.

Thank you nevertheless for your time and help!