Issue with singularity

HPC Technical
1

I also tried to singularity image built from docker:
singularity build scikit.simg docker://jupyter/scipy-notebook
(buile with singularity version: Singularity/2.4.5)

but when I try to start it by :
singularity run scikit.simg
or by
docker run -p 8888:8888 scikit.simg
I get these errors:

[WARN  tini (90053)] Tini is not running as PID 1 .
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, run Tini as PID 1.
[FATAL tini (90057)] exec -p failed: No such file or directory
(virtual_env) [proios0@login2 ~]$ singularity run scikit.simg              
tini (tini version 0.18.0)
Usage: tini [OPTIONS] PROGRAM -- [ARGS] | --version

Execute a program under the supervision of a valid init process (tini)

Command line options:

  --version: S...blabla

PS I do all this to run a scikit&pandas based program
and I need tables for python 3, but in baobab there is only for python 2
PyTables: PyTables/3.3.0-Python-2.7.11 as a temporary solution could we install it ?

2

You can use anaconda to install python dependencies. Anaconda is installed on baobab.

3

Dear Dimitrios,

I moved your post to a new thread as there is no direct relation with the one you posted into (Jupypter notebook). Please try as well to format your post (code etc) as it will be more readable.

We don’t have docker on Baobab, not sure where you tried to launch the container.

Let’s try to build scipy-notebook on Baobab:

module load GCCcore/8.2.0 Singularity/3.4.0-Go-1.12
PATH=$PATH:/sbin/ singularity build scikit.simg docker://jupyter/scipy-notebook

And run the result image:

singularity run scikit.simg 
[WARN  tini (15)] Tini is not running as PID 1 .
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, run Tini as PID 1.
Container must be run with group "users" to update files
Executing the command: jupyter notebook
[I 10:12:02.476 NotebookApp] The port 8888 is already in use, trying another port.
[I 10:12:02.476 NotebookApp] The port 8889 is already in use, trying another port.
[I 10:12:02.529 NotebookApp] JupyterLab extension loaded from /opt/conda/lib/python3.7/site-packages/jupyterlab
[I 10:12:02.529 NotebookApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 10:12:02.531 NotebookApp] Serving notebooks from local directory: /home/sagon/tests
[I 10:12:02.531 NotebookApp] The Jupyter Notebook is running at:
[I 10:12:02.531 NotebookApp] http://login2.cluster:8890/?token=ac88419e74d70e595fa0cb329334b51aa4592c14ca37ad16
[I 10:12:02.531 NotebookApp]  or http://127.0.0.1:8890/?token=ac88419e74d70e595fa0cb329334b51aa4592c14ca37ad16
[I 10:12:02.531 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:12:02.555 NotebookApp]

It works!

Out of curiosity, what is the advantage to launch it like that instead of running Jupyter as explained in my tutorial?

4

Thanks a lot! My major problem is how to run my python project.
For now I am running on login2 node my program and I have my dependencies in a virtual environment. (venv).

A friend told me that they used to zip everything and unzip in the node that a job will run. I guessed maybe with a sif image I could add volumes of my data and source code and send it to other nodes (but maybe volumes are symbolic links and it won’t work)

Jupyter on web could mean I could run quickly things from a web page (bit like google’s collab )

5

Hello,

It’s forbidden to run jobs on the login node. You can compile, edit files etc only. If you need a testing node, you should use the debug-EL7 partition.

Ahh friends:) The filesystems on Baobab are shared on every nodes, including the login nodes. So what is available on the login node is available on the compute nodes. No need to transfer them.

With my tutorial, you can launch jupyter notebook on a compute node available through a browser. But you need to connect to Baobab using x2go to open a browser which has access to your instance.

6

Hello Yann trying to follow your advice:

I try to run this


module load GCC/6.4.0-2.28  OpenMPI/2.1.2
module load PyTorch/0.3.0-Python-3.6.4

## Created with `virtualenv -p `which python3` virtual_env`
source ./virtual_env/bin/activate
module load GCC/6.3.0-2.27 Singularity/2.4.2
ls virtual_env

# ERROR 
python --version
# ERROR 
virtual_env/bin/python3.6  --version

whereis python

I am receiving this error when the batch tries to execute python (I tried with alias python and absolute path ):

##
############### ERRROR START #########################
virtual_env/bin/python3.6: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
############### ERRROR  END ##########################

### paths of python are correct 
python: /usr/bin/python2.7 /usr/bin/python2.7-config /usr/bin/python /usr/lib/python2.7 /usr/lib64/python2.7 /etc/python /usr/include/python2.7 /home/users/p/proios0/src/scripts_baobab/virtual_env/bin/python /home/users/p/proios0/src/scripts_baobab/virtual_env/bin/python3.6 /usr/share/man/man1/pyth
7

Eventually I use the module python instead of the virtual env it seems to have been causing the problem. The dependencies still need to be load from my virtual env. Another problem I have now is that the line os.path.dirname(__file__)which I used in python to determine some relative paths was not resolving correctly.

8

Hello Yan

Loading the same module and after building scikit.simg I am having these error logs:

2019/10/20 22:19:10  info unpack layer: sha256:46470ea16fb56c00a082aae75c1136c9b0b2b1cda0a934ee10c0f5f1ffa5589d
2019/10/20 22:19:10  info unpack layer: sha256:52dc74c3369ea3a85455b96e9b2c54bd9433c1f0c10857fa336f74a2d65ff460
INFO:    Creating SIF file...
INFO:    Build complete: scikit.simg
[proios0@login2 leScript]$ singularity run scikit.simg 
[WARN  tini (15)] Tini is not running as PID 1 .
Zombie processes will not be re-parented to Tini, so zombie reaping won't work.
To fix the problem, run Tini as PID 1.
Container must be run with group "users" to update files
Executing the command: jupyter notebook
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 528, in get
    value = obj._trait_values[self.name]
KeyError: 'runtime_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyter-notebook", line 11, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 266, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 657, in launch_instance
    app.initialize(argv)
  File "</opt/conda/lib/python3.7/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
  File "/opt/conda/lib/python3.7/site-packages/traitlets/config/application.py", line 87, in catch_config_error
    return method(app, *args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line 1676, in initialize
    self.init_configurables()
  File "/opt/conda/lib/python3.7/site-packages/notebook/notebookapp.py", line 1349, in init_configurables
    connection_dir=self.runtime_dir,
  File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 556, in __get__
    return self.get(obj, cls)
  File "/opt/conda/lib/python3.7/site-packages/traitlets/traitlets.py", line 535, in get
    value = self._validate(obj, dynamic_default())
  File "/opt/conda/lib/python3.7/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
    ensure_dir_exists(rd, mode=0o700)
  File "/opt/conda/lib/python3.7/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
    os.makedirs(path, mode=mode)
  File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.7/os.py", line 211, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/opt/conda/lib/python3.7/os.py", line 221, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/run/user'

Also during building there were several warning for rootless containers eg

2019/10/20 22:18:56  info unpack layer: sha256:b519db4443ee8326d4143c01935d7177c9039430006f469aa8e2ffd2142a448b
2019/10/20 22:18:56  warn rootless{opt/conda/bin/libpng-config} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2019/10/20 22:18:56  warn rootless{opt/conda/include/png.h} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2019/10/20 22:18:56  warn rootless{opt/conda/include/pngconf.h} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2019/10/20 22:18:56  warn rootless{opt/conda/include/pnglibconf.h} ignoring (usually) harmless EPERM on setxattr "user.rootlesscontainers"
2019/10/20 22:18:56  warn rootless{opt/conda/lib/libblas.so} ```

Do you have some idea what could be the issue in this case?

Thank you in advance

9

Hello,

according to the man page, by default the python interpreter is the interpreter that virtualenv was installed with. To avoid mistake, it’s better to not specify the -p flag.

You are loading a Singularity 2.4.2 with another dependency that Python 3.6.4 needs. That is the output of your command:

(virtual_env) [sagon@login2 ~] $ module load GCC/6.3.0-2.27 Singularity/2.4.2

Inactive Modules:
  1) FFTW/3.3.7          3) OpenMPI/2.1.2                  5) PyYAML/3.12-Python-3.6.4     7) SQLite/3.21.0                       9) Tcl/8.6.8       11) libyaml/0.1.7
  2) OpenBLAS/0.2.20     4) PyTorch/0.3.0-Python-3.6.4     6) Python/3.6.4                 8) ScaLAPACK/2.0.2-OpenBLAS-0.2.20    10) hwloc/1.11.8

Due to MODULEPATH changes, the following have been reloaded:
  1) CUDA/9.1.85     2) GMP/6.1.2     3) bzip2/1.0.6     4) libffi/3.2.1     5) libreadline/7.0     6) ncurses/6.0     7) numactl/2.0.11     8) zlib/1.2.11

The following have been reloaded with a version change:
  1) GCC/6.4.0-2.28 => GCC/6.3.0-2.27     2) GCCcore/6.4.0 => GCCcore/6.3.0     3) binutils/2.28 => binutils/2.27

As you can see, this has unloaded Python and Pytorch module.

I don’t understand why you need to load Python3 and Pytorch and THEN Singularity. You should choose between the two options: load Singularity and do everything on the image or not load Singularity and work with module/virtualenv.

1 Like
10

Dear Yann thank you a lot for your response,

you are right it’s redundant.

In my last logs I only run these commands after logging in :

module load GCCcore/8.2.0 Singularity/3.4.0-Go-1.12
PATH=$PATH:/sbin/ singularity build scikit.simg docker://jupyter/scipy-notebook
singularity run scikit.simg
11

Regarding jupyter with your helpful tutorial I managed to load jupyter it in X2GO.
I also try to apply local fowarding to see it in my local machine
ssh -L 8888:localhost:8888 uname@host_of_baobab

And in the terminal after logging in and goign to my local’s machine localhost:8888 after each reload I see

channel 2 access denied logs
channel 3 access denied logs`

Best regards

12

From what I gathered from a quick look and your log message the image you are using expect you to be in the users group. For security reason you are not root nor another user than yourself when using singularity. In addition, by default singularity image are not writable (cf singularity help):

-w, --writable by default all Singularity containers are
available as read only. This option makes
the file system accessible as read/write.
–writable-tmpfs makes the file system accessible as
read-write with non persistent data (with
overlay support only)

Only mounted folder are writable. By default your home is mountable.

I hope this help.