[tutorial] Jupyter notebook

HPC General
1

As Jupyter notebook is getting more and more popular, it’s time to write a small tutorial explaining how you could launch a Jupyter notebook on Baobab.

Jupyter notebook is a web based interface. In this tutorial, we expect that you will connect to Baobab through x2go.

Once connected to Baobab through x2go, you need to write a sbatch script like this one. Adapt it to your need. Be sure to not specify the port number is it may happens it’s already in use and the job will then fail. Jupyter notebook detect if the default port is already in use and use the next one automatically.

In a terminal, submit the job:

[sagon@login2 jupyter][master] $ sbatch launchJupyter.sh
Submitted batch job 20684588
[sagon@login2 jupyter][master] $

Be sure that the Jupyter instance is running (R for running in the ST column):

[sagon@login2 jupyter][master] $ squeue -j 20684588
         JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
      20684588 debug-EL7 launchJu    sagon  R       3:12      1 node001
[sagon@login2 jupyter][master] $

Get the url + token for authentication:

[sagon@login2 jupyter][master] $ grep "running at" -A1 slurm-20684588.out
[I 13:52:43.184 NotebookApp] The Jupyter Notebook is running at:
[I 13:52:43.184 NotebookApp] http://node001:8888/?token=f5282143b8c480822f4bdfa4ae3471ecca767428bbae46e0

On x2go, open firefox and point it to the Jupyter notebook url you got.

image
image770×514 96 KB

You can then create new notebook using the new button or open existing one.

:warning: The session is not available from outside Baobab and is not ciphered. Feel free to adapt this procedure to you need, like activating ssl.

.

3 Likes
2

Thank you ! It is an important tutorial. I run my notebook from a singularity container and it is perfectly compatible also with an ssh tunnel.

3

Hi Yann,

I’m trying to run a Jupyter notebook on the baobab cluster, but I need some dependencies, declared in a requirement.txt and an anaconda environment.yml. How can I install them in the current kernel? Is there any tutorial already available?

I found a tricky hack, put this in a notebook and run the cell:

import sys
!conda install --yes --prefix {sys.prefix} numpy

But I don’t know if there is a better way of achieving this.

EDIT: it seems that the home folder is shared among all the cluster nodes, so I just need to install a package in my local environment so it makes the package available in the notebook.

4

Hi,

if you need dependencies we provide using module, you can just load them or ask us to install them.

Yes your home is shared amongst all the nodes, you can as well just install the dependencies there or better, in the scratch space.

5

hi
thank you for the tuto,
i followed it. I have 6 notebooks reading data from a csv file and writing the results to their file result.
i used the given launchJupyter script but i can’t connect to the jupyter notebook after the job has been started.
Firefox says “Unable to connect”, but i checked and the network connection is working.

Firefox error

Unable to connect

Firefox can’t establish a connection to the server at 127.0.0.1:8799.

The site could be temporarily unavailable or too busy. Try again in a few moments.
If you are unable to load any pages, check your computer’s network connection.
If your computer or network is protected by a firewall or proxy, make sure that Firefox is permitted to access the Web.

Here is my adaptated script :

#!/bin/sh

#SBATCH --partition=public-cpu
#SBATCH --time=08:00:00

#SBATCH --job-name jobname
#SBATCH --error jobname-error.e%j
#SBATCH --output jobname-out.o%j
#SBATCH --cpus-per-task 1

# load Anaconda, this will provide Jupyter as well.
module load Anaconda3/2021.05

export XDG_RUNTIME_DIR=""

# specify here the directory containing your notebooks
JUPYTER_NOTEBOOK_DIR="/home/users/a/amurrio5"

# sepcify here the port to listen to
PORT=8798

# Choose one of the solutions below:

# i uncommented this
# if you want to access jupyternotebook through proxy socks or x2go, it should listen to the node's ip.
IP=$SLURMD_NODENAME

# if you want to access jupyternotebook through an ssh tunnel, it should listen to the localhost (safer)
IP=localhost

srun jupyter notebook --no-browser --port=$PORT --ip=$IP --notebook-dir=$JUPYTER_NOTEBOOK_DIR

Looking at the job, it shows it is running :
squeue -j 58991292
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
58991292 public-cp jobname amurrio5 R 12:38 2 cpu[237-238]

Is there an error in the LaunchJupyter script or anything else that might explain the reason i’m not able to connect to the notebook ?

6

Hi,

You mean, you launch 6 Jupyter notebook together or one by one?

How did you check “network is working”?

In you adapted script, you are using localhost as IP. And as the comment state it, you need to access the instance through an ssh tunnel. Did you setup one?

7

By the way, I suggest that you migrate to JupyterLab, the version we provide is newer: j/jupyter · master · hpc / softs · GitLab

8

I use x2go to connect you in ssh and I used Firefox to check the internet connection

As for the jupyternotebook they are in the directory indicated by the option
I thought that it would launch all the notebooks in the directory

9

I quote myself again:)

I’ve added the link to the ssh tunnel howto. If you don’t know what I’m talking about, comment out IP=localhost.

10

For those interested and thanks to @Adrien.Albert , here is the way to go to add a specific Conda environment as a kernel to Jupyter:

2 Likes
11

2 posts were split to a new topic: Problem launching Jupyterlab

13

A post was split to a new topic: Problem launching jupyterlab on Yggdrasil