Jupyter Lab and package/libraries installation

[Sajad.Hamzenejadi]

Hello,

I’m new to the HPC and have a few questions. I’d really appreciate your guidance.

I currently have a federated learning model written in a Jupyter notebook. From reading the forum, my understanding is that I can use Baobab and Bamboo through OOD, is that correct? If so, do you have any recommendations on how I choose between them?

Also, my federated learning model requires the following packages:

pip install timm
pip install torchinfo
pip install -q "flwr[simulation]" flwr-datasets
pip install thop

Should I install these packages directly within the JupyterLab environment, or do I need to request that they be installed by the HPC administrators?

Thank you very much for your help!

[Sajad.Hamzenejadi]

Dear @Yann.Sagon

If you have any suggestion/instructions, it is really appreciated!

Many thanks!

[Yann.Sagon]

@Sajad.Hamzenejadi

I can use Baobab and Bamboo through OOD, is that correct?

You can use the three clusters: hpc:how_to_use_openondemand [eResearch Doc]

@Sajad.Hamzenejadi

do you have any recommendations on how I choose between them

Check here:hpc:hpc_clusters [eResearch Doc] but the doc is not very helpful I must admit! My advice is to use Bamboo as this is the more recent cluster with faster storage.

I’ve posted a tutorial that should do what you want:

[Tutorial] Create a Custom Kernel for JupyterLab Using cotainr

This tutorial explains how to build a GPU‑ready container image using cotainr, install PyTorch and other Python packages inside it, and expose it as a custom Jupyter kernel in an HPC environment using Open OnDemand. 1. Create the Conda Environment File (env.yml) Create a file named env.yml with the following content: name: torch-gpu channels: - pytorch - nvidia - conda-forge - defaults dependencies: - python=3.10 - pip - pytorch - torchvision - torchaudio - pytorch-cuda=1…