How to chain port forwarding to cluster node?

My current workflow involves running a master script (or often interactive console) on a compute node, and the using that to generate a dask-jobqueue cluster object. This cluster object in turn spawns workers across multiple nodes and coordinated functions being mapped to these workers.

Dask offers some very nice web-based monitoring functionality for these runs. Particularly, it makes available a dashboard on port 8787.

My old workflow involved no computation on the master script, so I would just run the coordinating job on the login node. This allowed me to run:

ssh MYUSER@login1.yggdrasil.hpc.unige.ch -L localhost:8787:login1:8787

which would map the login node port 8787 onto my local machine where I would access the web interface.

Now I am looking to do some precursor computation on the master script and would like to move it away from the login node. This provides the complication, however, that I’m not sure how to effectively port chain 8787 from localhost to login1 to gpu006 for example.

Is this possible? Is there a recommended way to do it?

Port forwarding is also available from the computation node to the login one! You could do it via the standard ssh command: ssh -N -f -L localhost:8787:localhost:8787 gpu006.

So, the whole setup is following:

  1. Connect to login node with port forwarding
  2. From login node connect to computation node with the same port
  3. Open the web interface on your local machine!

(tested with forwarding jupyter notebook from computation node)

P.S. I am not official HPC representative.

Oh interesting! Thanks Yury! I tried something similar but it failed out with a non-descriptive error. However I only used the -L option and not -N -f so that is probably part of it.

Hi,

instead of:

why don’t you just modfiy as such:

ssh MYUSER@login1.yggdrasil.hpc.unige.ch -L localhost:8787:gpu006:8787

Anyway, if you only wants to access “web tunneling”, the cleanest is to use a socks proxy.

Example:

ssh ${user}@${cluster} -D 5000

Then on Firefox (example)

Once done, you can directly open an url from Firefox like this one: https://gpu006:8787 without the needs to modify your tunnel based on the node that was allocated.

Best

Yann

Ahh @Yann.Sagon this is perfect. I actually wasn’t able to get the forwarding using the earlier method working in the end, something about permission being denied to map the requested port, so this seems like a much better method, especially since indeed it’s just a web app.

Cheers!

@Yann.Sagon , The socks option works very well for me but when using the connection I opened using ssh -D 5000 I get frequent printed errors like the following:

channel 26: open failed: administratively prohibited: open failed

Which don’t actually seem to break anything, but is annoying when I’m trying to interact with the terminal, as it will print onto the currently active line.

For reference, I’m on Ubuntu 20.04 LTS using Brave (effectively the same as chromium) as my browser, With my Ubuntu connection settings set as follows:

Any advice?
Thanks!
Berk

Hi, a quick workaround is to open a second ssh connection without the -D flag and work from the latest.

Best

1 Like