ProxyJump ssh not working on Baobab

Berk.Gercek · September 27, 2023, 11:58am

Hello,

I have resolved the LDAP issue I was having earlier, but I have an issue attempting to use ProxyJump to ssh in to the node which I have reserved. My ~/.ssh/config is set up guided by this post:

[tutorial] ssh tunneling and socks proxy

From time to time, it may be interesting to be able to connect to a compute node from outside.

Example use case: you are running a JupyterNotebook on a compute node and you need to connect to this instance using your desktop’s web browser.

In this case, the solutions are:

create an ssh tunnel using the login node as “gateway”

create a socks proxy and use it from your desktop’s browser

other not described here

ssh tunnel

Example configuration to be added to your .ssh/config (in your laptop/desktop, not the cluster) to connect directly to a compute node from outside the cluster:
Host baobab
  HostName login2.baobab.hpc.unige.ch
  User your_user_name

Host node001
  HostName node001
  User your_user_name
  ProxyJump baobab
  LocalForward 1234 localhost:1234
You can now connect to the host node001 using ssh.
ysagon@lyann-dell-portable:~$ ssh baobab-node 
Last login: Wed Nov 17 15:08:22 2021 from login2.cluster
Installed: Fri Sep 10 09:05:36 CEST 2021
This will connect to the node through the baobab login node thanks to the ProxyJump pragma. The pragma LocalForward will forward the the port 1234 on the compute node to your pc where you instantiated the ssh connection.

From your browser, you should then open an url like this one: http://localhost:1234.

This method is working fine and quite secure, but not very flexible as you’ll have to update your ssh tunnel configuration according to the compute node where you are running your JupyterNotebook instance.

socks proxy

Another option is to connect to the login node using ssh and use the DynamicForward pragma.
Host baobab
  HostName login2.baobab.hpc.unige.ch
  User sagon
  DynamicForward 5000
When you connect to this host using ssh, this creates a socks proxy.

You can then configure your browser to use this proxy. Example for firefox:

image822×847 78.6 KB

You can then enter directly in the url the final destination: http://node001:1234. The advantage is that you don’t need to update the ssh configuration if you connect to another node.

ref: How to chain port forwarding to cluster node?

edit: correct ssh config.

With the actual contents being:

Host baobab
    HostName login2.baobab.hpc.unige.ch
    User gercek

#Host cpu*
#    HostName %h
#    User gercek
#    ProxyJump baobab

Host cpu327
    HostName cpu327
    User gercek
    ProxyJump baobab

When I attempt to connect to node cpu327 (where I have a running reservation) I am prompted for a password which does not accept my usual account pw. This occurs when I use the above configuration, as well as when I use the wildcard Host block in the config file instead of the cpu327 specific one.

Any advice on a workaround?

Regards,
Berk

Berk.Gercek · September 27, 2023, 12:03pm

For what it’s worth, this scheme works for Yggdrasil (even the wildcard Host entry)

Yann.Sagon · September 27, 2023, 12:13pm

Hello,

please check this post:

This is weird, we’ll check what is going on.

Berk.Gercek · September 27, 2023, 1:54pm

Thanks! For the moment, if anyone else is facing similar issues, this setup in my ssh config file works:

Host baobab
    HostName login2.baobab.hpc.unige.ch
    User gercek

Host cpu*
    HostName login2.baobab.hpc.unige.ch
    User gercek
    RequestTTY force
    RemoteCommand ssh %n

I think it works because HostBasedAuthentication isn’t used when using ProxyJump, so instead I just define the host cpu* as the login node and manually run a command to ssh into the passed node name.

Berk.Gercek · October 2, 2023, 6:06pm

@Yann.Sagon is there any news on this front? The above workaround is fine for ssh only, and when I try to use it to connect VS code to a compute allocation it fails and results in the VS code instance being run on the login node.

Yann.Sagon · October 4, 2023, 3:45pm

Hi,

we had an issue: it wasn’t possible to connect to a node from login2 without a key. This is now solved and the HostBasedAuthentication is working again. not sure if this helps you?

We’ll apply the fix to Yggdrasil during the maintenance next week.

Berk.Gercek · October 4, 2023, 4:22pm

No, unfortunately it’s still producing the same error: it requests a password when attempting to connect to an allocation. My Unige password does not work.

In this case cpu205 is where I have a job, and the following config in ~/.ssh/config

Host baobab
    HostName login2.baobab.hpc.unige.ch
    User gercek
    RequestTTY force

Host cpu*
    Hostname %h
    User gercek
    ForwardX11 yes
    ProxyJump baobab

results in a successful proxy jump into baobab but prompts password for gercek@cpu205.

The issue is that the expected behavior when using ProxyJump in ssh is to forward the authentication agent from the local machine to the final target, meaning that the login node and the keys necessary aboard for node connections are bypassed. The only way around this is the previous post I made, which executes ssh again on the jump host. This however breaks VS code and is really not great.

Edit: Potential solution:
Is there a reason you don’t propagate our public keys to the nodes as well? This would solve the issue completely I believe, since we wouldn’t need the baobab login node host key to connect, but the nodes would still be inaccessible to those who don’t have access to the login node.

Prior to the switch to LDAP, the ~/.ssh/authorized_keys file was shared among the login and compute nodes. This would allow a ProxyJump config to work as the node authentication agent would use that file to determine the user’s host key validity. Now that we use LDAP, it seems the LDAP public keys only live on the login node for some reason. That is why this broke recently.

If you want to keep people from running remote VS code process on the login node, this is a good way to encourage it, by allowing it to work on compute nodes.

Yann.Sagon · October 6, 2023, 12:50pm

I guess we still allows that, don’t we? I’m able to connect to cpu001.baobab directly from my laptop using ProxyJump:

[ysagon@localhost ~]$ ssh cpu001
Last login: Mon Sep 11 16:00:25 2023 from 192.168.100.12
Installed: Thu Aug 17 14:28:26 CEST 2023

This is my ssh config on my laptop:

ysagon@localhost ~]$ cat .ssh/config
Host bao
   HostName login2.baobab.hpc.unige.ch
   User sagon
Host cpu*
   HostName cpu%h (wrong) <<< typo>>> HostName %h (correct)
   User sagon
   ProxyJump bao

I’ve added as well my public key to Baobab .ssh/authorized_keys and I’m having the corresponding private key loaded on my laptop’s ssh-agent.

Berk.Gercek · October 10, 2023, 11:49am

For whatever reason the issue seems to have resolved itself! I changed nothing since my last post but it now works directly using SSH to a node where I have an allocation. Thank you!

Yann.Sagon · October 10, 2023, 12:19pm

Computing is magical

Tomke.Schroeer · October 13, 2023, 7:03am

Hi @Yann.Sagon,

despite having followed all suggested changes in the ssh config etc. I am still facing the problem described by @Berk.Gercek. Is there anything else I could try or check to solve this?

Thanks in advance,
Tomke

John.Raine · October 23, 2023, 1:27pm

@Yann.Sagon @Adrien.Albert @Gael.Rossignol would you have any ideas why the proxy jump doesn’t work for @Tomke.Schroeer ?

Running ssh -J -vvv login2.baobab.hpc.unige.ch cpu212 the ssh key is accepted by the login node but the same key when sent to the worker node is rejected (receive packet: type 51).
It then defaults to asking for a password, but the ISIS+ password is also not accepted.

Unfortunately I cannot reproduce the issue with my ssh config and user account.

Cheers,
Johnny

Yann.Sagon · October 23, 2023, 2:50pm

Dear Johnny,

I did a typo, please see the edited config.

Anyway it seems there is a real issue: it is working and not for my colleague @Adrien.Albert. I’m testing from Rocky9. @Adrien.Albert will investigate tomorrow.

Best

Adrien.Albert · October 24, 2023, 8:13am

Hi @Tomke.Schroeer

Which OS and kernel version do you use ?

Adrien.Albert · October 24, 2023, 9:30am

Hi @Tomke.Schroeer

I did the procedure again starting from the begening and it’s working.

On your local machine, Save old ssh key and create a new one

mkdir ~/.ssh/old
mv ~/.ssh/*  ~/.ssh/old
ssh-keygen

On the cluster, make sure you have not id_rsa key (make a back up too)

Copy the rsa.pub in https://my-account.unige.ch/main/home end wait for 5-10 min the synchronisation with AD is done.

3 On your local machine configure the proxyjump:

[alberta@localhost .ssh]$ cat ~/.ssh/config

host bao
   HostName login2.baobab.hpc.unige.ch
   User alberta
Host cpu*
   HostName %h
   User alberta
   ProxyJump bao

copy your public in the authorized_key_file by running:

[alberta@localhost .ssh]$ ssh-copy-id -f login2.baobab.hpc.unige.ch
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/alberta/.ssh/id_rsa.pub"

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'login2.baobab.hpc.unige.ch'"
and check to make sure that only the key(s) you wanted were added.

I check and my authorized key files contains my last key (and others)

Alloc a test job and open a new tab on your local machine and try to connect on the allocated node:

Make sure your test is on Baobab cluster

On baobab:

(baobab)-[alberta@login2 ~]$ salloc
salloc: Pending job allocation 5574654
salloc: job 5574654 queued and waiting for resources
salloc: job 5574654 has been allocated resources
salloc: Granted job allocation 5574654
salloc: Waiting for resource configuration
salloc: Nodes cpu001 are ready for job

On your local machine:
( My first test was on cpu026 this is the message a i got)

[alberta@localhost .ssh]$ ssh cpu026
The authenticity of host 'cpu026 (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:tKqp4nljL+EGVKl8T0VF2nS36DkHVFMpLxQOPg/gKvg.
RSA key fingerprint is MD5:8f:75:c4:18:8a:75:f1:f1:19:4d:85:92:3b:b6:2a:e1.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'cpu026' (RSA) to the list of known hosts.
Last login: Tue Oct 24 10:49:29 2023
Installed: Thu Aug 17 14:40:08 CEST 2023

But working on cpu001 too:

[alberta@localhost ~]$ ssh cpu001
Last login: Mon Oct 23 16:43:34 2023
Installed: Thu Aug 17 14:28:26 CEST 2023
(baobab)-[alberta@cpu001 ~]$

Tomke.Schroeer · October 24, 2023, 2:42pm

Hi @Adrien.Albert

Thanks for the comprehensive instructions! Unfortunately I still get asked for a password when trying to connect to the cpu node directly. I am using macOS Sonoma and kernel version 23.0.0.

Is there anything else I can try? Could there be a general problem with my account?

Cheers,
Tomke

PS: Also when I try to connect to the cpu on baobab from yggdrasil I get asked for a password for schroeer@cpu212

Adrien.Albert · October 24, 2023, 3:00pm

Hello @Tomke.Schroeer ,

I am sorry but I do not have any ( more )idea about your actual issue.

Are you available for a zoom meeting to see more in details what is going wrong ?

Tomke.Schroeer · October 24, 2023, 3:11pm

Hi,

yes sure, let me know which room

Adrien.Albert · October 24, 2023, 3:51pm

Following our Zoom meeting, we reached to connect directly to the compute node from a local machine.

TIPS:

Make sure to back up your .ssh directory on both your local machine and the clusters.

You should not have an id_rsa{,.pub} file in your ssh directory. By default, SSH uses this key, and having a different one from your local machine can cause conflicts.

However, by configuring your SSH profile in .ssh/config and specifying the keyfile used, you may be able to make it work. ( not tested)