Problems Connecting to Baobab via VSCode's "Remote - SSH"

[Imahn.Shekhzadeh1]

Hi,

I’m facing issues connecting to baobab/yggdrasil via VisualStudioCode. Usually, I use the extension
Remote - SSH to connect to the cluster, but since yesterday, I have been receiving this time out error:

Could not establish connection to “baobab”.

Do you have an idea why this is happening and potentially how to fix it? My VSCode is up to date.

Also, please note that this problems only concerns baobab, not yggdrasil, so I’m assuming there’s a problem on the HPC side rather than VSCode.

So maybe this is a VSCode issue rather than an HPC issue?

Cheers,
Imahn

[Bharathkumar.Radhakrishnan]

I’m facing the same problem since yesterday evening. Connecting using SSH keys from the terminal works normally but with VSCode I get the exact same error you’re getting.

Here’s the detailed error log for the admins if it helps

[12:28:15.545] Log Level: 2
[12:28:15.553] SSH Resolver called for "ssh-remote+cluster", attempt 1
[12:28:15.553] "remote.SSH.useLocalServer": true
[12:28:15.553] "remote.SSH.path": undefined
[12:28:15.554] "remote.SSH.configFile": undefined
[12:28:15.554] "remote.SSH.useFlock": true
[12:28:15.554] "remote.SSH.lockfilesInTmp": false
[12:28:15.554] "remote.SSH.localServerDownload": auto
[12:28:15.554] "remote.SSH.remoteServerListenOnSocket": false
[12:28:15.554] "remote.SSH.showLoginTerminal": false
[12:28:15.554] "remote.SSH.defaultExtensions": []
[12:28:15.554] "remote.SSH.loglevel": 2
[12:28:15.554] "remote.SSH.enableDynamicForwarding": true
[12:28:15.554] "remote.SSH.enableRemoteCommand": false
[12:28:15.554] "remote.SSH.serverPickPortsFromRange": {}
[12:28:15.554] "remote.SSH.serverInstallPath": {}
[12:28:15.557] VS Code version: 1.81.1
[12:28:15.557] Remote-SSH version: remote-ssh@0.102.0
[12:28:15.557] darwin arm64
[12:28:15.558] SSH Resolver called for host: cluster
[12:28:15.558] Setting up SSH remote "cluster"
[12:28:15.560] Acquiring local install lock: /var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-remote-ssh-bfc1474d-install.lock
[12:28:15.560] Looking for existing server data file at /Users/bharath/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-bfc1474d-6c3e3dba23e8fadc360aed75ce363ba185c49794-0.102.0/data.json
[12:28:15.560] Using commit id "6c3e3dba23e8fadc360aed75ce363ba185c49794" and quality "stable" for server
[12:28:15.562] Install and start server if needed
[12:28:15.563] PATH: /opt/homebrew/opt/openssl@3/bin:/Users/bharath/miniconda/bin:/Users/bharath/miniconda/condabin:/Users/bharath/.cargo/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/bharath/Library/Python/3.8/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/opt/X11/bin:/Library/Apple/usr/bin:/Library/TeX/texbin
[12:28:15.563] Checking ssh with "ssh -V"
[12:28:15.571] > OpenSSH_9.3p2, LibreSSL 3.3.6

[12:28:15.573] askpass server listening on /var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-ssh-askpass-855fd81519312150ce52abbd7f1f95bf5ba1af87.sock
[12:28:15.573] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-ssh-askpass-1391b346c77bb33a94e06e4c462e3e3bf03abc88.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","64437","-o","ConnectTimeout=15","cluster"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/bharath/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-bfc1474d-6c3e3dba23e8fadc360aed75ce363ba185c49794-0.102.0/data.json"}
[12:28:15.574] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.YqNmKHLvfK/Listeners","SHELL":"/bin/zsh","DISPLAY":"/private/tmp/com.apple.launchd.njm9xjN3zS/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/bharath/.vscode/extensions/ms-vscode-remote.remote-ssh-0.102.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/bharath/.vscode/extensions/ms-vscode-remote.remote-ssh-0.102.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-ssh-askpass-855fd81519312150ce52abbd7f1f95bf5ba1af87.sock"}
[12:28:15.574] Spawned 9939
[12:28:15.649] > local-server-1> Running ssh connection command: "-v -T -D 64437 -o ConnectTimeout=15 cluster"
[12:28:15.650] > local-server-1> Spawned ssh, pid=9947
[12:28:15.654] stderr> OpenSSH_9.3p2, LibreSSL 3.3.6
[12:28:15.687] stderr> debug1: Server host key: ssh-rsa SHA256:tKqp4nljL+EGVKl8T0VF2nS36DkHVFMpLxQOPg/gKvg
[12:28:15.794] stderr> Authenticated to login2.baobab.hpc.unige.ch ([129.194.9.190]:22) using "publickey".
[12:28:15.976] > ready: 3859b66d16fd
[12:28:15.981] > Linux 4.18.0-477.10.1.el8_8.x86_64 #1 SMP Tue May 16 11:38:37 UTC 2023
[12:28:15.981] Platform: linux
[12:28:15.985] > /bin/bash
[12:28:15.986] Shell: bash
[12:28:15.986] Creating bash subshell inside "bash"
[12:28:15.992] > bash version:  4.4.20(1)-release
[12:28:16.006] > 3859b66d16fd: running
[12:28:16.029] > Acquiring lock on /home/users/r/radhakrb/.vscode-server/bin/6c3e3dba23e8fadc360aed75ce363ba185c49794/vscode-remote-lock.radhakrb.6c3e3dba23e8fadc360aed75ce363ba185c49794
[12:28:16.032] > Installation already in progress...
> If you continue to see this message, you can try toggling the remote.SSH.useFlock setting
> 3859b66d16fd: start
> exitCode==24==
> listeningOn====
> osReleaseId==rocky==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/tmp==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> extInstallTime====
> serverStartTime====
> 3859b66d16fd: end
[12:28:16.032] Received install output: 
exitCode==24==
listeningOn====
osReleaseId==rocky==
arch==x86_64==
vscodeArch==x64==
bitness==64==
tmpDir==/tmp==
platform==linux==
unpackResult====
didLocalDownload==0==
downloadTime====
installTime====
extInstallTime====
serverStartTime====

[12:28:16.032] Server installation process already in progress - waiting and retrying
[12:28:16.032] Terminating local server
[12:28:16.037] Local server exit: 15
[12:28:17.041] askpass server listening on /var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-ssh-askpass-f2d81a6204f639def371a895132f0a6c281acdea.sock
[12:28:17.042] Spawning local server with {"serverId":2,"ipcHandlePath":"/var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-ssh-askpass-ac24f8e880b784076d0ae5a66aeb68f95d6f2b48.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","64437","-o","ConnectTimeout=15","cluster"],"serverDataFolderName":".vscode-server","dataFilePath":"/Users/bharath/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-bfc1474d-6c3e3dba23e8fadc360aed75ce363ba185c49794-0.102.0/data.json"}
[12:28:17.042] Local server env: {"SSH_AUTH_SOCK":"/private/tmp/com.apple.launchd.YqNmKHLvfK/Listeners","SHELL":"/bin/zsh","DISPLAY":"/private/tmp/com.apple.launchd.njm9xjN3zS/org.xquartz:0","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/bharath/.vscode/extensions/ms-vscode-remote.remote-ssh-0.102.0/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Plugin).app/Contents/MacOS/Code Helper (Plugin)","VSCODE_SSH_ASKPASS_EXTRA_ARGS":"--ms-enable-electron-run-as-node","VSCODE_SSH_ASKPASS_MAIN":"/Users/bharath/.vscode/extensions/ms-vscode-remote.remote-ssh-0.102.0/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/fl/l0mwrp4j43n4_2ckx_x_nx5h0000gn/T/vscode-ssh-askpass-f2d81a6204f639def371a895132f0a6c281acdea.sock"}
[12:28:17.044] Spawned 9948
[12:28:17.128] > local-server-2> Running ssh connection command: "-v -T -D 64437 -o ConnectTimeout=15 cluster"
[12:28:17.130] > local-server-2> Spawned ssh, pid=9956
[12:28:17.133] stderr> OpenSSH_9.3p2, LibreSSL 3.3.6
[12:28:17.160] stderr> debug1: Server host key: ssh-rsa SHA256:tKqp4nljL+EGVKl8T0VF2nS36DkHVFMpLxQOPg/gKvg
[12:28:17.263] stderr> Authenticated to login2.baobab.hpc.unige.ch ([129.194.9.190]:22) using "publickey".
[12:28:17.449] > ready: 3859b66d16fd
[12:28:17.454] > Linux 4.18.0-477.10.1.el8_8.x86_64 #1 SMP Tue May 16 11:38:37 UTC 2023
[12:28:17.455] Platform: linux
[12:28:17.459] > /bin/bash
[12:28:17.459] Shell: bash
[12:28:17.459] Creating bash subshell inside "bash"
[12:28:17.476] > bash version:  4.4.20(1)-release
[12:28:17.490] > 3859b66d16fd: running
[12:28:17.513] > Acquiring lock on /home/users/r/radhakrb/.vscode-server/bin/6c3e3dba23e8fadc360aed75ce363ba185c49794/vscode-remote-lock.radhakrb.6c3e3dba23e8fadc360aed75ce363ba185c49794
[12:28:17.514] > Installation already in progress...
> If you continue to see this message, you can try toggling the remote.SSH.useFlock setting
> 3859b66d16fd: start
> exitCode==24==
> listeningOn====
> osReleaseId==rocky==
> arch==x86_64==
> vscodeArch==x64==
> bitness==64==
> tmpDir==/tmp==
> platform==linux==
> unpackResult====
> didLocalDownload==0==
> downloadTime====
> installTime====
> extInstallTime====
> serverStartTime====
> 3859b66d16fd: end

The above error log keeps repeating a few times (because it retries to connect a bunch of times) and then stops

Cheers
Bharath

[Omar.Darwish]

Same problem for me, I get

Could not establish connection to "login2.baobab.hpc.unige.ch": Got bad result from install script.

[Imahn.Shekhzadeh1]

Issue is solved for me by now

[Yann.Sagon]

Hi, rebooting login2 indeed fixed the issue. But the real issue is that login2 was blocked… probably by a vscode process.

A process named “node” was running a process named “rg” that was blocked and unkillable. We’ll contact the user directly to check if he did something unusual.

In fact this isn’t the first time that we had issues with process launched by vscode and it is hard to debug. Maybe an issue is that the user accessed a directory with too many files in it or accessed to a remote location such as fuse/gio.

Suggestions welcome!

[Debajyoti.Sengupta]

I have been having issues connecting to a node through VS-Code since Sunday as well. I attempted deleting the .vscode-server folder, and followed the instructions for the new ssh-key setup ([Authentication] modification sshPublicKey managment)

It is able to connect briefly, but as soon as I try to open a file, it attempts a reconnect.
I am attaching the dump for your persual (VS-Code ssh dump - Pastebin.com)

[John.Raine]

@Debajyoti.Sengupta Looks to be a known issue and a fix is coming in an update.
Can set a setting to resolve for now

[Remote-SSH Bug]: Unknown reconnection token

[Imahn.Shekhzadeh1]

@Debajyoti.Sengupta @John.Raine For me, setting that setting to false did not work, so I reset the version to 0.105.1 and now I can connect via ssh in VSCode again!