Mounting from one cluster to another

HPC General
1

Dear HPC team,

I often switch between two clusters to be able to use available resources if one of the clusters is being used extensively.

I created a common folder on Yggdrasil to store files and mounted this folder to Baobab so that I do not need to move files between clusters frequently.

I successfully did the mounting with sshfs, and I can access all the files in the folder from the login node.

sshfs -o default_permissions user@login1.yggdrasil.hpc.unige.ch:path1 path2

However, on the compute nodes of Baobab, I can not see all files in the mounted folder.

Could you explain this problem, and would it be possible for me to access the mounted folder from the compute nodes ? Thanks !

Best regards,

Van Khoa

2

Dear @Van-Khoa.Nguyen

As you are only mounting the login, the compute node does not have access to this sshfs mount point.

However, sshfs is not installed on the compute node for a number of reasons:

  • Bandwidth limitations, impacting data transfer efficiency.
  • Latency dependency, impacting the responsiveness of computing tasks.
  • Reliability issues, with potential disruptions to network connectivity.
  • Impact on performance

Keep in mind that Yggdrasil is in Sauverny, Baobab in Uni dufour, and geographical distance has a significant impact on the network.

We are well aware of the need. The project for a common storage space for the clusters is currently under study, and we hope to find a solution compatible with the current IT infrastructure.

Best Regards

2 Likes
3

Dear Van-Khoa,

To complete my collegue answer, there are somme documentation about the sync fo data from both clusters using “rsync” :white_check_mark:

https://doc.eresearch.unige.ch/hpc/best_practices?s[]=rsync

If you can provide more information regarding the path size and the frequency of the data changes, we can help you to find a way.

Best regards,

2 Likes
4

Dear @Adrien.Albert ,

I understand better the problem with sshfs, and thank you for your thoughtful explanation.

Best regards,

Van Khoa

5

Dear @Gael.Rossignol,

Thank you for your answer and the proposed solution.

I will try with your instruction.

Best regards,

Van Khoa

6

Dear @Gael.Rossignol and @Adrien.Albert ,

Can we log in from Yggdrasil to Baobab with passwordless authentication?

I tried to add Yggdrasil’s public key to Baobab by following this tutorial, but it seems not working.

I try to get rid of the password request every time I use rsync. Thanks!

Best regards,

Van Khoa

7

Hi, better use our doc :stuck_out_tongue_winking_eye: hpc:access_the_hpc_clusters [eResearch Doc]

1 Like