My colleague Luca finished the installation and the testing of the new scratch space and you can use right now this space as a working space for your data.
The solutions is RAID6 protected and composed by 120 x 12TB hdds distributed in two servers + JBODS. It should be quite reliable, but keep in mind that this space isn’t backuped.
What you should put here:
simulations results before moving them to their final destination
temporary files
tests
recoverable data (hosted somewhere else)
your many millions small files that prevent the backup of your home directory to work well
What you should not put here:
your source code
your scripts
We have created in your home directory a symbolic link named scratch. We invite you to use this space and to move your existing good candidate data to it.
As the home storage is always almost full, it would be a good idea to do it right now. Thanks!
Feel free to send us your feedback about this new storage place!
Many thanks for this! About the symbolic link. Is it correct if you already had a scratch directory that this new storage is created as ~/scratch/name/? e.g.
in fact no, it was a mistake from us and you can remove the directory rands in the scratch directory.
People who already had a scratch directory in their home will be contacted by us individually. But they can as well proceed as follow:
cd ~
mv scratch scratch.o
ln -s /srv/beegfs/scratch/users/${USER::1}/$USER/ scratch
ls -lad scratch #check the symbolic link is correct. If red, it's incorrect.
mv scratch.o/* scratch
rmdir scratch.o
I am in the process of moving some data to the scratch space. I do however note that the operation is surprisingly slow: upwards of 10mn for a directory of 2-3 Gb, 120k files (mostly logs), direct moving (no copy). Is it normal? Should we expect a loss in I/O performances when running jobs against the scratch space?
maybe it’s taking time due to the relatively high number of files or the storage is under high load.
You are not doing direct moving as the scratch and the home aren’t on the same physical servers. You are really copying data.
You should not expect loss in I/O.