Non-accesible temporary folder in Yggdrasil

HPC Technical
1

Hi,

I am working on an R package that I use to run various simulations on the cluster.

In that package, I define a temporary folder to create some files that I use and that I delete afterwards.
In order to create this temporary folder I use an R code very similar to:

# create temporary folders
  working_folder = paste(tempdir(), stri_rand_strings(n=1,length = 16), sep = "/")
  
  if (cleanup == F) {
    message(paste("Working in", working_folder))
  }
  
  dir.create(working_folder, showWarnings = F, recursive = T)

where the function stri_rand_strings() is part of stringi R package and generates (pseudo)random strings of desired lengths.

Until very recently (roughly last week), the package was working perfectly fine in Yggdrasil and the function which use the code snippet above was creating a folder which was accessible and that I could access.

More precisely, the base R tempdir() function was returning a folder similar to:

> tempdir()
[1] "/tmp/RtmpjHYKkF"

However, now, the tempdir() function, when executed on the cluster returns something like:

> tempdir()
[1] "/scratch/RtmpoO7cYI"

which is not an accesible folder in the cluster.

Do you understand why the base R tempdir() function returns a non-accesible folder now and how we could counter this problem?

Just to be clear, the code works fine on a personnal computer and does provides a valid temporary folder that is accesible. It is just on the cluster that I observe the behavior described above, namely that tempdir() produces a non valid path.

Many thanks in advance

Best

Lionel Voirol

2

Hi, why do you say this path is not accessible? It should be.

3

Hello Yann,

Thanks a lot for your reply.

So no, I don’t have access to the absolute path /scratch

Imagine I log in into my session than the current directory is:

/home/users/v/voirolli

From there I could go to potentially:

/home/users/v/voirolli/scratch

with

cd scratch

However, the absolute path /scratch is not accessible while /tmp is accessible, indeed if I specify the absolute path and try to run:

cd /scratch

I obtain the following:

[voirolli@login1.yggdrasil ~]$ cd /scratch
-bash: cd: /scratch: No such file or directory

Hence, tempdir() does return a non-valid path.

Or is there something that I don’t understand?

Many thanks in advance

Best

Lionel Voirol

4

yes!

[sagon@login1.yggdrasil ~]$ ls -ld /scratch
ls: cannot access /scratch: No such file or directory
[sagon@login1.yggdrasil ~]$ srun ls -ld /scratch
drwx------ 2 sagon unige 4096 Jun 22 10:44 /scratch

As you can see, /scratch is available from a compute node, which is normal.

When you create a tempdir, the directory is created in /scratch because we export a variable to this place.

[sagon@login1.yggdrasil ~]$ srun env | grep TMP
TMPDIR=/scratch

Do you have this issue with a job running on a compute node?

5

Hello Yann,

Thanks a lot for your reactive and useful support.

You are right. We do have access to /scratch when on a compute node. I also tested it with:

salloc -n1 --partition=public-interactive-cpu --time=1:00:00 
module load GCC/9.3.0 OpenMPI/4.0.3 R/4.0.0 
Rscript -e "tempdir()"
[1] "/scratch/RtmpBD55A0"
cd /scratch

After careful debugging, the problem was not due to the accessibility of /scratch on a compute node and it’s now solved.

Many thanks

Best

Lionel Voirol

6

At least you learned something new today :wink:

Ref in our doc