Hi everyone,
This is just FYI, in case anyone else experiences the problem of randomly missing R/Python packages when spawning workers accessing a pixi environment library on your HOME directory. 
The script worked if I called 1-5 workers but started to fail when launching 10+ workers at the same time.
Edit: okay, I figured out what was the problem. It was just me being inexperienced on how HPC clusers work 
. Having a central library environment (of almost 100’000 tiny Python and R library files of 5-8GB in total) on my HOME that gets called from the worker nodes is very bad HPC practice from my side (see explanation in the HPC documentation at https[—]://doc.eresearch.unige.ch/hpc/storage_on_hpc#io_performance (cannot poste more than two links, so just delete the “[—]” to be able to follow the link). This is why a container (e.g. Apptainer (formerly Singularity)) should be used, which packages the whole 100k file library environment folder into one .sif file which is much better for HPC BeeFGS file serving between HOME and worker nodes storage. The HPC documentation literally says: “A performance killer is for example to read/write a lot of very small files or to traverse a directory with thousands of files.”. I had to find out the hard way. Seems BeeFGS I/O is very good at transferring a few large files (also very fast, like 2GB/s, transferring my whole 5-8 GB environment, now in one big .sif file, in seconds), while reading thousands or millions of tiny files chokes the I/O, leading to the worker nodes thinking “the file cannot be accessed in a reasonable amount of time, so I have to assume it does not exist”.
-> pixi/venv/renv might work an a personal computer but not on a cluster
→ Use Apptainer or other containerization for your environment on the cluster
(see HPC Documentation here: hpc:applications_and_libraries [eResearch Doc] )
-> See here for a tutorial: [tutorial] use conda in a container
Thanks @Yann.Sagon for pointing this out and your help! Very much appreciated!
I won’t delete the post (unless explicitly awsked by the admins), just in case it might help anyone else running into the same newbie problem.
Cheers,
Chris