I’m trying to figure out what is the best setup for my script. It is written in python, using at some point multithreading through an external python package. According to the documentation of that package (cosmoHammer), there are 2 ways to use multithreading :
- using OpenMP, using m threads set by environment variable
- using python built-in
multiprocessingmodule, using k threads
- both, using n = k x m threads
What should I use in terms of SLURM options (-N, -n, -c) to really have control over which of the three solutions above ? Do I have to set manually
So far, what I have noticed if that if I submit my job using say -c 8 :
- if I set
export OMP_NUM_THREADS=8, there is no observed speedup
- if I tell cosmoHammer to use 8 threads (in the python’s
multiprocessingsense), I get an error
OSError: [Errno 28] No space left on device, which I interpret as " you want a too high number of threads ", since I imagine it is like I’m asking for 8 x 8 threads (which is of course not what I want).
My apologies if this question is not supposed to be posted here, and thanks in advance for any help.