Number of cores, threads, and python

HPC Technical
1

Hi all,

I’m trying to figure out what is the best setup for my script. It is written in python, using at some point multithreading through an external python package. According to the documentation of that package (cosmoHammer), there are 2 ways to use multithreading :

  • using OpenMP, using m threads set by environment variable OMP_NUM_THREADS
  • using python built-in multiprocessing module, using k threads
  • both, using n = k x m threads

What should I use in terms of SLURM options (-N, -n, -c) to really have control over which of the three solutions above ? Do I have to set manually OMP_NUM_THREADS ?

So far, what I have noticed if that if I submit my job using say -c 8 :

  • if I set export OMP_NUM_THREADS=8, there is no observed speedup
  • if I tell cosmoHammer to use 8 threads (in the python’s multiprocessing sense), I get an error OSError: [Errno 28] No space left on device, which I interpret as " you want a too high number of threads ", since I imagine it is like I’m asking for 8 x 8 threads (which is of course not what I want).

My apologies if this question is not supposed to be posted here, and thanks in advance for any help.

Aymeric

2

Dear Aymeric,

The option -n is for MPI (not openMP) stuff. So you should probably just ask for the number of cores needed with -c. It would be a good idea to do some benchmark by running your job with one core, two, four, eight and so on and measure the time. When the improvement is too low, stick with the last number of cores.

Yes

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

You can connect with ssh from the login node to the node where your job is running and see the cpu usage with htop. Check if the eight cores are used.

We figured out today that the space /dev/shm wasn’t cleaned properly after a job has finished and on some nodes this 100% full. Maybe this was the issue. It’s fixed.

3

Hi Yann,

Thank for your answers! It is clearer to me now.

Indeed this issue is fixed now, thanks.

Aymeric