Get dynamically the number of cores available

HPC Technical
1

Hello,

I red the following trick in the doc : https://baobab.unige.ch/enduser/src/enduser/submit.html?highlight=sacct#multithreaded-jobs

I would like to pass to my python script the number of cores that must be used for multithreading. How can I proceed if I use the script above, which is designed to automatically assign all cores on a given node to the job being launched ?

As an example, what I currently do is that I set manually the --cpu_per_task option to 8, 12, 16… and then pass to my python script the environment variable $SLURM_CPUS_PER_TASK, which is automatically set to the --cpu_per_task value.
But in the case where this value is set by the --cpu_bind=mask_cpu:0xffffffff option, how can I know the effective number of cores being used, at execution time ?

I’m thinking of using the multiprocessing.cpu_count() method of the multiprocessing module of python, but I red somewhere that this may not be the right choice on a cluster, because it sometimes does not reflect the ‘real’ number of cores available.

How would you do, please ? Thanks in advance

Aymeric

2

Dear @aymeric.galan,

If you want to use the --cpu_bind=mask_cpu:0xffffffff trick, it means you want to use the whole compute node, right? In this case the multiprocessing.cpu_count() will return the correct number of cpu because it’s trivial for it to count them.

The issue with this package is when you set a cpu_per_task smaller than a full node and this package thinks he can use the whole node. This is because Python doesn’t know about cgroups. More on this https://bugs.python.org/issue36054.

Best

3

Understood, thanks for the answer.