Job priority explanation

HPC General
1

We have received an email from a user asking questions about job priority.

Here is the slightly edited request we received:

I am wondering why do my jobs always get the same low priority, For one I requested 5GM mem for the other 15 but for both show a PRIORITY 3804.

Like, what is e.g. difference to job like XXX, whose use is running order of 1000 in parallel while my few jobs, to test something, are stuck since 2 days…

The total number of running jobs per user is not counted at all in priority calculation, is it?

The priority we setup is made in general by four criteria, and each criteria has a different weight.

  • partition : 15000
  • age: 300
  • jobsize: 1000
  • fairshare: 30000

The criteria “partition” and “fairshare” have the biggest impact on the total priority calculation.

The partition criteria is always the same for a given user on a given partition. If you submit for example a job to the partition mono-shared-EL7 you’ll get a priority of 3750 for this criteria. But this criteria may have a multiplier. For example, if a user submit a job on a private partition, the multiplication factor is 4 and he will have a priority of 15000 for the criteria “partition”. The reason of this is because the private nodes belongs a private partition and to the mono-shared-El7 partition as well. This ensure that their owner have a faster access to their private node.

The age criteria starts to increase as soon as the job is submitted. This is used to avoid that a job with low priority for another reason stay in the queue for ages.

The jobsize criteria is here to favor big jobs, but the weigh is very low unless you submit a very big job.

The fairshare criteria is based on the past usage of the user. If you use a lot the cluster, your fairshare criteria will be lower. Each 2 weeks, half of your past usage is forgotten.

You can see how the priority of a job is calculated with the sprio command.

[root@master ~]# sprio  -j xxx
          JOBID PARTITION   PRIORITY       SITE        AGE  FAIRSHARE    JOBSIZE  PARTITION        QOS
       xxx      mono-shar       3810          0          6         53          2       3750          0

To answer the questions on this email:

  1. asking more ore less memory doesn’t change the priority. But asking more resources like memory may prevent slurm to find a free resources, but this is not related to priority. This led to the fact that it’s better to ask as few resources as possible to be scheduled quickly.

  2. if you are requesting for example 12h00 of wall-time and 16 cores, and another user is asking for a wall-time of 1 minute and one cpu, he may go in front of your job thanks to the backfill (to be explained in another post another day) mechanism even if the job has a lower priority.

  3. the total number of running jobs of a given user doesn’t change the priority, but his fairshare will lower with the time.

1 Like