OSError: [Errno 122] Disk quota exceeded

Hi,

When submitting sbatch jobs on baobab and saving files to scratch I get “disk quota exceeded”. Below you see the error and the batch script I am using.

Hope you can help.

Best,
Malte

I started receiving the same message after this.
I don’t know if the two things are related.

Hi @Malte.Algren

Indeed There is inode quota on scratch directory:

https://doc.eresearch.unige.ch/hpc/storage_on_hpc?s[]=quota#quota1

However, your quota doesn’t seem to have been reached.

(baobab)-[root@login2 ~]$ beegfs-ctl --getquota --uid algren --mount=/srv/beegfs/scratch --connAuthFile=/etc/beegfs/connauthfile

Quota information for storage pool Default (ID: 1):

      user/group     ||           size          ||    chunk files    
     name     |  id  ||    used    |    hard    ||  used   |  hard   
--------------|------||------------|------------||---------|---------
        algren|404573||    7.77 TiB|   unlimited||  2529699| 10000000

Strange behavior, let me find the root cause, I’ll keep you posted.

Dear @Malte.Algren @Andrea.Serpolla

The DPNC group quota was not updated after the extra was added.

I took the liberty of testing the file creations in your scratch directory to make sure everything was working properly:

(baobab)-[root@admin1 pillar] (master)$ su - algren
Last login: Mon Oct 30 10:57:12 CET 2023 on pts/7
(baobab)-[algren@admin1 ~]$ cd scratch/
(baobab)-[algren@admin1 scratch]$ touch toto

Let me konw if something is not working.

We apologize for the inconvenience.

Hi, I got the same message on yggdrasil. But it’s annoying that the error occurs sometimes and then it works again, back and forth. I submitted a number of jobs, about half of them failed without warning.

Dear @Zepei.Xing,

Here your quota:

(yggdrasil)-[root@admin1 ~]$ beegfs-ctl --getquota --uid xing --mount=/srv/beegfs/scratch --connAuthFile=/etc/beegfs/connauthfile 
      user/group     ||           size          ||    chunk files    
     name     |  id  ||    used    |    hard    ||  used   |  hard   
--------------|------||------------|------------||---------|---------
          xing|385443||    2.37 TiB|   unlimited||  3059934| 10000000

(yggdrasil)-[root@admin1 ~]$ beegfs-ctl --getquota --uid xing --mount=/home --connAuthFile=/etc/beegfs/connauthfile 

      user/group     ||           size          ||    chunk files    
     name     |  id  ||    used    |    hard    ||  used   |  hard   
--------------|------||------------|------------||---------|---------
          xing|385443|| 1023.92 GiB| 1024.00 GiB||   978547|unlimited

As you can see, you’re about to reach your home directory storage quota. When you’re often creating and deleting files, this is the reason why things sometimes work and sometimes they don’t. It’s as if you were playing a bit of a “quota limit game”.