Scratch quota and number of files?

HPC Technical
1

Hello,
I have a question about the quota on scratch on baobab.
When I use the quota command, I get:

home dir: /home/users/d/dumoulil
scratch dir: /srv/beegfs/scratch/users/d/dumoulil

        user/group                 ||           size          ||    chunk files
storage     |   name        |  id  ||    used    |    hard    ||  used   |  hard
----------------------------|------||------------|------------||---------|---------
home        |       dumoulil|372536||   68.83 GiB| 1024.00 GiB||    58086|unlimited
scratch     |       dumoulil|372536||    2.99 TiB|   unlimited||   323989| 10000000

But when I count the number of files on the scratch I get a different number, less than 100k. This “less than 100k” makes sense because I save a bit less than 150 files per simulation for (now) 610 simulations then less than 100k files.

I don’t understand the factor 3 between the quota chunk files and the “real number” of files. Probably because I don’t get what a “chunk file” is.

Thank you for your help.

2

Dear @Ludovic.Dumoulin

the quota on the number of files is not exactly about counting files. In fact it is about counting chunks of files. If you have a “small” file, you’ll have one chunk = one file. If your file is bigger, it will be stripped and distributed on multiple storage server, in this case, you’ll have many chunks per file.

This is the current configuration for the home directory:

(baobab)-[sagon@admin1 ~]$ beegfs-ctl --cfgFile=/etc/beegfs/home.d/beegfs-client.conf --getentryinfo /home
Entry type: directory
EntryID: root
Metadata node: home1.baobab [ID: 1]
Stripe pattern details:
+ Type: RAID0
+ Chunksize: 512K
+ Number of storage targets: desired: 4
+ Storage Pool: 1 (Default)

As you can see, for the home storage, the chunk size is 512kb and we try to store the chunks on 4 different storage targets.

https://doc.beegfs.io/latest/advanced_topics/striping.html

Best

1 Like
3

Thank you for the explanation ! :slight_smile: