Python pipeline slowdown on Baobab since update

1

Hi,

since last update on baobab, there’s been huge slowdowns in my python pipeline.

The most reproductible example I can come up with is the loading of big data files (.npy) using np.load. It used to take few seconds on any nodes (as on my local computer), but it now takes several minutes to open one.

you can find an example file here :
/home/share/pouget_lab/wide_field_imaging/CSK-im-002/downsampled_ims-002.npy (15.6 GB)

using numpy.load(file,allow_pickle=True) it takes 6.5s on my compute to load and used to take way less time on a cluster node. Now it takes over 3 minutes on a cluster node - at best.

A lot of other slowdowns have been noticed by me and my collegues.

Do you know why this is happening ? Can this be fixed ?

Thank you very much.

Félix Hubert

2 Likes
2

Hi,

unfortunately you are perfectly right:( This is something we need to add to our new monitoring service… The storage is using ethernet network instead of infiniband. We are checking.

3

It is now corrected! Thanks for the notification.

1 Like