I have implemented a very parallelizable algorithm (in Python). It is both data parallel and multithreaded, and very well suited to run on a cluster.
I could write some ad-hoc code to divide up tasks to be run in parallel on a Slurm job-array. However, I would like other people to be able to easily run my code on their cluster. Therefore I would like to write code to work for more platform-independent HPC clusters.
Do you have any advice on (Python) libraries for managing job queues in situations like this?
I also send a request on the SciCoS website, but I did not get a reply.