Error from the task prolog script

HPC Support HPC issues
1

Primary informations

Username: serpolla
Cluster: Baobab

Description

It seems external Python setups interfere with Slurm task prolog script.
I usually setup my shell with CERN software and launch several jobs at once, but now I have to setup the environment from the computing node to avoid errors.

I do not know if this is intended and the new standard should be setting the environment on the computing node.

Steps to Reproduce

(baobab)-[serpolla@login1 ~]$ (source /cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0/x86_64-el9/setup.sh && source /cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/setup.sh && srun -p shared-cpu -t 20 --pty bash -i) 
srun: job 13313802 queued and waiting for resources 
srun: job 13313802 has been allocated resources 
[2026-10-01T11:31:46.669] error: TaskProlog failed status=1 
srun: error: cpu350: task 0: Exited with exit code 1 

(baobab)-[serpolla@login1 ~]$ (srun -p shared-cpu -t 20 --pty bash -ic '(source /cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0/x86_64-el9/setup.sh && source /cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/setup.sh); echo $?') 
srun: job 13313834 queued and waiting for resources 
srun: job 13313834 has been allocated resources 
0

The problem seems to be in the Python setup:

(baobab)-[serpolla@login1 ~]$ (source /cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0/x86_64-el9/setup.sh && source /cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/setup.sh; env -u PYTHONHOME -u PYTHONPATH srun -p shared-cpu -t 5 hostname)
srun: job 13314605 queued and waiting for resources
srun: job 13314605 has been allocated resources
cpu335.baobab

(baobab)-[serpolla@login1 ~]$ (source /cvmfs/sft.cern.ch/lcg/releases/gcc/11.3.0/x86_64-el9/setup.sh && source /cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/setup.sh; /usr/bin/python3 -c 'print("ok")')
Python path configuration:
  PYTHONHOME = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt'
  PYTHONPATH = '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages/itk:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages/itk:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/python:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib:/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages'
  program name = '/usr/bin/python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt'
  sys.base_exec_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt'
  sys.platlibdir = 'lib64'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt'
  sys.exec_prefix = '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt'
  sys.path = [
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages/itk',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/python',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages/itk',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/python',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib',
    '/cvmfs/sft.cern.ch/lcg/views/LCG_107/x86_64-el9-gcc11-opt/lib/python3.11/site-packages',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt/lib64/python39.zip',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt/lib64/python3.9',
    '/cvmfs/sft.cern.ch/lcg/releases/Python/3.11.9-2924c/x86_64-el9-gcc11-opt/lib64/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x0000151661e25740 (most recent call first):
<no Python frame>
2

Dear @Andrea.Serpolla

After testing, I confirm there is an issue with the taskprolog.

The issue is the following: this script is executed with the same user environment. As you set your own python etc using your scripts, the system python that we use in taskprolog isn’t working anymore. The fact that taskprolog needs python is recent, this is why it was working in the past. We’ll need to purge the user defined variables in this script.

We’ll try to fix is by tomorrow.

my 2 cents: instead of using srun --pyt bash -i you should use salloc which is cleaner (but won’t solve your issue).

Best regards

3

Dear @Andrea.Serpolla it is solved, apologies for the inconvenience.

4

Dear Yann,

Thanks you all for the fix.

Best.