TensorFlow/2.1.0 and scikit-learn/0.21.3 conflict

HPC Technical
1

Hello,

I am trying to use and load the modules scikit-learn/0.21.3-Python-3.7.4 and TensorFlow/2.1.0-Python-3.7.4 together in Baobab but I believe they run into some conflicts. To be more precise, I believe scikit-learn enters in conflict with CUDA/10.1.243 module, needed to load TensorFlow.

  • To load scikit-learn/0.21.3-Python-3.7.4 I need:
    GCC/8.3.0 OpenMPI/3.1.4
  • To load TensorFlow/2.1.0-Python-3.7.4 I need:
    GCC/8.3.0 CUDA/10.1.243 OpenMPI/3.1.4

If I load the modules like this:

module load GCC/8.3.0  CUDA/10.1.243  OpenMPI/3.1.4 TensorFlow/2.1.0-Python-3.7.4 scikit-learn/0.21.3-Python-3.7.4

I get an error:

Lmod has detected the following error:  These module(s) or extension(s) exist but cannot be loaded as requested: "scikit-learn/0.21.3-Python-3.7.4" Try: "module spider scikit-learn/0.21.3-Python-3.7.4" to see how to load the module(s).

If load the modules in a different order:

module load GCC/8.3.0 OpenMPI/3.1.4 scikit-learn/0.21.3-Python-3.7.4 CUDA/10.1.243 TensorFlow/2.1.0-Python-3.7.4

I get no error but then my scikit-learn module (import sklearn) is not recognise. I get the same result if I remove TensorFflow and I just keep CUDA/10.1.243 in the loading list.

If you run into this issue and you have found a workaround, please let me know. Any help in this direction would be very useful.

Thanks,
Denisa

edit YS: formatting

2

Hi Denisa,

the issue is that GCC/8.3.0 OpenMPI/3.1.4 and GCC/8.3.0 CUDA/10.1.243 OpenMPI/3.1.4 aren’t the same toolchain and you can’t load both at the same time.

I’ll build scikit-learn/0.21.3-Python-3.7.4 for fosscuda/2019b.

3

Hi, I see that this issue may still be open: is this an option for you to use this version instead:

[sagon@login2 ~] $ ml fosscuda/2020b TensorFlow/2.4.1 SciPy-bundle/2020.11
1 Like
4

Hello Yann,

Thanks for your last suggestion. I have finally managed to test it and it worked for me.
This is the modules that I have loaded to have both tensorflow and scikit-learn:

module load fosscuda/2020b TensorFlow/2.4.1 SciPy-bundle/2020.11 scikit-learn/0.23.2

Thanks for all your support!

1 Like