Python virtual environment Anaconda3/2020.07

HPC Technical
1

Dear colleagues,

I’m (still) trying to use the SPMLE package in R built by Philipp Huntziker (GitHub - hunzikp/spmle: Binary Choice Models Using PMLE).

The package use a python virtual environment with scipy and numpy installed to speed up the calculation process. (R line of code: spmle::install_pydep()). Note that the aim is to run the following lines of code :

library(reticulate)
library(spmle)
reticulate::use_virtualenv(“r-reticulate”) # In Unix

  1. I created a python virtual environment following the “Custom python lib” tutorial in Applications and Libraries. First issue : when I try to activate my virtual environment using ~/name_venv/bin/activate, I receive a “Permission denied”.

  2. Anyways, without activating I managed to install scipy in my virtual environment, and then try to run my batch script. I receive the following error message :

hpc_error2
hpc_error2721×78 2.67 KB

Could someone explain to me how to create a virtual environment in Anaconda3/2020.07 ? Or how to link my python virtual environment to Anaconda3/2020.07 ?

Thank you for your help. Best,
Cécile.

2

In order to create a conda environment:

ml Anaconda3

conda create -n MyEnvName python=3.7 pip scipy numpy

Then usually to activate the environment you shall type conda activate MyEnvName but in Baobab for some reason you have to do source activate MyEnvName

You can find a lot more info here Managing environments — conda 4.10.1.post2+b6d32c8d7 documentation

ps in the conda create command I used you of course will have to personalize MyEnvName and python=3.7 to whatever you need/like and you can add or remove any package you like (and you can always install more later with conda install or pip installl)

3

Hi, as we already have NumPy and SciPy on the cluster (SciPy-bundle), you can probably use them instead of creating a virtual environment:

ml GCC/9.3.0  OpenMPI/4.0.3 R/4.0.0 rgeos/0.5-5-R-4.0.0 SciPy-bundle

Can you then try without creating the virtual environment: do not use reticulate::use_virtualenv(“r-reticulate”) nor spmle::install_pydep() and check if spml is working like that?

4

Hello,

Thanks for your answer.

@Yann.Sagon : spml is working perfectly without the python speed-up. :slight_smile: However, I still think it’d be best to use the python speed-up (also now it’s getting personal).

Following Maurice’s answer, I tried the following and it’s still not working:

ml Anaconda3/2020.7
conda create -n r-reticulate python=3.8.2 pip scipy numpy

Note: I also tried by giving it a random name, it changed nothing.

I can activate / deactivate my environment without a problem. Yet, when running my R script, I still have the same error “Error: could not find a Python environment for /opt/ebsofts/Anaconda3/2020.07/bin/python3”.

In case, my batch script looks like this :

#!/bin/bash
#SBACTH --job-name=testR
#SBACTH --ntasks-per-node=1
#SBACTH --cpus-per-task=1
#SBACTH --time=00:10:00
#SBACTH --partition=debug-cpu
#SBACTH --mail-type=ALL
module load GCC/9.3.0 OpenMPI/4.0.3 GEOS/3.8.1-Python3.8.2 R/4.0.0 Anaconda3/2020.07
INFILE=Rscript.R
OUTFILE=report-${SLURM_JOBID}.Rout
srun R CMD BATCH $INFILE $OUTFILE

Following Yann’s post, I activated the SciPy-bundle in my sbatch script, and still got the error for the python environment.

Thanks again for all your precious help. Best,
Cécile.

5

Hi again,

I fully agree!

I tried what I explained in my last post and it works.

Here is a working example I just uploaded.

You don’t need to create a virtual env nor to use *Conda.

Best

6

Hi again,

It works! Wonderful, thank you very much. All the best,

Cécile.