[VS-Code] Reticulate is forcefully overriding the requested virtual environment and picking the default python value. Happens with both use_python and use_virtualenv. Due to radian package VS Code setting.

[bryanwhiting]

Issue Description

reticulate is forcefully loading the same python package despite setting RETICULATE_PYTHON or PYTHONPATH.

All examples below are a new R session. You can see that no matter the settings I provide, it's forcing the wrong version.

Default Behavior

Starting a new session, I get:

r$> library("reticulate")
r$> Sys.getenv("RETICULATE_PYTHON")
[1] "/Library/Developer/CommandLineTools/usr/bin/python3"

r$> py_config()
python:         /Library/Developer/CommandLineTools/usr/bin/python3
libpython:      /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Python3
pythonhome:     /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9:/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9
version:        3.9.6 (default, Oct 18 2022, 12:41:40)  [Clang 14.0.0 (clang-1400.0.29.202)]
numpy:          /Users/b/Library/Python/3.9/lib/python/site-packages/numpy
numpy_version:  1.24.1

NOTE: Python version was forced by the current process

Trying use_virualenv()

Now I create a new R session and create a new virtual env, but when I use the use_virtualenv() its overridden by the default version:

r$> library("reticulate")
r$> virtualenv_create("r-reticulate3")
r$> use_virtualenv("r-reticulate3", required = TRUE)
Warning message:
The request to `use_python("/Users/b/.virtualenvs/r-reticulate3/bin/python")` will be ignored because the environment variable RETICULATE_PYTHON is set to "/Library/Developer/CommandLineTools/usr/bin/python3"

r$> py_config()
python:         /Library/Developer/CommandLineTools/usr/bin/python3
libpython:      /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Python3
pythonhome:     /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9:/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9
version:        3.9.6 (default, Oct 18 2022, 12:41:40)  [Clang 14.0.0 (clang-1400.0.29.202)]
numpy:          /Users/b/Library/Python/3.9/lib/python/site-packages/numpy
numpy_version:  1.24.1

NOTE: Python version was forced by the current process

r$> py_run_string("import sys; print(sys.executable)")
/Library/Developer/CommandLineTools/usr/bin/python3

set RETICULATE_PYTHON

If I try to set the RETICULATE_PYTHON, it doesn't do anything:

R version 4.2.2 (2022-10-31) -- "Innocent and Trusting"
Platform: aarch64-apple-darwin20 (64-bit)

r$> library("reticulate")

r$> Sys.setenv(RETICULATE_PYTHON="/Users/b/.virtualenvs/r-reticulate3/bin/python")

r$> # virtualenv_create("r-reticulate3")
    use_virtualenv("r-reticulate3", required = TRUE)

r$> py_config()
python:         /Library/Developer/CommandLineTools/usr/bin/python3
libpython:      /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Python3
pythonhome:     /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9:/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9
version:        3.9.6 (default, Oct 18 2022, 12:41:40)  [Clang 14.0.0 (clang-1400.0.29.202)]
numpy:          /Users/b/Library/Python/3.9/lib/python/site-packages/numpy
numpy_version:  1.24.1

NOTE: Python version was forced by the current process

r$> py_run_string("import sys; print(sys.executable)")
/Library/Developer/CommandLineTools/usr/bin/python3

Using use_python

r$> library("reticulate")
^[[200~Sys.setenv(RETICULATE_PYTHON="/Users/b/.virtualenvs/r-reticulate3/bin/python")^[[201~

r$> Sys.setenv(RETICULATE_PYTHON="/Users/b/.virtualenvs/r-reticulate3/bin/python")

r$> use_python("/Users/b/.virtualenvs/r-reticulate3/bin/python", required = TRUE)

r$> py_config()
python:         /Library/Developer/CommandLineTools/usr/bin/python3
libpython:      /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Python3
pythonhome:     /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9:/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9
version:        3.9.6 (default, Oct 18 2022, 12:41:40)  [Clang 14.0.0 (clang-1400.0.29.202)]
numpy:          /Users/b/Library/Python/3.9/lib/python/site-packages/numpy
numpy_version:  1.24.1

NOTE: Python version was forced by the current process

Session Info

Clean R sessionInfo():

$ R
r$> library(reticulate)
r$> sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.5.1

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.26

loaded via a namespace (and not attached):
[1] compiler_4.2.2  Matrix_1.5-1    Rcpp_1.0.9      grid_4.2.2     
[5] jsonlite_1.8.4  png_0.1-8       lattice_0.20-45

[t-kalinowski]

Are you observing this in the RStudio IDE, in the terminal, or both?

[bryanwhiting]

Terminal (VS Code), i haven't tried RStudio

[t-kalinowski]

Can you please try in the builtin Terminal.app that comes bundled with macOS?
It's possible that VScode is attempting to autoactivate a python environment.

[bryanwhiting]

Seems like that's the error, as when i use raw terminal it's fine:

[bryanwhiting]

I tried changing the VS Code python interpreter but didn't change anything.

[t-kalinowski]

NOTE: Python version was forced by the current process

This means that Python symbols were already loaded into the current process when reticulate tried to initialize it.

What VSCode extensions do you have enabled? Does disabling them fix the issue?

[bryanwhiting]

Thanks for your epic help.

Problem was that I was using R-extension with Radian, and I had radian path defined:

// "r.rterm.mac": "/Users/b/Library/Python/3.9/bin/radian", <- originally installed, didn't work
    "r.rterm.mac": "/Users/b/.virtualenvs/r-reticulate/bin/radian", <- now selected, works

I updated the path to the virtual env and this works. (Note, just setting VS Code's python interpreter doesn't work, as this Radian setting overrrides it.)

When I have "r.rterm.mac": "/Users/b/.virtualenvs/r-reticulate/bin/radian" set, I don't have to even declare the python path:

    library("reticulate") # Load the library

r$> py_config()
python:         /Users/b/.virtualenvs/r-reticulate/bin/python
libpython:      /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/Python3
pythonhome:     /Users/b/.virtualenvs/r-reticulate:/Users/b/.virtualenvs/r-reticulate
version:        3.9.6 (default, Oct 18 2022, 12:41:40)  [Clang 14.0.0 (clang-1400.0.29.202)]
numpy:          /Users/b/.virtualenvs/r-reticulate/lib/python3.9/site-packages/numpy
numpy_version:  1.24.1

NOTE: Python version was forced by the current process

r$> py_run_string("import nevergrad")

And now my code works! Thanks @t-kalinowski !!

[chenweifelix]

This discussion is super helpful!