How to handle life cycle of task package with direct-reference dependencies?

[tcompa]

Version 0.3.0 of the APX tasks package had the following pyproject.tom: https://github.com/Apricot-Therapeutics/APx_fractal_task_collection/blob/5f5c087696b60189f09700f8fad8bb526b1e7fb4/pyproject.toml.
This included a GitHub dependency, that is, the dev branch of the basicpy github repository:

"basicpy@git+https://github.com/peng-lab/BaSiCPy.git@dev"

In a fresh venv, if one installs (today) from that string and then runs pip freeze here is what you get:

$ pip install basicpy@git+https://github.com/peng-lab/BaSiCPy.git@dev
Collecting basicpy@ git+https://github.com/peng-lab/BaSiCPy.git@dev
  Cloning https://github.com/peng-lab/BaSiCPy.git (to revision dev) to ./pip-install-ze5cijwl/basicpy_edbedc8b03024ae98fddb26e5393b96d
  Running command git clone --filter=blob:none --quiet https://github.com/peng-lab/BaSiCPy.git /tmp/pip-install-ze5cijwl/basicpy_edbedc8b03024ae98fddb26e5393b96d
  Running command git checkout -b dev --track origin/dev
  Switched to a new branch 'dev'
  Branch 'dev' set up to track remote branch 'dev' from 'origin'.
  Resolved https://github.com/peng-lab/BaSiCPy.git to commit 166bf6190c1827b5a5ece4a5542433c96a2bc997
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
[....]

$ pip freeze | grep BaSiCPy
BaSiCPy @ git+https://github.com/peng-lab/BaSiCPy.git@166bf6190c1827b5a5ece4a5542433c96a2bc997

$ pip show BaSiCPy
Name: BaSiCPy
Version: 1.1.0

Trying to reinstall the same package from the two equivalent sources leads to an error:

$ pip install "BaSiCPy @ git+https://github.com/peng-lab/BaSiCPy.git@166bf6190c1827b5a5ece4a5542433c96a2bc997" "BaSiCPy @ git+https://github.com/peng-lab/BaSiCPy.git@dev"

[...]

ERROR: Cannot install basicpy 1.1.0 (from /somewhere/.cache/pip/wheels/5a/58/0e/d615171f09e95744057baec82cd365fe13af1f89bf9a05fb87/BaSiCPy-1.1.0-py3-none-any.whl) and basicpy 1.1.0 (from git+https://github.com/peng-lab/BaSiCPy.git@dev) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested basicpy 1.1.0 (from /somewhere/.cache/pip/wheels/5a/58/0e/d615171f09e95744057baec82cd365fe13af1f89bf9a05fb87/BaSiCPy-1.1.0-py3-none-any.whl)
    The user requested basicpy 1.1.0 (from git+https://github.com/peng-lab/BaSiCPy.git@dev)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

---

The situation above leads to an error when deactivating and reactivating that task environment.

The pip freeze output is like

...
apx-fractal-task-collection @ file:///somewhere/on/disk/apx_fractal_task_collection-0.3.0-py3-none-any.whl#sha256=71d194310f82eeb70322fa1110046e3091044aefd7dd5ea0eac2d49d5529e56f
...
BaSiCPy @ git+https://github.com/peng-lab/BaSiCPy.git@166bf6190c1827b5a5ece4a5542433c96a2bc997

When deactivating and reactivating this package, we hit the following pip install error:

The conflict is caused by:
    The user requested basicpy 1.1.0 (from git+https://github.com/peng-lab/BaSiCPy.git@166bf6190c1827b5a5ece4a5542433c96a2bc997)
    apx-fractal-task-collection 0.3.0 depends on basicpy 1.1.0 (from git+https://github.com/peng-lab/BaSiCPy.git@dev)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

[tcompa]

Note that PyPI currently forbids this kind of dependencies (see our past experience with the parsl fork, or see e.g. pypi/warehouse#7136), therefore I think this issue only affects locally-built wheels.

[tcompa]

Another (side) remark: this would obviously not be an issue if Python had standard lock files.

[tcompa]

Easily actionable (partial) fix: if the pip freeze information contain links to github, we prevent task-group deactivation - since we don't currently a way to reactivate the package.

[jluethi]

Ugh, all the package management edge-cases... Yeah, I'd be fine with this just not being supported for the time being.
If this occurs more often, we can consider having the case of "we deactivate the task package, but in such cases wouldn't remove the env"
And it's an admin decision to manually remove some of those envs at some point

[tcompa]

Yeah, I'd be fine with this just not being supported for the time being.

This is in-place as of fractal-server 2.10.2.