Can not create Customized Python Backends with Python 3.10 - 3.11 due to a Conda Pack issue

[luismarquezgft]

Description

Triton relies on Conda Pack to build the tar.gzip that contains the custom environment. Conda Pack has an issue with Python versions 3.10 and higher that impedes creating the pack and the custom backend. The result is that building customized Python environments with Python 3.10-3.11 impossible.

I would like to know if Triton offers an alternative way to package the content, as a workaround to this external issue detected a year ago, and still unfixed.

Evidence about this Conda Pack issue:

• December 2021: conda-pack fails when environment uses Python 3.10 conda/conda-pack#198
• November 2022 (opened by me): Conda pack does not work with Python 3.10 or 3.11 conda/conda-pack#244

Triton Information
What version of Triton are you using?
r22.09.

Are you using the Triton container or did you build it yourself?
Triton container.

To Reproduce
See the previously referenced conda pack issues. They are very easy to reproduce in a couple of minutes.

Describe the models (framework, inputs, outputs), ideally include the model configuration file (if using an ensemble include the model configuration file for that as well).
Not needed.

Expected behavior
Our business need is to create custom Python backends for Python 3.10 and 3.11. Due to the conda-pack issue, this is impossible. I ask for alternatives to overcome this long-existing problem (first reported in the conda-pack repo in December 2021).

[rmccorm4]

@luismarquezgft thanks for raising this issue. Just to double check, you require a python backend environment with Python >= 3.10? Or is it sufficient to build an environment with Python <= 3.9 in the meantime until conda pack fixes this issue?

CC @Tabrizian for any ideas on alternatives to package a python execution environment.

[rmccorm4]

@luismarquezgft Also after a quick glance at some linked issues, I see conda-pack==0.6.0 listed in your issue. But it looks like a supposed fix for this on the conda-pack side was released in v0.7.0: https://github.com/conda/conda-pack/releases/tag/0.7.0.

Support Python 3.10 in find_site_packages by @jtilly in conda/conda-pack#199

Can you verify if you still experience this issue with the latest conda-pack version?

[mahi80]

Still facing issues @rmccorm4 , below output on , tried all of the above still facing issues in 3.10.8 , any advice even after removing packages via pip uninstall from environment still showing same weird error

Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal

(310) onsumaye@onsumaye-Inspiro$ conda pack -n 310 -o out_name.tar.gz
Collecting packages...
CondaPackError:
Files managed by conda were found to have been deleted/overwritten in the
following packages:

• pyopenssl 22.0.0:
  lib/python3.1/site-packages/OpenSSL/SSL.py
  lib/python3.1/site-packages/OpenSSL/init.py
  lib/python3.1/site-packages/OpenSSL/_util.py
  • 27 others
• charset-normalizer 2.0.4:
  bin/normalizer
  lib/python3.1/site-packages/charset_normalizer-2.0.4.dist-info/INSTALLER
  lib/python3.1/site-packages/charset_normalizer-2.0.4.dist-info/LICENSE
  • 33 others
• flit-core 3.6.0:
  lib/python3.1/site-packages/flit_core-3.6.0.dist-info/INSTALLER
  lib/python3.1/site-packages/flit_core-3.6.0.dist-info/METADATA
  lib/python3.1/site-packages/flit_core-3.6.0.dist-info/RECORD
  • 159 others
• conda-pack 0.6.0:
  lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/INSTALLER
  lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/LICENSE.txt
  lib/python3.1/site-packages/conda_pack-0.6.0.dist-info/METADATA
  • 19 others
• wheel 0.37.1:
  bin/wheel
  lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/PKG-INFO
  lib/python3.1/site-packages/wheel-0.37.1-py3.9.egg-info/SOURCES.txt
  • 44 others
• pycparser 2.21:
  lib/python3.1/site-packages/pycparser-2.21.dist-info/INSTALLER
  lib/python3.1/site-packages/pycparser-2.21.dist-info/LICENSE
  lib/python3.1/site-packages/pycparser-2.21.dist-info/METADATA
  • 49 others
• six 1.16.0:
  lib/python3.1/site-packages/six-1.16.0.dist-info/INSTALLER
  lib/python3.1/site-packages/six-1.16.0.dist-info/LICENSE
  lib/python3.1/site-packages/six-1.16.0.dist-info/METADATA
  • 15 others

This is usually due to pip uninstalling or clobbering conda managed files,
resulting in an inconsistent environment. Please check your environment for
conda/pip conflicts using conda list, and fix the environment by ensuring
only one version of each package is installed (conda preferred).

[rmccorm4]

I'm not sure we have any alternatives to offer if this isn't fixed upstream in conda-pack at this time. CC @Tabrizian

[luismarquezgft]

Sorry for my late response. It should be fixed in Conda-Pack version 0.7.0, but the issue is still there.

[Tabrizian]

Looks like conda is installing python3.1 instead of Python 3.10/3.11. Alternatively, you can install the dependencies using pip3 and compile your custom python backend stub using the instructions provided here:

https://github.com/triton-inference-server/python_backend#building-custom-python-backend-stub

In this alternative approach, you won't be able to convert your environment into a tar file and need to make sure that the dependencies are installed in the environment.

[jtilly]

Looks like conda is installing python3.1 instead of Python 3.10/3.11.

No, you're being tricked by some of the symlinks that were introduced for 3.10 (xref conda-forge/python-feedstock#611).

I commented on the original conda-pack issue at conda/conda-pack#244 (comment).

[luismarquezgft]

It works!!

The subtle difference is using conda-pack, I was using conda pack.
I'm going to close this issue, and the one I opened in the conda-pack repo.

Thank you very much for your help.

This is the code (adapted from your example @jtilly):

conda create -y -f -n python310forge -c conda-forge python=3.10 conda-pack
conda activate python310forge
pip install -r stripped_requirements.txt

export PYTHONNOUSERSITE=True
git clone https://github.com/triton-inference-server/python_backend -b r22.11
cd python_backend && mkdir build && cd build
cmake -DTRITON_ENABLE_GPU=OFF -DTRITON_BACKEND_REPO_TAG=r22.11 -DTRITON_COMMON_REPO_TAG=r22.11 -DTRITON_CORE_REPO_TAG=r22.11 -DCMAKE_INSTALL_PREFIX:PATH=`pwd`/install ..
make triton-python-backend-stub

conda-pack