[python-package] support Python 3.13

[jameslamb]

Add support for Python 3.13, now that it's available: https://docs.python.org/3/whatsnew/3.13.html

This also proposes a new standard process for Python version upgrades in this project. Proposing that when a new Python version is generally available, we do the following:

• bump every Python version in all CI up by 1 major version (e.g., all 3.10 -> 3.11)
• add any end-of-life versions to the test-old-versions CI job (as I've done here with Python 3.8)
  • based on https://devguide.python.org/versions/

This makes the updates easy to make and review, and reduces the risk of issues like #5969 and #6680 leading to maintenance work.

[jameslamb]

We'll need to wait for the rest of LightGBM's dependencies to get Python 3.13 packages on conda-forge.

LibMambaUnsatisfiableError: Encountered problems while solving:
  - package mypy-1.11.1-py310h5b4e0ec_0 requires python >=3.10,<3.11.0a0, but none of the providers can be installed

Could not solve for environment specs
The following packages are incompatible
├─ mypy >=1.11.1  is installable with the potential options
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python_abi 3.12.* *_cp312, which requires
│  │     └─ python 3.12.* *_cpython, which can be installed;
│  ├─ mypy [1.11.1|1.11.2] would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  ├─ mypy [1.11.1|1.11.2] would require
│  │  └─ pypy3.9 >=7.3.15 , which requires
│  │     └─ python 3.9.* *_73_pypy, which can be installed;
│  ├─ mypy [1.11.1|1.11.2|1.12.0] would require
│  │  └─ python >=3.9,<3.10.0a0 , which can be installed;
│  └─ mypy 1.12.0 would require
│     └─ python_abi 3.13.* *_cp313 with the potential options
│        ├─ python_abi 3.13 would require
│        │  └─ python 3.13.* *_cpython, which can be installed;
│        └─ python_abi 3.13 would require
│           └─ python 3.13.* *_cp313, which can be installed;
├─ pyarrow-core >=17.0  is installable with the potential options
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ __cuda >=11.8 , which is missing on the system;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.10,<3.11.0a0 , which can be installed;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.11,<3.12.0a0 , which can be installed;
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python_abi 3.12.* *_cp312, which can be installed (as previously explained);
│  ├─ pyarrow-core 17.0.0 would require
│  │  └─ python >=3.8,<3.9.0a0 , which can be installed;
│  └─ pyarrow-core 17.0.0 would require
│     └─ python >=3.9,<3.10.0a0 , which can be installed;
└─ python 3.13 *_cp* is not installable because it conflicts with any installable versions previously reported.

(build link)

But hey, at least lightgbm itself already has Python 3.13 packages on conda-forge! (conda-forge/lightgbm-feedstock#63). That's thanks to lightgbm having such a small set of required dependencies... a good reminder that it's worthwhile to keep it that way 😁

We can check the state of LightGBM's dependencies here: https://conda-forge.org/status/migration/?name=python313

[jameslamb]

We can check the state of LightGBM's dependencies here: https://conda-forge.org/status/migration/?name=python313

pyarrow now has Python 3.13 packages (conda-forge/pyarrow-feedstock#139). I just merged latest master into this... let's see if all of LightGBM's recursive dependencies are now updated.

[jameslamb]

The conda-forge Python 3.13 migration is making progress, but looks like not all of lightgbm's dependencies have made it across yet. Looks like the next one we need is cffi.

PR to subscribe to: conda-forge/cffi-feedstock#53

[jameslamb]

good news: conda-forge migrations we needed for Python 3.13 appear to be done! (successful macOS Python 3.13 build)

bad news: seems like conda isn't able to find a solution for our CI environment on x86_64 Python 3.9 (failed macOS x86_64 Python 3.9 build, failed Linux x86_64 Python 3.9 job

Not sure about the root cause yet, will try to investigate.

[jameslamb]

Found that changes on this line were the reason for the failures I saw in #6668 (comment).

python=3.9=*_cp* and python=3.9[build=*_cp*] are handled differently by conda. I reported that at conda/conda-libmamba-solver#604, but we should just continue with the [build form here as it does work as expected.

[StrikerRUS]

I'm nervous about this particular change. Could you please trigger RTD build for this branch to check that everything is OK?

[jameslamb]

Sure! Here's a build: https://readthedocs.org/projects/lightgbm/builds/26929088/

(will check back in a bit)

[jameslamb]

The build failed with what looks like an internal error on readthedocs. Triggered another one: https://readthedocs.org/projects/lightgbm/builds/26929224/

[jameslamb]

The build is failing repeatedly without any logs showing an issue. I tried with manually-triggered builds and builds triggered by pushing a commit here.

I've reported this to RTD: readthedocs/readthedocs.org#11949