build against NPY2

[njzjz]

Summary

Major changes:

• fix 1: Build against NPY2 instead of NPY1. This will support both NPY1 and NPY2. See https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice.
• fix 2: Replace np.int_t (missing in NPY2) with np.int64_t and np.float_ with np.float64.
• fix 3: Replace long with np.int64_t, as "The default integer type on Windows is now int64 rather than
  int32, matching the behavior on other platforms" per the change log of NPY2.

Todos

If this is work in progress, what else needs to be done?

• feature 2: ...
• fix 2:

Checklist

• Google format doc strings added. Check with ruff.
• Type annotations included. Check with mypy.
• Tests added for new features/fixes.
• If applicable, new classes/functions/modules have duecredit @due.dcite decorators to reference relevant papers by DOI (example)

Tip: Install pre-commit hooks to auto-check types and linting before every commit:

pip install -U pre-commit
pre-commit install

[njzjz]

Unluckily, there are still several errors. Some are related to the upstream chgnet and scipy and cannot be fixed here.

[janosh]

@njzjz sorry for the lack of reply here! looks like this is almost finished. thanks for the effort! feel free to skip the failing chgnet tests for now. re scipy, we'll have to pin to <1.14 as the newest release dropped Python 3.9 support. but 1.13.0 already supports numpy v2

[DanielYang59]

I really appreciate the pioneering work @njzjz! Meanwhile in case anyone missed this, ruff now has a dedicated rule NPY201 to handle API migration automatically :)

ruff check path/to/code/ --select NPY201

[njzjz]

For cross reference, I've submitted the SciPy issue to scipy/scipy#21052

[njzjz]

I do not know how to fix the rest of the errors and need help.

[janosh]

@njzjz feel free to skip the two failing chgnet tests. we'll migrate it to numpy v2 once pymatgen has. for test_delta_func, you can conditionally skip it like this:

@pytest.mark.skipif(sys.platform == "win32" and np.__version__ > "2.0.0", reason="Fails on Windows with numpy > 2.0.0, awaiting https://github.com/scipy/scipy/issues/21052 resolution")
def test_delta_func():
    x = np.array([0, 1, 2, 3, 4, 5])

not sure about TestWavecar.test_get_parchg. that might need a closer look. maybe @esoteric-ephemera has some ideas on what's going wrong?

[janosh]

thanks a lot @njzjz for this work and esp. for reporting multiple upstream issues! 🥇 and thanks @DanielYang59 for supporting! good team effort to get the many failing tests here sorted out.

[DanielYang59]

The failing tests for tests/io/abinit/test_abiobjects.py looks like another randomly failing test. I cannot recreate it on my Windows PC.

[DanielYang59]

@janosh This doesn't look like something we should keep inside the code (it's pretty easy to find)? What do you think?

[janosh]

i'm generally in favor of linking relevant docs. even if easy to find, people might not think to look for them. but no strong opinion in this case

[DanielYang59]

This one looks more like a "disposable" migration guide which we would only do once :)

[janosh]

The failing tests for tests/io/abinit/test_abiobjects.py looks like another randomly failing test. I cannot recreate it on my Windows PC.

probably something to report to https://github.com/Unidata/netcdf4-python, even if just to say the error message is self-contradictory...

E OSError: Error loading charset-233f44715e5aacaf3a688c2faff5ddf7.dll; The operation completed successfully.

[DanielYang59]

The failing tests for tests/io/abinit/test_abiobjects.py looks like another randomly failing test. I cannot recreate it on my Windows PC.

probably something to report to https://github.com/Unidata/netcdf4-python, even if just to say the error message is self-contradictory...

Agreed, just for my own record, it failed in another PR randomly https://github.com/materialsproject/pymatgen/actions/runs/10268736299/job/28412461926?pr=3892

[njzjz]

E OSError: Error loading charset-233f44715e5aacaf3a688c2faff5ddf7.dll; The operation completed successfully.

This part of the code is generated by delvewheel here. I think I can report there.

According to the Windows documentation, this error message means no error. So it may be incorrect to throw an error (while it is also strange that LoadLibraryExW returns nothing but gives ERROR_SUCCESS).

ERROR_SUCCESS
0 (0x0)
The operation completed successfully.

See also https://stackoverflow.com/questions/7524142/what-does-windows-error-0-error-success-mean

[njzjz]

E OSError: Error loading charset-233f44715e5aacaf3a688c2faff5ddf7.dll; The operation completed successfully.

Submitted to adang1345/delvewheel#51

[janosh]

@njzjz can you pin pyproject to the fixed delvewheel>=1.7.4

[njzjz]

@njzjz can you pin pyproject to the fixed delvewheel>=1.7.4

This may not work, as delvewheel is a build-time dependency, not a runtime dependency.

[janosh]

This may not work, as delvewheel is a build-time dependency, not a runtime dependency.

ah, didn't realize that. so the place to pin that would be here

pymatgen/pyproject.toml

Lines 1 to 4 in 1dfc9e4

	[build-system]
	requires = [
	"Cython>=0.29.23",
	# Building against NPY2 will support both NPY1 and NPY2

but looks like we're in the green anyway so would be good to remove the temp install command in test.yml

[janosh]

thanks again @njzjz! 👍

[DanielYang59]

Just recorded another failure in https://github.com/materialsproject/pymatgen/actions/runs/10319015504/job/28566565139:

OSError: Error loading libxml2-14375db763c77d38e40bbe5c028add84.dll; The operation completed successfully.

[DanielYang59]

@janosh Janosh can I have a quick comment on this bump of netcdf4 version here from commit 1dfc9e4?

It seems to cause CI failure (more details on #4128 (comment)) in Ubuntu CI runner for netcdf4>=1.7.1.post1 (1.7.1 is yanked), 1.6.5 seems to work though

I also tried to install delvewheel but doesn't seem to help

There seem to be a lot of discussion around this error but didn't see a solid solution yet. And the issue for me is that I cannot recreate this with my local Ubuntu machine (with exactly the same dependency versions).

[janosh]

netcdf needed to be bumped in #3894 to fix the following CI errors:

_____________ ERROR collecting tests/io/abinit/test_abiobjects.py _____________
tests\io\abinit\test_abiobjects.py:10: in <module>
    from pymatgen.io.abinit.abiobjects import (
src\pymatgen\io\abinit\__init__.py:5: in <module>
    from .netcdf import (
src\pymatgen\io\abinit\netcdf.py:25: in <module>
    import netCDF4
C:\Users\runneradmin\micromamba\envs\pmg\lib\site-packages\netCDF4\__init__.py:28: in <module>
    _delvewheel_patch_1_7_0()
C:\Users\runneradmin\micromamba\envs\pmg\lib\site-packages\netCDF4\__init__.py:25: in _delvewheel_patch_1_7_0
    raise OSError('Error loading {}; {}'.format(lib, ctypes.FormatError(ctypes.get_last_error())))
E   OSError: Error loading charset-[23](https://github.com/materialsproject/pymatgen/actions/runs/10265440264/job/28401647005#step:8:24)3f44715e5aacaf3a688c2faff5ddf7.dll; The operation completed successfully.

if it's causing problems again and this time downgrading helps, by all means let's do it :)

[DanielYang59]

Current pymatgen dependency declare has duplicate (would be fixed #4128), you must have been mislead too :)

The netcdf4 bumped is not really the one installed in CI (the one in optional section is the one actually installed in CI, surely you know):

pymatgen/.github/workflows/test.yml

Lines 41 to 44 in 3ee17e2

	- os: ubuntu-latest
	python: "3.12"
	resolution: lowest-direct
	extras: ci,optional

pymatgen/pyproject.toml

Line 95 in 3ee17e2

abinit = ["netcdf4>=1.7.1"]

pymatgen/pyproject.toml

Lines 100 to 113 in 3ee17e2

	optional = [
	"ase>=3.23.0",
	"beautifulsoup4",
	# BoltzTraP2 build fails on Windows GitHub runners
	"BoltzTraP2>=24.9.4 ; platform_system != 'Windows'",
	"chemview>=0.6",
	"chgnet>=0.3.8",
	"f90nml>=1.1.2",
	"galore>=0.6.1",
	"h5py>=3.11.0",
	"jarvis-tools>=2020.7.14",
	"matgl>=1.1.3",
	"matplotlib>=3.8",
	"netCDF4>=1.6.5",

So it still installed: + netcdf4==1.6.5

---

I looked into the error log again, and turns out it's + netcdf4==1.7.1.post1 causing the following error:

 ______________ ERROR collecting tests/io/abinit/test_pseudos.py _______________
tests\io\abinit\test_pseudos.py:11: in <module>
    from pymatgen.io.abinit.pseudos import Pseudo, PseudoTable
src\pymatgen\io\abinit\__init__.py:5: in <module>
    from .netcdf import (
src\pymatgen\io\abinit\netcdf.py:25: in <module>
    import netCDF4
C:\Users\runneradmin\micromamba\envs\pmg\lib\site-packages\netCDF4\__init__.py:28: in <module>
    _delvewheel_patch_1_7_0()
C:\Users\runneradmin\micromamba\envs\pmg\lib\site-packages\netCDF4\__init__.py:25: in _delvewheel_patch_1_7_0
    raise OSError('Error loading {}; {}'.format(lib, ctypes.FormatError(ctypes.get_last_error())))
E   OSError: Error loading charset-233f44715e5aacaf3a688c2faff5ddf7.dll; The operation completed successfully.

So I might revert that pin to <1.7.1.post1 and (1.7.1.post2 also triggers error), same for latest 1.7.2

It's a huge headache because I cannot recreate these errors locally, which make things very hard.