test_optimization_with_linear_constraint in integration tests is flaky.

[mmahsereci]

Test fails about 50% of the time due to infs or nans.

integration_tests/emukit/bayesian_optimization/test_constrained_loop.py::test_optimization_with_linear_constraint

[apaleyes]

I can confirm that it's a bit flacky, but not nearly as bad as 50% for me. Running this test locally, it fails once every 20 runs or so. Is that worse for you @mmahsereci ?

[apaleyes]

Starting to suspect this might be related to #291

[mmahsereci]

I can confirm that it's a bit flacky, but not nearly as bad as 50% for me. Running this test locally, it fails once every 20 runs or so. Is that worse for you @mmahsereci ?

It failed repeatedly during my last PRs which was quite annoying. If I had to guess it was more often than every 20 times. Did not try locally.

[mmahsereci]

It seems that there are two more tests failing now. These ones do not seem to be simply flaky but there are some shape and index errors.

• tests/emukit/bayesian_optimization/test_bayesian_optimization_loop.py::test_loop[3]
• tests/emukit/bayesian_optimization/test_local_penalization_calculator.py::test_local_penalization

See e.g.,
https://github.com/EmuKit/emukit/runs/7689362107?check_suite_focus=true

[mmahsereci]

Perhaps it's due to scipy again. They released version 1.9.0 some days ago and the failed tests seem to be using it while the tests that pass use an older version

https://docs.scipy.org/doc/scipy/release.1.9.0.html

[apaleyes]

It was scipy indeed. Some explanation is in the fixing PR. Tests should be passing again!

[mmahsereci]

Closing since it should be fixed by #437 .