ci: relax recall thresholds for CI

[westonpace]

I am concerned that a number of our tests test with random data and have quite strict recall thresholds. These tests seem to fail on a regular basis.

On the one hand, this may be a concern that we have broken recall but on the other hand I believe these are just in the category of "it's possible to get bad results sometime with the wrong random data".

In other words, if our CI thresholds are set at P=0.05 then that is too strict because if we have a dozen tests and they fail 5% of the time we will have too many CI failures.

I have proposed some very relaxed suggestions here but feel free to propose alternative suggestions. I'd like these tests to be designed so that if the test fails it is a sign that we broke something and we need to fix it, not just something we ignore.

We could potentially add recall benchmarks to our benchmarking suites if we are concerned about the slow degradation of recall or our recall performance. These should probably be designed with real (not random data) and could hopefully be made more reliable in that way.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.07%. Comparing base (022135b) to head (659dea1).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3282      +/-   ##
==========================================
+ Coverage   79.00%   79.07%   +0.07%     
==========================================
  Files         246      246              
  Lines       86900    87471     +571     
  Branches    86900    87471     +571     
==========================================
+ Hits        68655    69168     +513     
- Misses      15377    15439      +62     
+ Partials     2868     2864       -4     

Flag	Coverage Δ
unittests	79.07% <ø> (+0.07%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[BubbleCal]

let's low the recall requirement for only 4bit IVF_PQ:

• L2 -> 0.85
• Cosine -> 0.85
• Dot -> 0.75

[westonpace]

Ok, I've scoped the change to just 4bit IVF_PQ

[westonpace]

@BubbleCal this case failed (recall is 0.89 and target is 0.9). Is this a bug or should we relax this threshold too?

index::vector::ivf::v2::tests::test_create_ivf_hnsw_pq::case_3

[BubbleCal]

@BubbleCal this case failed (recall is 0.89 and target is 0.9). Is this a bug or should we relax this threshold too?

index::vector::ivf::v2::tests::test_create_ivf_hnsw_pq::case_3

let's relax dot to 0.85

[westonpace]

Looks like you beat me to it in 39f12dc#diff-6de816b72e7c722316243c57df4f809ad34dc8581367c72335154dada48c40ed

Thanks!