[FT] Rerun evaluations with new metrics based on completions saved in details file

[JoelNiklaus]

Issue encountered

Rerunning an evaluation with a new metric requires rerunning the entire inference currently, which can be very costly.

Solution/Feature

It would be great, if we could specify a details file containing the predictions and use that to compute more metrics on.

[JoelNiklaus]

@clefourrier @NathanHB I am happy to implement this. Do you have suggestions for how to best solve this?

[NathanHB]

It would be great ! I think the best way would be to recreate the sample_id_to_response from the details file and run the metric on these.

from the pipeline.py file:

sample_id_to_responses = self._run_model()
self._compute_metrics(sample_id_to_responses)

you would need to inspect what is in sample_id_to_response and try to make it from the details file.

[JoelNiklaus]

Great, will try that, thanks Nathan!