[BUG] Dataset 'lighteval/MATH-Hard' doesn't exist on the Hub

[ChangLiu-DrPatient]

Describe the bug

When I'm using the openllm leaderboard evaluation from llm_eval_harness, this pops up as an error: Dataset 'lighteval/MATH-Hard' doesn't exist on the Hub.

The URL https://huggingface.co/datasets/lighteval/MATH-Hard is empty now.

To Reproduce

Go to https://huggingface.co/datasets/lighteval/MATH-Hard

Expected behavior

At least show something

[stijn-zyphra]

Same problem here.

[zxs789]

I meet the same error too.

[stijn-zyphra]

Did anyone find a temporary solution for this with an exactly similar dataset? I reckon this dataset would probably be uploaded to HF multiple times.

[gadmin7]

Same problem with 'lighteval/MATH' : "DatasetNotFoundError: Dataset 'lighteval/MATH' doesn't exist on the Hub or cannot be accessed."

[gadmin7]

Did anyone find a temporary solution for this with an exactly similar dataset? I reckon this dataset would probably be uploaded to HF multiple times.

Can use this checkpoint : https://huggingface.co/datasets/hendrycks/competition_math

[LenDigLearn]

We have created a drop-in replacement for the original dataset here: DigitalLearningGmbH/MATH-lighteval.

It contains appropriate builder configs s.t. it can be directly used with https://github.com/huggingface/lighteval/blob/main/examples/nanotron/custom_evaluation_tasks.py#L252 (you just need to adjust hf_repo="lighteval/MATH" to hf_repo="DigitalLearningGmbH/MATH-lighteval". Hope this helps someone.

[clefourrier]

Hi! The original dataset has been taken down by a DMCA notice (more info here) so we had to remove access to it from the hub - we're very sorry and hope this will be solved soon, it's super inconvenient for all of us.

Closing the issue in the meantime as there is nothing we can do on our side.