how to reproduce the results of alpaca_eval

[jxs129]

Dear authors,

Thanks for the interesting paper!

I am trying to reproduce Table 4, some questions about AlpacaEval:

• how to build the AlpacaEval dataset?
• which script can reproduce the AlpacaEval results in Table 4?

thanks

[huangtiansheng]

Hi,

Thanks for reaching out. Check out this script:

https://github.com/git-disl/Vaccine/blob/main/script/sft_finetune/alpaca.sh
https://github.com/git-disl/Vaccine/blob/main/script/vaccine_finetune/alpaca.sh

For initial experiment/reproduce, we don't recommend alpacaeval because it needs money and actually the benchmark itself is not that stable :)

[jxs129]

thanks for your response.

where can I find the data/alpaca.json?

[huangtiansheng]

It is this file https://huggingface.co/datasets/tatsu-lab/alpaca_eval/blob/main/alpaca_eval_gpt4_baseline.json

Download and rename it to alpaca.json and put it to data directory. Then it should work.

[jxs129]

ok, great!