New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perplexity for quantized models #643

Open

Hongao0611 opened this issue Oct 17, 2024 · 1 comment

Hongao0611 commented Oct 17, 2024

Hi, If I intend to compute the perplexity for a quantized model, what parameters should I pass to the .compute() function?

For example,

perplexity = load("perplexity", module_type="metric")
good_result = perplexity.compute(predictions=SENTENCES,model_id=MODEL_NAME,add_start_token=False)

Much thx!

Hongao0611 closed this as completed

Hongao0611 reopened this

Author

Hongao0611 commented Oct 17, 2024

I tried loading a quantized model, but it failed:

ValueError: .to is not supported for 4-bit or 8-bit bitsandbytes models. Please use the model as it is, since the model has already been set to the correct devices and casted to the correct dtype.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment