🔖 Issues Auto-Labeller

[August-murr]

I was playing around with OpenAI API and was thinking about useful things we could do with the repos and all the data like issues and feedbacks and trying to automate some tasks or just do something useful with them like analysis or reports and I thought of an auto labeller.

This auto labeller uses OpenAI API, so it needs an OPENAI_API_KEY secret. It will cost a few bucks per month, but there are also string-based and regex-based labellers that may be faster.

If you are interested then we could test and see how consistent it is.

couple of ideas for improvement :

we could one-shot or few-shot prompt it for more accuracy or use different kinds of models both of which can cost more.

as for the speed, it takes about 30 seconds to label an issue.

any other ideas on anything useful we could do with OpenAI API or other stuff to either automate or analyze and get some value?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[qgallouedec]

Fun feature! Do you have a demo repo?

[qgallouedec]

Have you tried with the HF api? It could be a free alternative

[August-murr]

Fun feature! Do you have a demo repo?

Just pushed it to my own fork

[qgallouedec]

I'll open a batch of issues to test it

[August-murr]

Have you tried with the HF api? It could be a free alternative

Honestly, this was really effortless since I simply forked a mostly functional actions extension. Modifying it to work with the HF API will require much more effort. also it uses GPT-4o, there aren't many open-source models that are this accurate.

If it's absolutely necessary, then I can do it, but I honestly don't think it's worth the effort.

However, if you believe it is important, then I'll go ahead and do it.

[qgallouedec]

It doesn't seem like a big deal to me. Probably something like this could work

from huggingface_hub import InferenceClient

client = InferenceClient(model="meta-llama/Llama-3.2-1B-Instruct", token="your_token")
content = "Find the label among these: question, issue."
completion = client.chat_completion(messages=[{"role": "user", "content": content}], max_tokens=256)
response = completion.choices[0].message.content

there aren't many open-source models that are this accurate.

This task is very simple, I don't think we absolutely need GPT-4o here. And even if the labeled fail, it's not a big deal.

[August-murr]

It doesn't seem like a big deal to me. Probably something like this could work

from huggingface_hub import InferenceClient

client = InferenceClient(model="meta-llama/Llama-3.2-1B-Instruct", token="your_token")
content = "Find the label among these: question, issue."
completion = client.chat_completion(messages=[{"role": "user", "content": content}], max_tokens=256)
response = completion.choices[0].message.content

there aren't many open-source models that are this accurate.

This task is very simple, I don't think we absolutely need GPT-4o here. And even if the labeled fail, it's not a big deal.

ok got it

[qgallouedec]

Do you know if you can access the tag description? It could help the model in its prediction

[August-murr]

Do you know if you can access the tag description? It could help the model in its prediction

tag description as in the label description?
like:
🚀 deepspeed --> Related to deepspeed

If so, yes, it is part of the prompt.

[August-murr]

I tried using the Llama 1B model, and it "functioned," but for the TRL, I switched to the 70B model. However, I couldn't test it with the 70B because it requires a subscription.

Don't forget to add the HF_API_KEY to the secrets.

I got a context length error (limit of 4096 tokens) when using the Llama 1B model, which was weird since it supports up to 128k tokens. Since I can't use the 70B model, I'm unsure if it's a problem or not.

[qgallouedec]

Nice! Just commit the suggested change please

[August-murr]

I got a context length error (limit of 4096 tokens) when using the Llama 1B model, which was weird since it supports up to 128k tokens. Since I can't use the 70B model, I'm unsure if it's a problem or not.

This can be problematic when dealing with issues that require a long context. The exact error message received was:
Input validation error: inputs tokens + max_new_tokens must be <= 4096. Given: 9223 inputs tokens and 50 max_new_tokens
I couldn't find a solution or parameter to set, possibly from the inference endpoint.

[qgallouedec]

A bit hacky but you can take the 15000 first strings. It should be enough for most issues:

content = content[:15000]

[August-murr]

A bit hacky but you can take the 15000 first strings. It should be enough for most issues:

content = content[:15000]

more like 4000
But it works well.