We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Is there a tutorial on how to start the server with a downloaded model from hugging face?
So far, I have figured out how to build:
cargo build -r
Thought I have downloaded the WizardCoder-Python-34B-V1.0 but can't start the server. Ive tried:
WizardCoder-Python-34B-V1.0
❯ ./target/release/llm-ls /vault/models/WizardCoder/ Content-Length: 75 {"jsonrpc":"2.0","error":{"code":-32700,"message":"Parse error"},"id":null}%
❯ ./target/release/llm-ls /vault/models/WizardCoder/WizardCoder-Python-34B-V1.0.bin Content-Length: 75 {"jsonrpc":"2.0","error":{"code":-32700,"message":"Parse error"},"id":null}%
The text was updated successfully, but these errors were encountered:
@aemonge llm-ls does not run the model for you, it only sends request to a backend API running a model. I'd suggest looking at https://github.com/huggingface/text-generation-inference, either on your computer or on a remote host. You can also use https://github.com/mlc-ai/mlc-llm IIRC. Finally, https://ollama.ai/ should be compatible to llm-ls pretty soon.
llm-ls
Sorry, something went wrong.
I've never heard about ollama, but seams to be what I was looking for <3
Is there an issue liken to the ollama integration, so that I can be notified when it's on ?
Moreover, thank you very much for the support and guides
#40
No problem!
No branches or pull requests
Is there a tutorial on how to start the server with a downloaded model from hugging face?
So far, I have figured out how to build:
Thought I have downloaded the
WizardCoder-Python-34B-V1.0
but can't start the server. Ive tried:The text was updated successfully, but these errors were encountered: