You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When I call the server using openai python package:
fromopenaiimportOpenAIclient=OpenAI(
base_url="http://localhost:10322/v1", # "http://<Your api-server IP>:port"api_key="sk-no-key-required"
)
chat_completion=client.chat.completions.create(
model="models/mys/ggml_llava-v1.5-13b/ggml-model-q4_k.gguf",
messages=[
{"role": "user", "content": "Write a limerick about python exceptions"}
],
max_tokens=3,
)
print(chat_completion.usage.completion_tokens) # returns 3, ok.print(chat_completion.choices[0].finish_reason) # returns "length", ok.chat_completion=client.chat.completions.create(
model="models/mys/ggml_llava-v1.5-13b/ggml-model-q4_k.gguf",
messages=[
{"role": "user", "content": "Write a limerick about python exceptions"}
],
max_completion_tokens=3,
)
print(chat_completion.usage.completion_tokens) # returns much more than 3 (complete answer).print(chat_completion.choices[0].finish_reason) # returns "stop".
According to OpenAI API, max_completion_tokens argument is replacing the deprecated max_tokens argument.
It's seems that only max_tokens is not ignored by the server.
Environment and Context
llama_cpp installed with pip install llama-cpp-python[server] print(llama_cpp.__version__): 0.3.6 print(openai.__version__): 1.59.7
The text was updated successfully, but these errors were encountered:
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Current Behavior
I'm running llama-server with following command:
(models downloaded from https://huggingface.co/mys/ggml_llava-v1.5-13b/tree/main)
When I call the server using openai python package:
According to OpenAI API,
max_completion_tokens
argument is replacing the deprecatedmax_tokens
argument.It's seems that only
max_tokens
is not ignored by the server.Environment and Context
llama_cpp installed with
pip install llama-cpp-python[server]
print(llama_cpp.__version__)
: 0.3.6print(openai.__version__)
: 1.59.7The text was updated successfully, but these errors were encountered: