Meet LLM cost problem

[jasonhp]

I am trying to add a new model provider (Novita AI) to Skyvern-AI. When I was debugging with model meta-llama/llama-3.1-70b-instruct, the llm_api_handler method calls litellm.completion_cost and it threw an error:

This model isn't mapped yet. model=meta-llama/llama-3.1-70b-instruct, custom_llm_provider=openai. Add it here - https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json

Then I went to LiteLLM and discovered that this JSON file was updated by a GitHub action, which requests https://openrouter.ai/api/v1/models to update models' info. But this action has been failing for a long time.

Can I jump over this problem? Or should my models be listed on LiteLLM before I can use them on Skyvern?

[suchintan]

@jasonhp I think they need to be listed in LiteLLM for them to work in Skyvern, although we shouldn't block Skyvern if cost information is not available.

Any chance you can add a try-catch around this code and continue gracefully?

[jasonhp]

@suchintan Hi thank you for the suggestion!

I’m not sure what the appropriate approach is here. Should I set llm_cost to 0 and update the database?

I think the final solution is to fetch the model information from the /models API provided by LLM providers? For now, it looks like I need to manually specify them in setup.sh, is that correct?

[suchintan]

#1 - I think setting it to 0 is a good workaround in this situation!
#2 - That's sort of correct. The setup.sh config is for others to use it out of the box — the biggest change that's needed is updating LLMConfig

skyvern/skyvern/forge/sdk/api/llm/config_registry.py

Line 247 in 9674e83

if settings.ENABLE_AZURE_GPT4O_MINI:

[jasonhp]

Done and created a PR #1508. Please have a check @suchintan.

[suchintan]

Got this merged in -- thank you so much @jasonhp