Replies: 3 comments 2 replies
-
|
I think we definitely would like to support local models via API on either vLLM or ollama in the future. From a technical standpoint, the LLM needs to:
From my experiments, all frontier models from GPT-4o to Gemini are capable of 1 & 2 out of the box, and even cost-optimized models like GPT-4o-mini can be fine-tuned to understand 1&2 as well. I think it's an open question whether the 72b version would work well in this format but it definitely be worth a shot. The barrier to running this experiment would be implementing a |
Beta Was this translation helpful? Give feedback.
-
|
That's likely above my skillset, but I'll take a look! |
Beta Was this translation helpful? Give feedback.
-
@wbste Molmo definitely does look promising. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah molmo is the first open model I found that should provide reliable coordinates for these sort of use cases. I also saw this..not sure if that opens more doors to other models or not: https://github.com/microsoft/OmniParser |
Beta Was this translation helpful? Give feedback.
-
|
We're starting to gather data for a fine-tune of Molmo! https://github.com/theredsix/cerebellum/tree/mind2web/training |
Beta Was this translation helpful? Give feedback.
-
Any chance that a model like Molmo (that can "point") could be used as a self-host solution? I believe vLLM could be used as an endpoint?
https://molmo.allenai.org/blog
Beta Was this translation helpful? Give feedback.
All reactions