JSON parsing errors with Ollama vision model outputs

[nicklas2116]

• [x ] This is actually a bug report.
• I am not getting good LLM Results
• I have tried asking for help in the community on discord or discussions and have not received a response.
• I have tried searching the documentation and have not found an answer.

What Model are you using?

• gpt-3.5-turbo
• gpt-4-turbo
• gpt-4
• [ x] llava, llama3.2-vision, etc.

Describe the bug
Even though these models produce valid structured JSON outputs, the library is unable to parse the output correctly. Different parsing errors each time.

To Reproduce
Just run any of these models.

Expected behavior
Expected the output to be parsable, since it is correct JSON

[rma-rripken]

I'm surprised this issue hasn't attracted more comments. I tried using llama3.2-vision with instructor and also had issues. Sometimes the JSON output looked fine and I couldn't figure out why it wouldn't parse. What extra info would help troubleshoot? @nicklas2116 how were you building your client? Was it like instructor.from_openai(ollama_client, mode=instructor.Mode.JSON) ?