Add fine-tuning documentation

[johnbradley]

Provide documentation on how users could fine-tune BioCLIP.

[johnbradley]

@samuelstevens Suggested some example code from biobench:

binary SVM on vision models: https://github.com/samuelstevens/biobench/blob/main/biobench/newt/__init__.py
ridge classifier (kind of a linear probe): https://github.com/samuelstevens/biobench/blob/main/biobench/plantnet/__init__.py
simpleshot (sort of an upgraded KNN): https://github.com/samuelstevens/biobench/blob/main/biobench/birds525/__init__.py

[johnbradley]

My understanding of the three example above: These use only the visual part of the BioCLIP model and train SVM/ridge classifier/simpleshot based on the image embeddings. No text embeddings are used.

[samuelstevens]

Yep, that's correct. biobench is a method to evaluate vision-only models like the vision encoder from BioCLIP, or DINOv2, etc.

[johnbradley]

@samuelstevens I want to add some documentation comparing the different approaches above. How would you suggest a user go about deciding between the three options?

[hlapp]

Maybe we should try and ask Copilot or ChatGPT, or as seems to be the latest rave, the most recent version of DeepSeek 😉 (Though only half joking)

[samuelstevens]

Yeah I actually think asking an LLM for a summary is a good idea. Fundamentally, BioCLIP embeddings are designed to be linearly separable. The SVM and ridge classifier are linear (unless you choose a nonlinear kernel in the SVM), while the KNN/Simpleshot is non-linear.

Another point is that SVM and ridge classifier are parametric ML methods that explicitly learn parameters, while KNN is non-parametric. But in practice, all three will produce similar results, and I'm not sure that I have any recommendations on what methods are likely to produce the best results based on different tasks.

BioBench would be a good way to test this but it's not set up to evaluate multiple ML methods on a given task right now---I am trying to write a short paper for CV4Animals about BioBench and some findings, but I won't get to that until February. If you're willing to wait, I am happy to share any insights I gain from that project.

[johnbradley]

Asking ChatGPT

Compare SVM, Ridge Classifier, and SimpleShot for fine tuning image embeddings 
that linearly separable for classification.

ChatGPT Output

Recommendations

• SVM: Best suited for tasks where embeddings are high-quality and you need precise decision boundaries.
• Ridge Classifier: Ideal for large datasets where computational efficiency is crucial and embeddings are linearly separable with noise.
• SimpleShot: Excellent for few-shot learning scenarios, or when you want a fast, non-parametric approach without retraining.

If your embeddings are linearly separable and classification is the goal, SVM often provides the best performance in terms of accuracy, but Ridge Classifier may be preferred for faster computation. If simplicity or few-shot adaptability is key, go for SimpleShot.

[johnbradley]

Link to full details from ChatGPT

[hlapp]

Perhaps @samuelstevens can double check what is true and what might be hallucinations, but I'd say pretty good on the face of it.

[samuelstevens]

Perhaps the best strengths of each are:

1. SVMs can support linear and non-linear boundaries and are suitable for binary classification or fewer than ~20 classes (because you train a one-vs-rest for each class).
2. Ridge classifiers are best for linear classification tasks. They require training but are powerful classifiers for many, many tasks, especially with sufficient data.
3. SimpleShot is extremely data-efficient and works well for multiple (20+) classes.

This table summarizes this well I think:

Method	Maximum Classes	Minimum Training Data
SVM	~20	5+ examples
Ridge Classifier	No maximum	10+ examples per class
SimpleShot	No maximum	1+ example per class

So if you have 3 classes with 5 examples each, I would expect SVMs > SimpleShot > Ridge Classifier. If you have 40 classes with 50 examples each, I would expect Ridge Classifier > SimpleShot, and I would expect SVMs to take too long to train reliably.

Does that clarify things?