Poor performance using top-down model: What parameters to change?

[glinnes12]

Hello SLEAP-Team,

I've been labelling and training models for the past few weeks and while the bottom-up model seems to work quite well, I just can't seem to make the top-down model work (which is the one we'd preferably use).
The predictions are unfortunately very inaccurate and many body parts are often missing... I tried changing all sorts of parameters but none of that seems to help and I am running a bit out of ideas what else I could do or whether it might be a specific configuration of multiple parameters that would help improving the model. Maybe you have some suggestions?

Here some infos about our data and things that I've tried so far:
I have 10 videos in SLEAP, each around 55'000 frames long, and labelled 708 frames (approximately 70 frames per video). We're trying to track 2 rats, which are kept separate (see screenshot below). I assigned tracks too, but I believe they're mainly for using the top-down ID model (which I haven't been using, since SLEAP seems to be able to distinguish between the two animals quite alright). I tried various skeletons to see if that might make a difference too (but no success with that...).

When training the model, I chose the centroid as the anchor point, and kept the sigma for anchor point and nodes both at 2.50 (though I have to admit I don't quite understand what the sigma exactly expresses). The other parameters I adjusted based on things I read in the forum (e.g. 180 degree rotation angle for overhead camera setting, adjusting input scaling and max stride etc.), but maybe I should change some of them?

I was wondering whether the issue might also stems from the inference parameters I use? So far, I always set the max instances to 2. I initially used a tracker but since the animals never cross, I set it now to 'none' and it doesn't make any difference. Here is an example of a predicted frame (rearing events seem to be especially tricky to detect):

These are the evaluation metrics for my current model:

Thank you for any suggestions on what else to twitch to make the top-down model work more accurately :)

[Marielamend]

Hello @glinnes12!

Thank you for reaching out! So sorry for the delay.

To best help you, if you could please provide us with the training_log.csv from your model folder; this will allow us to visualize the loss curves from training!

In the meantime you can try a few tweaks in your
Centered Instance Configuration:

• You can try to increase your sigma value since it makes it easier to learn coarser features. Some extra insight about the sigma:

The sigma trades off between precision and ease-of-training. If sigma is <2, SLEAP will have a hard time training the model to predict confidence maps since they'll be too sparse. If sigma is >5, SLEAP will probably have an easy time training the model, but your localization precision will be pretty bad (i.e., the points will jump around a lot).

• Increase your scale augmentation

Best,
Mariela

[roomrys]

Hi @glinnes12,

Most often, when the top-down model does not detect an animal it is because it fails to find the anchor part in the first model (centroid). The anchor part is then used in the second model (centered instance) to take a centered crop about that point and then find all other body parts.

1. Try using the tail base as the anchor point

I would say that your anchor point (the "centroid" node) is still visible in that one example where it failed to detect the up-close animal. I wonder if the node looked different enough to not be detected? I'm not convinced this is the case, but regardless, the "centroid" node is not actually closest to the center of your labeled skeleton. With the added tail node, the tail base is the most central node for a crop.

2. Use Auto for Crop Size

Even switching the anchor nodes, I still have some concerns that the animal is too close (takes up too much video space) to use top-down as I recall running into a hard-coded limit on the max crop size allowed. But if the max crop size was coming into play, I would expect to see some external nodes left undetected (like there are in the predicted animal). I would recommend always clicking Auto for the crop size as this will recalculate how big the largest animal in your labeled set is.

3. Head back to the defaults and only change rotation and scale

You have changed a lot of settings from the baseline.centroid and baseline_medium_rf.topdown - the only ones I would recommend changing are the +-180 rotation (and scale wouldn't hurt and also makes sense here). The increased receptive field size only really matters when trying to group body parts using the locations of other body parts (i.e. in the bottom-up model). The top-down does this grouping based on the centered crops.

4. Reconsider if top-down is the best model to use

Usually we use top-down when the animal only takes up a small portion of the screen - is there a reason you want to use the top-down model on this dataset?

Thanks,
Liezl