Front-Back confusion + control #predicted_skeletons

[RubenTeunisse]

Hi!

The models coming out of the sleap framework are working great and we are looking for two further optimizations.

1. We are often experiencing this particular mistake where the front and back of the mouse are confused by the model. The tail start is recognized as the snout and vice versa, while the orientation is rather clear to the human eye:

Do you have any specific recommendations or paramaters to tune to target this?

The model is based on 285 labeled frames, and we also see it for a model train trained on over a thousand frames from slightly different videos.

The model is trained using the following settings:

2. A different issue we experience is that two mice are 'recognized' where there really only is one and the real other mouse is not recognized at all, which then results in difficulties for the cross-frame tracking.

Is there a way to get more candidate instances out of the inference_model.predict_on_batch() function (like changing some paramater for training)? Even before trimming based on the instance_scores/centroid_vals we get about the number of suggested instances equal to the number of mice in view. This is usually perfect, but in the case of a mistake like just described, our custom cross-frame identity tracking is unable to select a candidate instance that was considered less likely based on the single-frame inference, but much more likely given the temporal information.

It would be great if you could guide us in the right direction.

Cheers,
Ruben

[Marielamend]

Hello @RubenTeunisse!

Sorry for the delayed response,

1. Some quick configuration tweaks you can try is adjusting your augmentation to -+180 since your data set is in top view.

2. You can also try tweaking the max stride by decreasing it to 16 since you want your receptive size to be approximately as large as the animals in frame.

A follow-up question:
Have you tried using the top-down approach? We find that this approach may yield better results when the animals are small relative to the frame such as we see in your data set.

let us know how this goes,
Mariela

[RubenTeunisse]

Hi Mariela,

Thank you for your suggestions!
At some point I have tried these things and it did not improve, but maybe I then did not have enough frames labeled or something, I will try it again, thanks!

Cheers,
Ruben

[Marielamend]

Hi again @RubenTeunisse!

If you haven't already, you can also double-check your user-labeled frames to check for any mislabeled frames mixing the front for the back nodes. Here is a relevant discussion post and helpful direction on how others have approached this Find mislabeled frames? #1253. In addition, if there were frames in which not all animals were labeled, this will also typically result in more missed detections.

Is there a way to get more candidate instances out of the inference_model.predict_on_batch() function

You can reduce the peak threshold during inference to try to recover more detections, but ultimately it'll be hard to prevent noise from making the tracking worse.

• Have you tried tracking using v 1.3.3? --> this version has a new approach for keeping a max number of tracks CLI example here

Lastly,
If you want to share your custom tracking code, we can assist in handling tracking across gaps!

Best,
Mariela