SLEAP Inference labeling only first 20 frames instead of random

[Asolee24]

Hello,

I am trying to run inference on a slp file I have annotated after running training for centered-instance and centroid on a separate HPCC. I was able to finally (at least I believe so) successfully train these two models and was able to export them back to my computer. Then I was able to reopen my slp file I annotated and went to run inference opening my training_config.json files for both centroid and centered instance respectively. I was able to then set parameters (see attached below) to what I believed made sense based on a similar setup I was shown by another student who has used the program. Then run inference and it gave me a result of inference ran on 20 frames. Instances were predicted on 20 frames (0 frames with no instances found). In the video I saw that it had predicted on only the first 20 frames instead of random as I had indicated and I am unsure as to why, I am still very new to SLEAP and any advice would be greatly appreciated.

Best,

Autumn
Setup within SLEAP and outputs:





training_config.json
training_config.json

[eberrigan]

Hey @Asolee24,

Inference can be found on the dark blue frames (in the bottom seek bar). You can turn off the tracker until you get predictions on all of your frames, since tracking needs predictions on consecutive frames. You have the simple tracker turned on right now.

Best,

Elizabeth

[Asolee24]

Hello Elizabeth,

Thank you for getting back to me. The dark blue bars are the frames I had annotated previously. I am still very new to sleap so I am not sure how you turn off the tracker and the other aspects you mentioned.

Thanks,

Autumn

[roomrys]

Hi @Asolee24,

You are correct:

• Bold dark blue: user labels on labeling suggestions
• Regular light blue: predictions on labeling suggestions
• Regular black: user labels not in labeling suggestions
• Thin black: user labels not in labeling suggestions

The tracker is run post-inference and can be disabled in the Inference Pipeline tab by setting the "Tracker (cross-frame identity) method" to None.

Thanks,
Liezl

[Asolee24]

Hello,

Also, I am a bit unclear as to where the light blue labeling suggestions are? Were you just mentioning what all the different types you might see/come across are?

Thanks,

Autumn

[roomrys]

Hi @Asolee24,

Yes, I was just mentioning all the different types you might encounter.

Thanks,
Liezl

[roomrys]

Hi @Asolee24,

I think it likely has to do with trying to run inference on a .pkg.slp file - this file type is only intended to be exported for training not further annotation. The .pkg.slp file imbeds the images for select frames (based on your selection while exporting) which is why I believe it might be causing the problem. I will verify this.

In the meantime, do you have access to the original .slp (not .pkg.slp) to further annotate with prediction-assisted labeling?

Thanks,
Liezl

[roomrys]

Summary

I think what is happening is that the HDF5Video.frames is returning a fairly low number - something less than 21. Hence when we go to grab random frames to sample from, we actually return all a list of range(video.frames), i.e. numbers 0 to less than 20. The remove_user_labeled function:

sleap/sleap/gui/app.py

Lines 1453 to 1459 in 59e1f70

	def remove_user_labeled(video, frame_idxs):
	if len(frame_idxs) == 0:
	return frame_idxs
	video_user_labeled_frame_idxs = {
	lf.frame_idx for lf in user_labeled_frames if lf.video == video
	}
	return list(set(frame_idxs) - video_user_labeled_frame_idxs)

doesn't affect this list of indices since there are no user labeled frames in this dataset that intersect.

---

Trace

In MainWindow._show_learning_dialog, we update the LearningDialog.frame_selection by calling MainWindow._get_frames_for_prediction:

sleap/sleap/gui/app.py

Line 1557 in 59e1f70

self._child_windows[mode].frame_selection = self._get_frames_for_prediction()

.

In MainWindow._get_frames_for_prediction, for the "random" frame selection key, we sample either 20 random frames from each video (or return all video frames if video.frames is less than 20) and then remove user labels from the list of sampled frames:

sleap/sleap/gui/app.py

Lines 1484 to 1489 in 59e1f70

	selection["random"] = {
	video: remove_user_labeled(
	video, random.sample(range(video.frames), min(20, video.frames))
	)
	for video in self.labels.videos
	}

.

Now, what might video.frames be for a Video in a pkg.slp? Well, when we save the pkg.slp out, we create a dataset with shape matching the number of frames we want to save out:

sleap/sleap/io/video.py

Lines 1457 to 1462 in 59e1f70

	dset = f.create_dataset(
	dataset + "/video",
	(len(frame_numbers), max_frame_size),
	dtype="int8",
	compression="gzip",
	)

and change the Video.backend to HDF5Video

sleap/sleap/io/video.py

Lines 1486 to 1493 in 59e1f70

	return self.__class__(
	backend=HDF5Video(
	filename=path,
	dataset=dataset + "/video",
	input_format="channels_last",
	convert_range=False,
	)
	)

and also set the Video.backend.filename to ".":

sleap/sleap/io/format/hdf5.py

Lines 298 to 303 in 59e1f70

	# Replace path to video file with "." (which indicates that the
	# video is in the same file as the HDF5 labels dataset).
	# Otherwise, the video paths will break if the HDF5 labels
	# dataset file is moved.
	for vid in new_videos:
	vid.backend.filename = "."

.

And, when we read in the pkg.slp, we first replace the unstructured Video.backend.filename to point to the current filename (i.e. the pkg.slp path):

sleap/sleap/io/format/hdf5.py

Lines 97 to 102 in 59e1f70

	# Video path "." means the video is saved in same file as labels, so replace
	# these paths.
	for video_item in dicts["videos"]:
	if video_item["backend"]["filename"] == ".":
	video_item["backend"]["filename"] = file.filename

, not really the point here... the point is to follow:

But, really, since we saved out to an HDF5Video, the frames comes from the HDF5.dataset attribute which is of shape matching the number of frames exported with the pkg.slp:

sleap/sleap/io/video.py

Lines 248 to 251 in 59e1f70

	@property
	def frames(self):
	"""See :class:`Video`."""
	return self.__dataset_h5.shape[0]

.

[Asolee24]

Hello Liezl,

This is all extremely helpful. I wanted to just figure out the full pipeline first and have it working (just to know the pipeline with a few frames) then I was going to finish annotating a more robust .slp file to train. I do have access to the original, but correct me if I am wrong I thought prediction-assisted labeling still involved running inference. So is it that you should be running inference on the original slp, a seperate video entirely, or a different file type entirely like an HD5F video. Also, is it unable to work with 20 frames trained at all? Does it have to be from the beginning a greater number?

Thank you,

Autumn

[roomrys]

Hi @Asolee24,

I do have access to the original, but correct me if I am wrong I thought prediction-assisted labeling still involved running inference.

Yes, you are correct. To obtain the predictions that assist with labeling, you will need to run inference.

So is it that you should be running inference on the original slp, a seperate video entirely, or a different file type entirely like an HD5F video.

The problem here is that you are attempting to use the exported .pkg.slp for the prediction-assisted labeling (it is designed to be used for training on a different machine if your current machine doesn't have access to a GPU - not for adding more prediction/labels). Once you have run training, you will obtain a model. Then:

1. Head back to your original slp and generate a few more Labeling Suggestions
2. Use the model you obtained to run inference on your original slp (Predict On: suggestions)
3. Correct the predictions
4. If your machine does not have a NVIDIA GPU: export the .pkg.slp for training on a different machine
5. Run training to obtain an updated model
6. Repeat

Also, is it unable to work with 20 frames trained at all? Does it have to be from the beginning a greater number?

Training on just 20 frames is perfectly acceptable and encouraged to get some sort of model to help with prediction-assisted labeling. The error you ran into was because the logic for selecting 20 random frames to predict on isn't compatible with pkg.slp's - we could open a pull request to fix this, but also note that pkg.slp's were not intended to be used for adding labels to (prediction or user), just for exporting annotation/images for training elsewhere.

Let us know if that is helpful and/or if we can help some more.

Thanks,
Liezl

[Asolee24]

Hello,

Thank you, it seems to be working and doing random frames. I appreciate the help!

Best,

Autumn