Reencoded Vid Adding Extra Frame

[jannhan]

I ran into the error of being unable to load the second to last frame of entire current video (Discussion #1895), and used the reencoding method to make a copy of the video. However, when I try to replace the video in my sleap file (with the same labels), the labels are off because the new version of the video has 9-11 extra frames for some reason. Is there any way to fix this? Thanks

[roomrys]

Hi @jannhan,

The first thing that comes to mind is maybe the fps of the original and encoded differ?

We can check the frame rate of the video(s) with:

ffprobe -v error -select_streams v:0 -show_entries stream=r_frame_rate -of default=noprint_wrappers=1 "input.mp4"

.

If they differ, I would suggest trying to re-encode specifying the rate:

ffmpeg -y -i "input.mp4" -c:v libx264 -pix_fmt yuv420p -preset superfast -crf 23 -r <original_fps> "output.mp4"

.

Let us know if that helps any or if we should keep searching for a solution. Some additional information for us that would be helpful in debugging is the output of:

ffprobe -i input.mp4 -hide_banner

for both the original and encoded videos.

Thanks,
Liezl

P.S. The other take on this problem is that we can provide a script to shift labels (assuming a constant offset) if that is acceptable.

[jannhan]

The frame number is nearly the same, but for some reason some of the labels are still off by quite a bit. Is there a way to shift the labels?

[roomrys]

Hi @jannhan,

Do all the labels (per video, not across videos) shifted by a constant amount?

Thanks,
Liezl

[jannhan]

Yes, I think they are all 1-2 frames ahead.

[roomrys]

Hi @jannhan,

I created this script that will shift the labels for you.

1. You will need to download and unzip.

2. Modify the if __name__ == "__main__" section of the file where I left a few TODOs:

   if __name__ == "__main__":
       ds = r"path/to/dataset.slp"  # TODO: Replace with the path to the dataset file
       output_ds = (
           r"output_dataset.slp"  # TODO: Replace with the path to the output dataset file
       )
   
       # Load the labels and videos from the dataset file
       labels, videos = load_labels_and_videos(ds)
   
       # TODO: Remove this block of code (AFTER first run if using `shift_per_video`)
       raise NotImplementedError(
           "\n\n1. See print statement above for video indices (before 'Traceback').\n"
           "2. Specify either `shift_per_video` (using above indices) or `shift`.\n"
           "3. Remove this entire block of code.\n"
           "4. Rerun the script."
       )
   
       # TODO: Replace with the number of frames to shift the labels by for each video
       shift_per_video = {
           videos[0]: 1,  # This will shift the video at index 0 by 1 frame
           videos[1]: -1,  # This will shift the video at index 1 by -1 frame
       }
       main(labels=labels, output_ds=output_ds, shift_per_video=shift_per_video)

   . Note that the shift_per_video argument allows you to pass in specific Video objects as the key (e.g. videos[0]) and then pass in different shift amounts for each video. If some videos require 0 shifting, then you can exclude them. To help you figure out the video indices in your slp project, I included a print statement above:

       for video_idx, video in enumerate(videos):
           print(f"Video {video_idx}: {video.filename}")

   . The first time you run the script, you will get a NotImplementedError - this is intentional so that you can see a print statement of the video indices (IF you need them... keep reading).

   Alternatively, if all labels across all videos are shifted the same amount, then you can instead replace the lines

       # TODO: Replace with the number of frames to shift the labels by for each video
       shift_per_video = {
           videos[0]: 1,  # This will shift the video at index 0 by 1 frame
           videos[1]: -1,  # This will shift the video at index 1 by -1 frame
       }
       main(labels=labels, output_ds=output_ds, shift_per_video=shift_per_video)

   with

       shift = 1  # TODO: Replace with correct shift amount for ALL frames in ALL videos
       main(labels=labels, output_ds=output_ds, shift=shift)

   .

3. Open a terminal and activate the sleap environment

   conda activate sleap

4. Run the script with the command

   python path/to/sleap_shift_labels_temporally.py

5. See NotImplementedError, take note of video indices to construct shift_per_video, and remove this block from if __name__ == "__main__":

       # TODO: Remove this block of code after first run if using `shift_per_video` (or before first run if using `shift`)
       raise NotImplementedError(
           "\n\n1. See print statement above for video indices (before 'Traceback').\n"
           "2. Specify either `shift_per_video` (using above indices) or `shift`.\n"
           "3. Remove this entire block of code.\n"
           "4. Rerun the script."
       )

6. Rerun script

7. Check shift amounts by opening the output slp in the GUI

Let me know if this works out for you and apologies for all the trouble.

Thanks,
Liezl

[jannhan]

This worked perfectly! Thank you so much for taking the time to write all this and reply so quickly, Liezl, you're a lifesaver!