Question about usage of VFI model

[Qianqian3764]

Hi, thanks for your great job！

For TA method，why use VFI model to generate intermediate frames? Isn't using intermediate frames as labels when training VFI models? My question is, since the middle frame can be obtained, why do we need to use an additional VFI model to generate it?
If the VFI model serves the subsequent multi frame feature fusion, it is reasonable because it needs to be used to generate the optical flow from the target frame to the source frame.

[LiuJF1226]

Thank you for your attention! The data split used in VFI training is identical to that employed in self-supervised MDE training. In standard self-supervised MDE training, a data sample consists of three consecutive frames ($I_{t-1}$, $I_{t}$, $I_{t+1}$), where $I_{t}$ is the target frame and the other two are source frames. In VFI training, we interpolate $I_{t}$ using $I_{t-1}$ and $I_{t+1}$, with $I_{t}$ as the label, and no other intermediate frames are utilized as labels. Once the VFI model is trained,, we utilize ($I_{t-1}$, $I_{t}$) to synthesize an intermediate frame $I_{t-0.5}$, and ($I_{t}$, $I_{t+1}$) to synthesize $I_{t+0.5}$. $I_{t-0.5}$ and $I_{t+0.5}$ serve as two additional target frames in our following self-supervisef MDE training, for both single-frame and multi-frame depth models. This diversifies data distribution in terporal dimension. Also, the VFI model serves the subsequent multi frame feature fusion as you said.

[Qianqian3764]

Thanks very much for your reply！
This question is completely dispelled. That's a great idea!
I looked at the training code train.py.

Why not count the loss of [-0.5,0,0.5] for a single frame?