[Problem/Bug]: Delay in rendering frames using VideoDecoder with optimizeForLatency set to true

[cuijixiong1]

What happened?

I am encountering a bug when using the VideoDecoder API to decode and render video frames on a canvas element. Despite setting the optimizeForLatency flag to true in the decoder configuration, I am experiencing a delay in rendering. Specifically, frames are not rendered on the canvas until the decoder.flush() method is called, or until 10 P-frames have been processed. This results in a latency of rendering 10 frames, which is undesired. I do not encounter this issue when using other decoders like FFmpeg.

Importance

Blocking. My app's basic functions are not working due to this issue.

Runtime Channel

Stable release (WebView2 Runtime), Prerelease (Edge Canary/Dev/Beta)

Runtime Version

No response

SDK Version

No response

Framework

ALL and edge browser

Operating System

Windows 10, Windows 11

OS Version

No response

Repro steps

Steps to Reproduce:

Create a div and canvas element, and append the canvas to the div, and the div to the document body.
Create a new instance of VideoDecoder and configure it with the codec set to 'avc1.64001E' and optimizeForLatency set to true.
Create a Uint8Array and populate it with some video data.
Call the decoder.decode() method with a new EncodedVideoChunk object, passing the Uint8Array as the data property.
After a delay of 10 seconds, call the decoder.flush() method.

Expected Result:
Frames should be rendered on the canvas as they are decoded, without having to wait for 10 P-frames or a call to decoder.flush().

Actual Result:
Frames are not rendered on the canvas until decoder.flush() is called or 10 P-frames have been processed, resulting in a delay in
test.zip

Regression

No, this never worked

Last working version (if regression)

No response

AB#47369056

[cuijixiong1]

test.zip
this test code

[victorhuangwq]

Just to clarify, by the Unity Plugin you are referring to the Hololens Unity plugin right?

[cuijixiong1]

Just to clarify, by the Unity Plugin you are referring to the Hololens Unity plugin right?

I repeated the problem on all ends, including the edge browser

[victorhuangwq]

@srwei could you take a look at this bug?

[cuijixiong1]

@srwei @victorhuangwq Any progress?

[victorhuangwq]

Hi @cuijixiong1 sorry for the delay - this seems like a browser problem, as you have also replicated it on the Edge Browser. I will track it internally in our backlog and route it accordingly, but at the same time I recommend that you use the send feedback function to report it as well.

[cuijixiong1]

@victorhuangwq Is there any progress on this issue now? I tried to submit feedback in the browser, but did not receive any results. I have provided a way to reproduce it. If you need my cooperation, please feel free to contact me at any time

[victorhuangwq]

@cuijixiong1 This is an issue also replicated in Chromium, so it's not just an Edge browser / WebView2 issue. We aren't sure if it's a bug, or by design but we are following up on it.

[cuijixiong1]

@victorhuangwq Is there any progress?

[victorhuangwq]

As mentioned, this is an upstream chromium behavior. I just checked - and it seems like you have already opened a similar request on the chromium repo here, and they have stated it's a won't fix.

https://issues.chromium.org/issues/40937776

Here's their reply:

This bitstream does not specify a bitstream_restriction, and as such will uses the maximum DPB size as its reorder window. The maximum DPB size is 15 given the profile, level, and resolution. I would expect your first output to happen after enqueueing 16 chunks, regardless of the decoder implementation being used.

Closing this issue