[Problem/Bug]:Holograms not showing in the webview video call

[ragavendranbala]

What happened?

I'm trying to create an webrtc video call app by webview plugin for Unity. I have used this link(https://muazkhan.com:9001/demos/video-conferencing.html) to test the webview. It is working as expected but the Holograms are not sharing the video call.

Is there is any way to show the holograms in the webview video call on Hololens 2?

Importance

Blocking. My app's basic functions are not working due to this issue.

Runtime Channel

Stable release (WebView2 Runtime)

Runtime Version

No response

SDK Version

No response

Framework

Unity Plugin

Operating System

Hololens

OS Version

No response

Repro steps

Open the Unity getting started demo
Navigate to Webview Prefab, and in the url, paste this link (https://muazkhan.com:9001/demos/video-conferencing.html)
Create a Room by entering a unique code inside the URL
Open web browser on PC and Join the room by inputting the same room name.
Now You can see the camera feed of the HoloLens. But the Holograms are not visible in the video call.

Regression

No, this never worked

Last working version (if regression)

No response

[michaelfarnsworth]

Unfortunately, you will not see holograms in the video feed without some custom code. When capturing frames from the devices photo-video (i.e. PV) camera, it will return just what the camera "sees". In order to include the holograms, addition processing must happen to compose the holograms in the camera frames. You can accomplish this by setting up a MediaCapture pipeline and adding the MrcEffect to it. You can find some details on how to do this here: https://learn.microsoft.com/en-us/windows/mixed-reality/develop/advanced-concepts/mixed-reality-capture-overview#creating-a-custom-mrc-recorder

[ragavendranbala]

Hi @michaelfarnsworth , I can see only C# APIS. Since the WebRTC Call implementation is running on Microsoft Edge webview, the MrcEffect option needs to implemented in browser or Unity App?

[michaelfarnsworth]

@ragavendranbala I'm not familiar with WebRTC. However, I suspected you'd need to modify WebRTC itself. Or, if possible, write some kinds of WebRTC plugin. In either case, you'll need to modify the camera-capture pipeline.