Video unmute race condition

🎻 December 2022 fiddle

Want to learn more about WebRTC?

Look us up here for the WebRTC Fiddle of the Month, created… once a month.

Or just enroll to one of our excellent WebRTC training courses.

Resources

• The fiddle: https://jsfiddle.net/fippo/yomxzacj/4/

Transcription

Tsahi: Hi, and welcome to our WebRTC Fiddle of the Month.  with me is Philipp Hancke, and this time we’re going to talk about race conditions between signalling and media when we try to unmute a video channel. That was a bit of a mouthful, but what does it mean exactly? Philipp?

Philipp: Right. This is something about an effect I observed in the WebRTC Working Group meeting last week and basically there was a participant who was muted, had to be unmuted, and then they unmuted. Then I saw the video of that person jumping basically, that was super annoying, so I was like why does it happen?

—

Okay, so we have a basic fiddle. We start as usual with two peer connections whereas an up to each other. We call GetUserMedia to get a video, we add tracks to the peer connection, then we do negotiation.

And then we have a bunch of buttons here. We have this connection which is live. And now we have a series of steps to show muting, receiving the signalling message, and unmuting. Let’s try the first version which is the one which really annoys me.

We hide the video so I basically pretend that I’m stopping a video. I stopped sending. Then I show it again to simulate that the unmute signalling message arrives, which means you get the old video you see I’m not moving my hands are in a different position. And if I now click resume and raise the hand you will see this jump.

And, in the phone setting, it’s even more annoying than this. So, what we can do is we can actually only show the video once the actual frame arrives, which is better in terms of experience because you don’t have this old video hanging around for a split second and you don’t see…

Tsahi: Okay, just to explain what happens now is that if I want to unmute the video, as the user, I click the unmute button on my video.

And when I do that, I start sending video, because that’s what I asked to do, and in parallel I’m probably sending some kind of signalling message to say that I want to unmute.

Now the signalling message might come before or after the video starts arriving. And if it arrives before the full first frame arrives, then I am probably going to show to the user, the last image that was shown before that, on that video element.

Philipp: Exactly.

Tsahi: So what you’re trying to do is to eliminate that by waiting for the full video frame to arrive after the unmute was let’s call it; clicked in the right to my machine.

Philipp: Correct. And this happens more often if you have distributed settings with signalling servers and participants in different regions. So we do that we again hide our video, stop the track, click the synchronise button, then we resume and it will automatically show the video once a new frame arrives. So there was no jump this time.

Tsahi: And how do you make that magic happen?

Philipp: That’s a good question. So there’s a nice API called requestVideoFrameCallback(), it’s something that is happening on the video element. It’s an API on that. It is supported, I think, in Chrome and Safari. And basically that gets called whenever the video element is rendering a video frame. And you can see that down here, if the checkbox is checked and  the video is not hidden, then we hide it or the other way around. If the video is hidden, then we unhide it.

Then we need to call requestVideoFrameCallback() again. So it’s not a WebRTC API, but it’s also supporting API’s in Chrome that are incredibly useful and provide by user experience.

Tsahi: Okay, so what we did today is created a small let’s call it polish, to how we deal with unmuting video channels by making sure that we actually have a video frame to show and then we don’t show a frozen image instead.

Philipp: Correct.

Tsahi: Okay. Thank you for that, Philipp. And see you all in our next WebRTC Fiddle of the Month.