Identifying Chrome field trials

🎻 October 2021 fiddle

Want to learn more about WebRTC?

Look us up here for the WebRTC Fiddle of the Month, created… once a month.

Or just enroll to one of our excellent WebRTC training courses.

Resources

• Focus change: https://groups.google.com/g/discuss-webrtc/c/hpAVnDA9Ung/m/KIsHFqP2BwAJ
• Facebook bug report: https://bugs.chromium.org/p/webrtc/issues/detail?id=13212
• Understanding the field trials you are running on chrome: chrome://version/?show-variations-cmd

What to run in your browser on the version page:

const parts = document.getElementById("variations-cmd").innerText.split(" --").map(s => s.split("="));
experiments = new Map();
forced = parts[0][1].split("/");
for (i = 0; i < forced.length; i += 2) {
  experiments.set(forced[i], forced[i + 1]);
}
console.log(experiments)

Transcription

Tsahi: Hi everyone, and welcome to this WebRTC Fiddle of the Month. And this time we’re going to talk about Field Trials or you’re going to find field trials and understand it onward. So we’ll start by what field trials are. And the easy part of it, or the way to explain that it’s A/B testing. Chrome has what, 1-2 billion users around the globe. And when they want to roll out a new feature, they need to test it. So the first test it in their QA, and then then they move forward and want to test it with real users. And in order to do that, they put it to some of the population of these 2 billion users across the globe, collect the results of the implementation they’ve done in that field trial, in that A/B test, and then decide how to move forward. Do they roll it out to everyone or just shut it down and move to something else? 

Now, what I want to share with you is the fact that field trials are a kind of a pain, for us developers at least. And for that, I want to show you something. Okay. And the question is, you know, what can possibly go wrong with a field trial? And this is something that happened a few years back, it’s taken from discuss WebRTC and I’ve seen it with a few clients of mine. Someone comes and complains, well, I see something weird in the SDP now. Instead of an IP address, I see this random hash value with a dot local in the end of it. It’s great, but it kills my media server. I had a client that had general servers and he found that for some users, he’s got this thing within his SDP and that caused the sessions not to get connected for him. And when you go and check why, that was mDNS, which was an attempt by Google to clamp the privacy issue around the private IP address, that needs to be exchanged in WebRTC. And replacing that with a main DNS address. It worked nicely at the end, so it stayed there. But it broke quite a bit of services. And that happened between Chrome 73 to Chrome 76, where the official release actually went out. It went out with no getUserMedia() so this was for receive only, but it might be everywhere, somewhere in the future. 

So this is something that Google started with an experiment, kind of a field trial. They told no one or maybe they did somewhere, in some web page on their website. And then you got your clients complaining about not being able to connect, because they tried out mDNS which is a good thing, just not in the way that it happened.

Now, we had the recent thing as well, which caused us to actually come and record this specific session with you, this fiddle. Phillip, do you want to go over this one?

Philipp: Yes. So Facebook found an issue with a field trial named WebRTC-Audio-OpusAvoidNoisePumpingDuringDtx. And that caused a very large delay between audio and video for them. And we see that in the graph, we can see the delay between audio and video increases linearly by 80 milliseconds per second, which is huge. And the problem really is you cannot see from the statistics of this field trial is enabled. And you need to know that it’s unable to reproduce the issue. Because if it happens to some users who are on the field trial, but you are not on the field trial, you’ll have trouble reproducing the issue locally.

Tsahi: So I think the problem is that there’s an A/B test going on but you’re not the owner of the A/B test. You’re not Google going and checking exactly what goes to whom. But you’re the one affected because users are going to complain to you and not to Google, about that field trial. 

Philipp: Yes. 

Tsahi: And what we’re trying to achieve now is understand how to, or we’re trying to check how do you understand what field trials you’re on because everyone in the population is on some kind of field trial. At any given point in time with Google.

Philipp: We have the chrome version page, which has a special parameter show variations command. And in addition to the normal variations that you see here, and that only Google can be code, it will also give you a decoded variant of it. It will give you the command lines that you can use to reproduce a current set of field tracks in that browser, and that is incredibly helpful. 

Tsahi: I need to actually go into that specific browser, open a new tab, look at the chrome version show variations command and then look at the command line variations.

Philipp: Yes. And if you look at that, it’s not that nice. So we have a bit of JavaScript that makes it much, much nicer. So it takes the text content of that element, creates a map and then split the string appropriately. So first, it splits by the space –force– and things. And then all the field trials are separated by a slash. And if we do that, we get a map with 192 entries. So 192 experiments are currently running in my Chrome browser. And we can see most of them are not WebRTC related even though we have this audio work that real time priority thing, which is kind of WebRTC. It’s more work audio, we have what I found earlier today, a gUM focus study, which might be related to the changes and get using your focus behavior. And then we have further down two blocks, which are relevant for WebRTC. The first is pretty small. We have RTC, disallow, plan B, the outset deprecation trial, which is used to disable plan B in Chrome Canary. And we have also the RTC video decoder stream study, which I don’t even know what it is about. I’ll say there is no information in the Chromium source code. 

Tsahi: Okay. 

Philipp: And we have present down a lot of things starting with WebRTC. And these are the ones that are really interesting for WebRTC developers. In the past weeks, we’ve seen the noise pumping DTX flags there, we currently see for example, WebRTC, Data Channel, dcSCTP, which is enabled. So Google is trying out new dcSCTP library in Canary, and trying to see how it behaves in the field.

Tsahi: So at the moment running more than 10 different field trials related to WebRTC. This might affect your application.

Philipp: Yes, there is no way to opt out of them really, as a developer.

Tsahi: There is some kind of way to at least understand what is running in your browser in terms of a field trials run.

Philipp: Yes. And if you see weird behaviors that you can’t reproduce, asking for the variations command and see if the field trials are in the browser is a good way to debug things.

Tsahi: Okay. So what we’ve seen that we have field trials in Chrome that are going to affect WebRTC, and within these field trials, you can go and check into your browser, which field trials are running and try to deduce from there if there is anything that is affecting the issues that you are seeing with your users. Okay, thank you, and see you in our next fiddle, next month. Bye.