Submitting a WebRTC bug
🎻 November 2020 fiddle
Want to learn more about WebRTC?
Look us up here for the WebRTC Fiddle of the Month, created… once a month.
Or just enroll to one of our excellent WebRTC training courses.
Resources
Transcription
Tsahi: Hi, and welcome to the WebRTC monthly Fiddle. This time what we’re going to do is to explain how to exactly submit to bug to Chrome.
Before I begin, I want to show you how that feels on both ends, because filing a bug can be frustrating for everyone. Not only for you.
So let’s start.
For me, this is how it looks, what we’re in is a kind of ping pong game. I’m going to start: I have got the bug.
I am sure that this is in Chrome. They’ve done stupid things with WebRTC. So I’m going to just go file a bug and say: “you have a problem. My application doesn’t work.”
Philipp: Yes, and then if I where the Google Chrome developer, the Firefox or the Safari developer, I would ask: “OK, why do you think it’s a bug on my end? Why is it not the bug in your application”.
Tsahi: You threw the ball back at me and, well, definitely it is on your end, because if I do A, B and C and D and E, that turns into a crash.
Philipp: OK, can you please provide a sample page so I can take a look?
Tsahi: This is the URL. I also created for you a username and password. Two of them, because you need to also join from another machine with another user. After you log in, you need to go and assign yourself a specific role. And on the other side you need to call and then you need to go back to that one, press the accept button, and then you need to mute on this end. And there are a certain set of steps that you need to take in order to get to that point where things don’t work.
Philipp: You expect me to follow these very complicated steps to understand your application and why should I spend so much time on that?
Tsahi: Because there is a bug.
Philipp: Well, if you haven’t proven that, it’s a bug in Chrome.
Tsahi: OK, how do I do that, how do I help you in understanding that? Here’s a webrtc-internals dump on when the bug happens, I just publish them and pushed them into the bug.
Philipp: Well. That’s a great time, but what am I supposed to see?
Tsahi: You see that it doesn’t work.
Philipp: OK, but did it ever work?
Tsahi: What do you mean? It doesn’t work now.
Philipp: Well, does it work in an older version of Chrome?
Tsahi: Yes, we tried it in the previous release, and it now runs well. So this is a new thing that is breaking for us.
Philipp: OK, can you identify the version in which it is working and identify the version in which it is broken?
Tsahi: 85, 86.
Philipp: OK, can you do a bisect, which basically means taking the Chrome version 85 and version 86 and then you need to try to narrow down the number of changes to a single one.
Tsahi: What is a bisect exactly?
Philipp: A bisect is basically a binary search. It’s typically a way to identify a change in git which change the behavior somehow. You basically start with a good version and a bad version and then you go in the middle, decide if it is it good or is it bad? If it’s a good version. You continue with this binary search.
There is a great script called bisect-builds.py, which downloads chromium versions and automate the process. You basically download the version runs Chrome with that version, and then you can do the testing steps, close the browser and then say this is good or this is bad.
Tsahi: OK, so I went ahead, did the downloads. Ran the tests on my application or whatever. I got to the point where I know that the problem is, let’s say from this point of commit 44.
Philipp: At that point, I would say, OK, who made the change in commit 44? Is it WebRTC related?
If it’s related, then maybe it’s time to ask the person what they think of this bug report.
Tsahi: OK, but that’s internal to Google, I don’t really care, I just want this bug fix.
Philipp: Well, if you want to have this bug fixed, maybe you should spend some effort.
Tsahi: But I already did. I ran this bisect and I know this is commit 44.
Philipp: Well, all the Google people are very busy. So why should it be a problem for them to fix?
Tsahi: So how can I help them out? Because at the end of the day, I want my bug fixed.
Philipp: So the thing is, you should provide very easy steps to run. And let’s take a look at that.
Tsahi” OK, so that would be doing something like what we’re doing every month here in kind of a jsFiddle, right?
Philipp: Yes. OK, so let me share my screen. So we were looking at a bug Twilio filed, and after some back and forth we talked about earlier, they came up with this demo.
Basically they had this demo, if I open the console. It says, OK, step one, addTrack, then 10 steps and it says step 10 fails in Chrome 86. It works in Chrome 85. And that is 10 steps and it’s a lot of manual action.
Like change the port to zero. That’s manual that isn’t very easy to automate.
So what I did in this bug, I went ahead and made it into a jsFiddle.
Tsahi: We started with the bug, that Twilio found and filed: “we have a problem”. This is what you need to do. Then they worked on the bisect and even came up with the actual steps that causes the problem. But you’re saying that this isn’t enough.
Tsahi: And the reason it’s not enough is because if you’re from Google and you want to go and follow the steps, that’s still a lot of work. Then if I’m Google, I still don’t know if this is my bug or you are just messing with me with your own problems.
Philipp: Exactly.
Tsahi: OK, so I need to show you an effort to make the time that you need to spend and the work that you need to do as simple as possible.
If I want my bug to be fixed, get prioritized and for the guys at Google to not be pissed off at me, which if they mean it or not, would mean that I would get less priority with them anyway. OK, so we did two things.
The first one I did bisect
The second one I tried to simplify
The third step with jsFiddle would be to automate that.
Philipp: Yes. To minimize the problem. I mean, what I did here was basically to boil it down to two button clicks: start and renegotiate.
And renegotiate shows the problem. So that’s from 10 steps down to two.
Tsahi: OK. And then you just go and say, here is the jsFiddle that shows the problem, it’s very small and you need to start, negotiate, and then it shows the problem.
Philipp: Yes. Then in this case, it led to the bug being accepted that it got fixed. And that it got fixed before Chrome 85 went stable and was set back on Chrome 86.
Tsahi: OK, but the thing is this. If I have a bug. I need to show and explain why this is not my bug but rather Chrome’s bug. I need to show proof for that. And I need to make sure that the use case that shows that is as simple as possible so that someone in the other end will be even happy to go and troubleshoot and fix that bug for me.
But – even if I do all of these things, there is no guarantee that someone will actually go and fix that, because then it depends on their own priorities and how complicated the bug is to fix and how much risk involves fixing that versus other things that need to happen inside Chrome.
Philipp: Yes, I mean, if you can show that it is going to be broken for everyone, it will get a high priority. If it’s just broken for you, how important are you to them?
Tsahi: OK, so if I’m going to file a bug I need to reduce all of the crap of my application that is going to add friction to the one that is going to try to fix that bug. I should probably bisect to show where the problem started appearing, assuming this is a new problem and most problems are new today. And it would be nice if I can give it as jsFiddle that is independent from my application altogether, because that shows that the problem is not me.
Philipp: Hmm. Exactly, and the best strategy is to really make the other person want to fix your bug and make it joyful for them.
Tsahi: So I need to an effort because he needs to put an effort as well.
Philipp: Yes.
Tsahi: OK! Thank you for being with us this month and wait for our next fiddle next month.