WebRTC in the Trenches – A Survival Guide

Feross Aboukhadijeh at UXDX Europe. Video: https://www.youtube.com/watch?v=KSr2XtEsMRU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Building Speakeasy: what it is and who I am

[00:00:00] Hello and welcome to my talk, Building Speakeasy: WebRTC in the Real World. Today I'm going to share with you a story of how I built an application called Speakeasy. It's a web application for meeting people online through video calling, and you can think of it like this: if you took a bar or a party and you tried to bring it to the online world, that's what Speakeasy is. My talk today is going to be the story of how I built it, why I built it, and what went right and what went wrong along the way. Hopefully there'll be some lessons in there for you if you're ever interested in building a video app or developing a product on your own. That's what we're going to talk about today, so let's get started.

[00:00:49] First off, my name is Feross, and you may know me from some of my open source work. I created the project WebTorrent, which is a streaming torrent client, and this is a cool project because it takes the BitTorrent protocol and makes it work in a web browser, which is pretty sweet. I also work on StandardJS, which is a code style linter and code enforcer for your JavaScript projects. Other than that, I've worked on hundreds of open source projects. If you build JavaScript applications, you probably have my code somewhere in your application, at some layer, coming in through an npm or JavaScript dependency somewhere in there. I've been working on software for probably half my life at this point.

[00:01:39] Let's get going. I'll show you what the application looked like when I finished prototyping it. This is the idea: you can select if you want to do a video or an audio chat, and you get matched up with a partner who the site recommends that you speak with. In this case we're given this conversation starter question, "What do you value more, intelligence or common sense?", and then we can discuss that with the person we're matched with.

[00:02:05] The idea behind the site was this. It was created at the beginning of the coronavirus lockdowns, and also in the United States we were having a bunch of protests going on, and the police actually created a curfew, so a lot of people couldn't leave the house after 8 p.m. There was a certain feeling of social isolation that I wanted to help address. I wanted to make a way for people to connect with each other and to have conversations even without leaving their homes, and to create a little bit of a sense of community. That's the idea behind the design of the site.

[00:02:48] One other aspect you'll notice is that there's a countdown timer at the top right that is counting down how much time is left in the conversation. That, along with the conversation starter question, creates a very intense conversation with the person you're connected to, because you just have two minutes. You also have this question that's encouraging you to skip past the small talk in the typical conversation. A lot of times when you meet somebody, you'll start your conversation with the basics, like where you work and where you live. In the case of this application you only have two minutes to talk to the person, so if you spend those two minutes reciting the basic information that you tell everybody that you meet, the whole conversation will be over by the time you do that.

[00:03:43] This app really encourages you to skip past the small talk, get past the basics that you've said hundreds of times before, and really get to something interesting. Then, if you do connect with the person that you're speaking with, there's this button that pops up in the last 15 seconds of the call which allows you to extend the conversation, but it only works if both people agree to extend the conversation. That's the idea of the site; hopefully that makes sense.

What went right: Simple Peer and launch day

[00:04:11] Now what I want to share is what went right in the process of building this. The first thing I'll focus on is how I built it. There's this open source library that I created called Simple Peer, and it's one of the most popular libraries for building WebRTC applications. If you're not familiar, WebRTC stands for Web Real-Time Communications, and this is the API in the web browser that allows you to do video and voice calling. Pretty much any video or voice app that you've ever used in a browser, something like Google Meet or Skype, is using WebRTC to enable that video calling or voice calling to actually work.

[00:04:56] The thing, though, is it's actually a really complicated API. There are dozens of methods you need to understand, there are a lot of acronyms, there are terms like STUN, TURN, ICE, and a whole bunch of concepts you have to understand in order to get an application like this off the ground. It turns out all those concepts are there for a reason: it's actually pretty hard to do a video call and get all the little technical pieces in place to make it work. But when you're building an application, you don't really want to be faced with all those minor details, so it really helps to pick some kind of library or framework to give you a simpler way to access that same functionality.

[00:05:42] In this case, I wrote this library, Simple Peer, when I was building WebTorrent, which is that other open source project I mentioned. I had never really used Simple Peer for video or voice calling before, but it turns out, because of the great work of some other open source contributors on the project, it was really ready to go for this video and voice calling use case. I was really pleasantly surprised how well it worked. If you're building any kind of video or voice app, I really recommend you take a look at this library, or maybe even some kind of third-party service for WebRTC calling. It will really be a great help to you.

[00:06:22] Now I want to focus on how I promoted the site on launch day. This right here is our Product Hunt listing page, and this is where I posted the site when I first created it. One thing you'll notice is the site was actually called Virus Cafe when I first created it. Now it's called Speakeasy, which I think is a much better name. The idea of the original name, Virus Cafe, was: well, we all have this virus and we can't really meet up in a real cafe to hang out with people or to meet people, so why don't we do a virtual cafe? That's where the name came from.

[00:06:54] You'll see here that when we launched it, we got a lot of votes, a lot of interest, and a lot of people trying the app out. One tip, by the way, if you're ever launching a product on Product Hunt, is that you should post it at midnight, or really a couple of minutes after midnight. The way the site does ranking is it resets the number of upvotes that all the other products have down to zero at midnight. If you submit your app right after midnight, you'll be more fairly in the running with all the other applications that are on the site, and you'll have a better shot at making it into that top list of products for the day.

[00:07:40] As you can see here, we were number four for the day. Throughout the day we were at number one and number two; we were fighting for that top position. In the end we got number four, which is a little bit disappointing, but as you can see we still got a ton of interest, people coming through to try it out, and a lot of really good feedback. I would say that the launch on Product Hunt was quite a success for us.

[00:08:01] The other thing we did is launch on Hacker News. For those of you who don't know, this is a really developer-centric place to hang out. It's a link and news sharing site, but they also have a feature where you can show products that you've built to the community, and that's what we did there. One of the things about Hacker News that's typical is there are a lot of critics in the community. You can see here, as I'm introducing the site to people, I was mentioning that the goal was to eliminate small talk and get people to really connect over a good conversation. Then the very first comment is somebody saying, "Actually, small talk is really great, and I can't believe you would want to get rid of small talk."

[00:08:43] That's just how these things are. When you put your product out there into the world, there are always going to be critics, and there are also going to be people who have something to say. It's not a reason not to launch stuff, and it's not a reason not to create stuff. You've just got to take it and roll with it.

User feedback and moderation

[00:08:58] Then on Twitter, of course, there was quite a lot of feedback. People were really happy with the app, they thought it was amazing, and they wanted to share it with people. You can see this guy said that it was a much-needed replacement for talking to his colleagues in between breaks. As he transitions to remote work, having a place where you can actually hang out and socialize with people was really valuable. This person was saying that they had really wholesome conversations on the site, which is really great.

[00:09:30] The other thing that was interesting is people would spend more time talking with others than they intended to. They would come in thinking, "I'm going to check this app out for a couple of minutes," but then they would get really addicted to this two-minute conversation where you get matched to a stranger, and at the end it disconnects and you get matched to somebody else. It was this really interesting opportunity to just meet all kinds of different people from around the world, and some people really spent hours and hours on the app. One user in the first week actually spent 17 hours talking to people on this app, which is pretty wild.

[00:10:03] This is the kind of feedback I was getting. Here's another one: someone said they made a friend in Paris, so people were making friends all over the world. This person said they made two really good friends in different parts of the world within an hour, which is really great. Right here, this person met a bunch of strangers and never realized they were going to spend this weekend just chatting with people on the internet, and they had a lot of fun.

[00:10:30] This story here was actually my favorite story from a user of the app. This person said that they met a German girl on the app and they had a really great time chatting, and right when they were exchanging telephone numbers, the server restarted. This was actually my fault. Any time I deploy a new version of the site, I built it in a way where it forces all the calls on the site to end for a quick second there while the server restarts. It's not the best design, but one thing I did is I sent a little message to the users on the app that says, "Hey there, please wrap up your call, because the server is about to restart in 10 seconds."

[00:11:13] In this case these two people were trying to exchange their telephone numbers really quickly before the call ended, and the server restarted and they ran out of time, so this guy was complaining about it. Then, lo and behold, the German girl that he was trying to talk to comes into the comment section and says, "Hi, it's me, I just sent you an email, hopefully we can stay in touch." I thought that was a really great story, where they actually met in the end. This is one of the fun things about working on social apps: you're helping humans connect.

[00:11:45] One thing that you might be thinking at this point is, "This is great, all these great interactions are happening, but what about the bad actors? What about the malicious people? Won't there be a moderation problem with a site like this?" That was actually my concern as well when building this. I was really worried that people would come onto the site and misbehave and create a really bad experience for people. One of the first things I did when I was building this is I built a moderation dashboard, or control panel.

[00:12:18] I'll just show you a little screenshot of me testing it out, so you can get an idea. On the left is the moderation dashboard, and what I can do is click on a little message and click send. Then you'll see, over on the right-hand side of the screen, that's the user, and they're going to get this little message from the admin saying, "Please behave appropriately; this app is for friendly people." I could also send custom messages to people, to teach them or train them how to behave on the site correctly. I had all this built and ready to go because I was really worried about this problem of users not being aware of the cultural norms I wanted to create around the site.

[00:12:59] Fortunately, I actually never really had to use this too much. Nobody was behaving in a really terrible way, so I was really pleased that that was the case. But one really cool thing came out of building this feature: I now had the ability to make these little thumbnails of anyone who's talking on the site, and this actually made its way into the public-facing product. Now when you go to a Speakeasy, you can see all the people that are at the event and that are talking to each other. You see all the thumbnails of the other calls that are going on, even the ones that you're not part of.

[00:13:37] What's great about this is it gives you the feeling that you're at a party or you're at an event. Even if you're talking to one person, you have this sense of, over your shoulder, to the left and to the right, you can see all the other people that are chatting in your peripheral vision, which is really cool.

What went wrong: iOS bugs

[00:13:57] Now I'm going to focus on some things that went wrong in building this, some things that could have been better. I apologize to the product people in the audience, but this is going to be a pretty developer-centric part of the talk, because the stuff that went wrong is mostly coding related. It's mostly to do with iOS bugs. There were a whole lot of bugs on iOS in particular. It turns out Safari was one of the last browsers to add WebRTC support, and even though it's been a couple of years now that you've technically been able to do this kind of video calling on Safari, they still have quite a lot of bugs that they've yet to iron out, in particular on iOS.

[00:14:45] I'm just going to give you a little taste of the suffering that I had to go through in dealing with these bugs in building the product. Hopefully, if you're building a video or voice app of your own, some of the issues I mention now will save you the weeks of debugging time that I had to spend figuring out these bugs. Without further ado, let's get into it.

[00:15:10] This one was really a joy. 10% of the time, audio would be muted for one end of the WebRTC call. As you can imagine, having your video calling app fail 10% of the time randomly would be a terrible user experience. People were really confused why their call wouldn't connect 10% of the time. Fortunately this was fixed in iOS 13.6, so this is not a problem anymore, but I'll share with you how I managed to fix this before it was fixed in iOS.

[00:15:44] The way I figured out the solution was really strange. I opened the web inspector, I selected the video element, and then I typed pause and play into the console, and it turns out if you pause the video and you play the video, suddenly the video starts to work. Completely inexplicable. But if you write this code, you can hack around the problem. What this code is saying is: when the video starts to play, go ahead and pause it, and then after it's been paused, make it play again. This seems to fix this problem of the video not working 10% of the time. Extremely gross code, but it turns out it fixes the problem.

[00:16:31] Cool, so here's iOS bug number two. There's this API called getUserMedia, which allows you to get control of the user's camera and microphone so that you can build these kinds of applications. On iOS, if you call this function twice, it actually breaks the video stream that you got from the first time you called it. This causes a lot of problems if you're trying to build a WebRTC application, and unfortunately this seems like it was an intentional decision from Apple, because it's still an issue in the current version of iOS. There's some code here that I used to try to work around it; it's not that important. Long story short, it's a really annoying iOS issue.

[00:17:18] Let's talk about iOS bug number three. This bug is also completely hard to understand; it's really hard to understand how these kinds of bugs made it into iOS. It's just so surprising. This one is: if you take an app and you add it to the home screen, so you save the website to your home screen and now you have a little icon there, and then you open up the app from that icon, the camera and the microphone will not work. That obviously completely breaks the application. For this bug I just had to wait until Apple fixed it in iOS 13.5.1. At least that's fixed now.

[00:18:00] Here's iOS bug number four. The way to trigger it is, while you're in a video call in Safari, you switch to another application and then you switch back to Safari. What that happens to do is break your local video stream, so the video of yourself that you'd see on the app just turns to black and you can't see yourself anymore. This one was fortunately fixed in iOS 13.5, but here's the code I used to fix it in the meantime, before that fix came out from Apple. Again, this is going to be really gross code, but you can see what's going on here. I listen for visibilitychange, which is the event that fires when the application is brought into focus or sent away from focus. When that happens, I pause the video and I play the video, and that happens to also fix the bug. It's like the video tag gets stuck in some way.

[00:18:55] I just have a couple more bugs I'm going to share. I could go on and on all day about these iOS bugs, because I really have to emphasize that the amount of time I spent on these iOS bugs was at least 50% of the time building the app. I was able to prototype it in about three weeks, and then I spent more like a month or more just fixing it on iOS. That's why I'm spending so much time on these bugs in this talk: that's where the time was spent when building the app.

[00:19:25] Without further ado, let me explain what this bug is. What happened is, if you ever took your AirPods out during a call, or put them in, the audio would stop for the person on the other end, so you wouldn't be able to hear the person you're chatting with anymore. This bug seems like it's fixed in the latest iOS, iOS 14, but the WebKit bug is still open, so I'm not sure if it's fully fixed, but it seems to mostly not be a problem anymore, fortunately. Here, again, my hacky fix was: if the video freezes, when it pauses, just call play. That seems to improve the situation a bit.

[00:20:10] I'm going to throw in just a couple more bugs here. This one is if you try to get an HD 1080p video stream from the camera. What's supposed to happen in WebRTC is, when you ask for a particular video quality and that's not available because the camera happens to be lower resolution than you've asked for, the browser is supposed to give you the closest match. If I want 1080p but only, I don't know, 480p is available, then it should give me 480p without a problem. But it turns out on iOS, if you ask for 1080p, you just get an error, and the highest that you can ask for without it erroring is 720p.

[00:20:48] Which is silly. There's no reason why I shouldn't be able to get the highest quality video when I'm building my application, and of course native apps on iOS don't have this problem. So this turned into a pretty disappointing bug, and the fix was of course to just have an if statement: if iOS, then we'll use one resolution, and otherwise we'll use the 1080p resolution. That's the solution.

[00:21:15] Now I'll share the very last iOS bug that I'm going to share with you today, which is to do with web views. This is actually really useful information if you don't know it already: on iOS, it turns out all the third-party browsers, Chrome, Firefox, Brave, are not actually running true Chrome, true Firefox or true Brave. Really, what these browsers are is Safari with a skin, a UI layer on top of Safari. That's because Apple doesn't allow third-party browser engines into the App Store. Chrome, Firefox and Brave are just a set of UI enhancements, but under the hood you're really running Safari. The implication of this is that whatever decisions Apple makes about how the browser is going to work apply to all of these third-party browsers.

[00:22:12] The way that these browsers are built is they use this feature called the WKWebView, and that's a way that an application can embed a Safari instance in their application. The WKWebView doesn't support the WebRTC API. Long story short, what does this mean? It means that all third-party browsers on iOS can't use WebRTC. If your users come to your site from Chrome or Firefox or Brave, then WebRTC will be absent, and unfortunately there's nothing that you can do for these users other than telling them they have to switch to Safari. That's what we do here. You can see this is what we show the user on Chrome: we say, "Sorry, you have to use Safari," which is very unfortunate.

Surprises from watching users

[00:22:57] Anyway, I could go on and on about this, but what I want to focus on now is a couple of final learnings about things that happened that I didn't expect to happen while I was building this. The first thing is the URL of the site, virus.cafe. It turns out people thought that this site was actually a virus that was going to infect their computer. I don't know why they'd think that a virus site would declare itself to be a virus in the URL, but I thought the URL was cute when I came up with this name, and it turns out that it seems to have been scaring people away from trying the app out.

[00:23:36] One of the things I did was I renamed the site to Speakeasy, as I mentioned before, which I think is a much better name. This is how the site looks and how it works now. It's called Speakeasy, and it's a lot friendlier to potential users.

[00:23:53] Let's go to the next thing. I noticed this one behavior of a couple of users which was extremely shocking to me. Those of you that have built applications before and observed your users, I'm sure you've seen all kinds of really strange, completely inexplicable user behavior that you can't explain, and this is one of those cases for me. I noticed that there were a couple of users who would use the app and get matched with a random person, but instead of talking to them, they would immediately disconnect, and continue to do so until they got matched to each other. They were friends who were using the app to find a way to match with each other, and once they did that they would just talk for hours and hours and hours and talk to nobody else.

[00:24:38] I was mystified. Why would they be doing this? I managed to get matched to them myself while using the app, and I asked them, "Why are you using the app this way?" They said, "Well, I just want to have an audio chat with my friend, and your site is the only site that lets me do that without making an account." It turns out, if you let people chat to each other without making an account, some people will value that, and they will literally force the app to be used in a way that it's not really meant for, just because of this one feature. Very surprising.

[00:25:17] The other thing that happened that was surprising is that this Saudi influencer shared the site out to all of his Twitter followers, and overnight the feeling, the culture on the site, changed. It used to be all Silicon Valley people who found the site on Product Hunt, and then suddenly I'm getting matched with people who may be speaking Arabic. That was a very surprising turn of events. I noticed this behavior that some of them were doing, which is they would block the camera with their finger. At first I was like, this seems really bad. Why would a user want to block their camera? They must be up to no good.

[00:25:54] But after getting matched to several of the people doing that, I learned that actually a lot of them were women who just didn't want to show their face to the people they were talking to. Maybe some of the conversation starter questions that the site was presenting are pretty personal, and they felt more comfortable discussing these questions with people if they weren't showing their face. It was very interesting behavior, and it actually led to me building that audio-only option that you saw earlier. It turns out, by observing your users and talking to them, you can learn a lot.

Lessons: critical mass and shared context

[00:26:28] I'm going to finish up with a couple of final ideas here, and then we'll do questions. What did I learn? One of the most important lessons from this whole experience is the importance of having critical mass when you're building an application like this. You're doing synchronous video calls, so you need to have people online to match to each other. Even if the site had thousands of people coming to it throughout the day, if they all came at different times, let's say they came one minute apart, you're going to end up in a situation where you can't actually match anyone to each other, because someone comes, there's no one online, and they leave. If you want an app like this to work, you really have to think about how you're going to get a lot of different people on the site at the same time to create this matching.

[00:27:16] The second thing is the importance of shared context. As I mentioned from that example when all the Saudis came onto my site, if you don't have any context with the people that you're getting matched with, you're just not going to have as good a conversation. Don't get me wrong, you can have a great conversation with a random person, and that can be really fun, but there's nothing that can replace having some kind of context with the person that you're talking to, knowing why you were matched and what you're both interested in, because it makes the conversation flow a lot better.

[00:27:47] That's actually why the current version of the site, Speakeasy, is based around events instead of just matching random people. The way it works is you make an event, and then you as a host invite the people that you want to come to the event, and now there's some context around it. In this example here, we're actually making an event for working out. The reason I chose this as an example is it demonstrates the flexibility of the tool: you can create all kinds of different events using the tools that Speakeasy has. In this case we're making a Speakeasy for doing 60-second workouts with people that you don't know.

[00:28:32] You can see here we're going to set the dark mode to make it look cooler, and we're going to set the length of the chats to 60 seconds, so that we do a 60-second workout with the person we're matched with, and then we're going to automatically get matched to somebody else to do a different workout. We're going to disable the ability to extend the chat, so it's forced to be 60 seconds. Lastly, we're going to set up the prompt that the two users see when they get connected to be one of the workouts from this list here. Maybe I'll get matched to somebody and it'll say, "Do a plank for 60 seconds with them."

[00:29:09] Once I've done that, I'll just hit the back button, and you can see that just like that, with a couple of clicks, we were able to create a completely new video app experience for working out with strangers in 60-second segments. That's how Speakeasy works.

Closing: experimenting with video formats

[00:29:27] What I want to emphasize is that building WebRTC apps is extremely fun. WebRTC lets you build stuff with video and voice, and especially in the current environment that we're in with the coronavirus pandemic, I'm so excited to see that one of the silver linings from this is all this interesting experimentation going on with video and voice meeting formats. There are so many apps now where you can meet in all kinds of different, interesting ways, and people feel really empowered to experiment with the format. I feel really confident that at the end of this we're going to see some new video format emerge that's really interesting and really exciting, and that will change the way that we do meetings online.

[00:30:09] To that point, one of the socializing platforms that UXDX is offering to you as an attendee is Speakeasy. You can go and use Speakeasy and meet with people and check out the app for yourself. If that sounds interesting at all, make sure to look for the Speakeasy option, select it, and go forth and meet other UXDX attendees using the platform. Hopefully this talk today will have given you more insight into how and why I built the site, and hopefully that makes it all the more fun when you're using it to talk to people from UXDX. With that, thank you very much for coming to my talk, and I hope that you enjoy the rest of UXDX.