3 Takeaways From Duolingo's Product Culture

Zan Gilani at UXDX Europe. Video: https://youtu.be/ug4v7HUaAMc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

What Duolingo is and why it works

[00:00:05] Okay, hi everyone. My name is Zan Gilani, close enough, and I'm a product manager at Duolingo. The topic for this talk is slightly different from what was presented. I'm going to be talking about three takeaways from Duolingo's product culture that hopefully you can apply to the projects or products that you're working on. But before I do that, just a quick summary of what Duolingo is and what we're about. How many people here, by the way, have heard of Duolingo before? Okay. And how many people have actually tried it? Okay, cool, very cool. So this shouldn't take too long, then.

[00:00:45] Duolingo is an app that teaches languages. We teach languages for free, and we have 30-plus languages on the app right now. One of them is actually Irish, and Irish is pretty special to Duolingo, especially in our hearts, because there are a hundred thousand native speakers of Irish, but on Duolingo there are around 1 million active learners of Irish. In fact, last year the Irish president gave an award to Duolingo for our contributions towards revitalizing the Irish language. There he is, in the center. Can I point? Yeah, there he is.

[00:01:21] In fact, since we did that, there's been a whole movement to get Scottish Gaelic onto our app as well, including this tweet from a Scottish Member of Parliament. But we haven't gotten around to them, unfortunately, if anyone in the crowd can speak Scottish Gaelic. We are very proud, though, to help revitalize languages such as Irish. In fact, later this year we're going to launch Hawaiian and Navajo courses as well. But that's actually a pretty small piece of our larger mission, which is to provide free language education to the entire world.

[00:01:56] There are over a billion people learning languages around the world, and most of them satisfy the following three criteria. First, they're learning English. The second is that they're not affluent. And the third is that they're learning English to get ahead in life, to make a better life for themselves. So our goal, hopefully, is to provide a free and high-quality way to help those people do so. And we've got a head start. We have around 300 million users. We're actually the most downloaded educational app of all time so far.

[00:02:35] So how did we get so far? Well, the first point is that we're accessible. Anyone with an internet connection can use Duolingo for free. Now, in order to pursue this mission, we have to be a sustainable business. So rather than charge for content inside the app, we have ads, we have a premium subscription called Duolingo Plus, and we have in-app purchases as well. I'm going to speak a bit about these later on, but basically every lesson in every course you can learn on Duolingo for free.

[00:03:06] The second is that we're effective. An independent study by the City University of New York found that 34 hours of Duolingo is roughly equivalent to one university semester or term of language instruction. That's because we try to help people stay motivated; that's really the gist of it. Self-directed anything is pretty hard. You can think about the difficulty of forcing yourself to go to the gym every day, but self-directed education is even more so. We try to help by making language learning a bit fun and a bit convenient, and hopefully that keeps you motivated.

[00:03:45] The other thing is that we run hundreds of A/B tests. We're actually running 173 of them at this very moment; I checked last night. That helps us make Duolingo more fun and smarter, just bit by bit, incrementally. But where do the ideas for our tests come from? That's the crux of this talk. As I think has been a theme in this entire conference, every team is strapped for resources, whether it's time, money or people. So working on the right ideas and the right experiments really is the name of the game.

Borrow from outside your industry

[00:04:19] In this talk I'm going to share three takeaways from our product culture that help us consistently come up with good ideas, and I'm going to sprinkle in some parts of our experiment process and illustrate with our experiments as well, just to give you a bigger sense of how Duolingo operates. The three points are: first, we borrow, particularly from outside our industry. The second is that we remix. We adapt these borrowed ideas, particularly to our users in their specific context. And the third is that we revisit. We take old ideas that we previously dismissed, and we revisit them when something changes, whether it's our users, our product or our hypotheses.

[00:05:06] Let's start with the first thing: borrowing. Whenever you're in your brainstorm, or your vision phase, or your discovery phase, it's very, very tempting, as I'm sure everyone knows, to look to your direct competitors for inspiration, because, one, they're probably very smart, and the second thing is they're working on exactly the same problems as you are. Sometimes this works out fantastically well; this is Instagram Stories. But generally at Duolingo this is discouraged, because, one, you can lose sight of your own product vision if you're hell-bent on copying and really chasing what others are doing. And the second is that if you are copying, you are going to end up with a product that looks no different from everything else out there.

[00:05:55] Instead, what we do is try to look outside of our own industry, which is education, and we particularly learn from the gaming world. We learn from games. Why? Because Duolingo is in the business of motivation, and some of the most effective techniques in motivation come from games. Games like Super Mario Brothers or Candy Crush do something very special, which is that when you fail a level, you immediately go back and try again and again and again until you actually succeed. That's something really important to cultivate in students, but it's pretty rare to find that in the educational context.

[00:06:38] So Duolingo, generally speaking, feels like a game, as I'm sure you've seen for yourselves. Each of these circles is a skill, which is a group of lessons, and you have to complete one skill in order to progress. Each lesson is interactive. There's no passive learning; you have to actually answer things. And we also have little touches, like a progress bar at the top to show you how much you have left in your lesson. But I want to talk about two particular features we've directly borrowed from games to illustrate this point about borrowing.

[00:07:11] The first is streaks. Streaks are pretty common, I would say. You have especially seen them in games such as first-person shooters, where you might have a kill streak, or a login bonus. The way that we do streaks at Duolingo is that you pick a daily goal first, say one lesson or two lessons per day, and for every consecutive day that you come back and meet it, your streak is extended. This has ended up being one of our most powerful retention mechanics, because, one, it gives a meaningful, relevant reason to send people notifications and emails telling them to come back that doesn't feel too spammy. And the second is that people get very, very, very attached to their streaks.

[00:07:51] The longest streak ever is 2,088 days. This guy, who has probably made history, is John Arnold, who is a chemist and horse farmer. To put this in perspective, John Arnold, the chemist and horse farmer, has done Duolingo every single day for the last five and a half years, which is inconceivable.

[00:08:11] The second feature that we directly borrowed from games was this concept of levels. A game that we love deeply and are inspired by at Duolingo is Clash Royale, the mobile free-to-play game by Supercell. Many of you might have played it. In Clash Royale, a very core mechanic is taking these different cards of characters and leveling them up, and leveling them up makes them more powerful, which you can then use in combat. We liked this for a couple of reasons. One, in terms of the mechanics: whenever you came back to the app, you had these progress bars on each of the different cards, so lots of different progress bars, and every day you're trying to fill them up more in order to level them up. It took this bigger concept of winning the game and broke it into small, discrete, manageable chunks, which we liked. The second is that you could choose your own adventure. You could decide which cards to level up in a way that made sense for your particular strategy.

[00:09:16] So we used this and applied it to Duolingo. Instead of just having a skill, which is a group of lessons, now when you complete that skill you level it up to the next level, which is these little numbers in the crowns over here, and every time you level it up, you get access to harder content. The reason we did that is because of a conundrum that we've had in all of our time working on language education, but it's bigger than us: whenever we tried to add harder content, it was pretty discouraging to our more casual learners, especially because they're learning on a phone, basically. They might be learning on the bus, they might be learning at home. But in order to actually teach a language, in order to get people more fluent, you have to give them harder content. They have to struggle with it.

[00:10:07] So this was a conundrum for us, and this allowed us to work on the problem in a way that ended up being successful, because now users could go through the app if they were casual users, but if they wanted to learn something harder, they could start leveling up. This ended up being really positive on basically all our core metrics, whether it's daily active users, day 14 retention or lessons completed.

[00:10:32] Now, just for the sake of transparency, of course borrowing doesn't always work. I want to talk about an experiment that failed horribly, and this is what we call moves. Another set of games that we were inspired by are called match-three games. I'm sure many of you have played Gardenscapes or Candy Crush. These are mobile games, and the way they work is very simple. You have all of these different objects on your screen, and whenever you match three or more of them, you collect those, and the goal is to collect a certain number within a set number of moves. These games are immensely, immensely, immensely successful.

[00:11:15] So we tried to borrow this and bring it to Duolingo with the idea of moves. Moves made it such that at the beginning of every lesson, you had a fixed number of moves, this number in the corner, and every time you answered a question, it would use up one move. The goal was to complete each lesson within that fixed number of moves. The idea with this was, one, it's a bit more of an interesting, strategic lesson experience, but also we could use what are called difficulty curves, which are very common in the gaming industry: we could vary the difficulty lesson to lesson in a way that was more interesting than just slowly ramping up difficulty over time. But as I mentioned before, this really did not work. This was either neutral or actually pretty negative on the metrics we cared about.

[00:12:04] So that's the first point: borrow, particularly from outside your own industry. And I would put in a quick plug to also just borrow from games, because they do a lot of things that the tech industry just ignores, basically.

Remix ideas for your users

[00:12:22] Now, when you have an idea that you've borrowed and it's actually successful, what do you do next? That brings us to the second point, which is to remix. When you have an idea that you've borrowed and it's successful, you then want to remix it to adapt it to your particular users and their specific needs and context. I have a couple of illustrations of this that also have to do with streaks, which I mentioned before. Streaks can be incredibly motivating when you're actually keeping them, but when you lose your streak, that is disastrous, and people get very upset. You can imagine what would happen when John Arnold, the chemist and horse farm owner, loses his streak.

[00:13:07] We looked a little deeper, and we found that it's over the weekend that people are disproportionately losing their streak, which makes sense to us. So we came up with an idea to remix streaks: the weekend amulet. The weekend amulet is something that you only see on Fridays, and when you activate it, it allows you to keep your streak over the weekend even if you don't practice. That ended up being really successful. People had longer streaks, and they ended up retaining for a longer period of time.

[00:13:42] The second way that we remixed an idea borrowed from games was with the streak repair. This is an interesting one, because for the longest time, when people lost their streaks, many would write to us saying, "We will pay you, can you please bring them back?" This was very useful to us. At the same time, we had introduced something new, which was this premium subscription that had two features. One, it took away ads, and the second is that it allowed you to learn offline. Again, we weren't blocking content; these were extra features you could get. So we decided to add this third feature, learn anytime [?]: basically, you can get a free streak repair once every month. So if you lose your streak this month, it actually gets repaired, and this is a perk of subscribing to Duolingo Plus.

[00:14:31] We liked this idea because, on one hand, users were literally writing to us saying, "We want to pay you for this," before it was even on our radar. And the second is that it didn't kill the integrity of the streak. The streak is meant to be a motivational tool, and it would not have any value if you could revive it at any time without any price, monetary or otherwise. We iterated on this a bunch of times, to the point where now the streak repair, as it stands, contributes 9 percent of our daily revenue. The crying Duo also really helps.

[00:15:06] So you borrow ideas from outside your industry, not from your competitors, and then you remix them to suit your users' needs and context. Streaks existed everywhere, but no one had really done anything like a weekend amulet or a streak repair before, and that made sense for our users.

Revisit ideas when something changes

[00:15:25] These are externally facing ideas, but sometimes the best ideas come from within the company, within the team. That brings us to the third and final point, which is to revisit: revisit ideas that have already been attempted, because something changes. This is something you're probably very familiar with: the scenario where someone, maybe it's you, comes up with an idea in a team meeting, and the most senior team member goes ahead and dismisses it, saying, "Oh, we tried that before, it didn't work, there's no point in doing it again."

[00:15:57] First off, you should recognize HiPPOs. These are the highest paid person's opinion. Even if they look like elephants, you should recognize that they are HiPPOs, because they are pretty insidious, and they're going to lead to all of these ideas that you would have considered being dismissed for good, because the highest-paid person in the room decided that it wasn't a good idea.

[00:16:22] But there are plenty of good reasons why you should revisit ideas. The first is that your users could be completely different now than when you first attempted something. Andrew Chen, who is this guru of all things growth, talks about the law of shitty click-throughs, which basically says tactics decay over time, usually because user behavior actually changes. Banner ads today have a click-through rate of about 0.05 percent, but the first banner ads had click-through rates of 78%. So people really change in how they use your product.

[00:17:00] The second is that your product itself could be different. You can imagine that when you're first making an app, it's very simple, and an onboarding experience would actually hurt, because it adds one extra step before users get to the good stuff. But over time your app could become much more complex, and suddenly you need an onboarding experience, because you need to drive users directly to the thing that's most valuable to them just as they're starting out. And third, your hypotheses could be different. You could just have more experience, or a different take on why something didn't work, and also a new take on what the right way to iterate on it is.

[00:17:33] I want to illustrate this with achievement badges, which are a part of Duolingo, and I think people are probably familiar with achievement badges. For the longest time, we hadn't added them to Duolingo because we tried them once before, they didn't work, so there was no point in trying again. But when we dug deeper into that first experiment, things didn't make sense to us a year later, when we were more experienced. That's because this experiment to validate achievement badges, and this was basically the minimum viable test we did, only showed you a screen at the end of your registration process saying, "Congratulations for registering."

[00:18:13] Of course, now we know that that's not how achievements work. That's not what makes them effective. In order to have effective achievements, one, you need to have a bunch of them, and they have to be for meaningful actions. Registration is super meaningful to us, but not meaningful at all to a user who's trying to learn a new language. And the second is that you want to show them all together in a nice way, so that you're preening over the ones that you've already earned and coveting the ones that you haven't earned yet. That's what makes effective achievement badges. So when we ran this experiment, and this is what achievements look like in Duolingo today, it ended up having a really positive impact on our core metrics.

Revisiting referrals

[00:18:58] Finally, I just want to end with an experiment and a feature that we're actually working on right now in my team, which is the acquisition team. This is something that we've tried a bunch of times over the last couple of years. Duolingo doesn't pay to acquire users. All of our growth comes from word of mouth, basically, and some PR. But we're always trying to optimize and accelerate this organic growth channel that we have. So we've long wanted to master referral. This is something that you've probably seen in many apps, like Deliveroo [?], Dropbox, Uber: you refer a friend, they sign up, and you both get something, whether it's free space, free food, rides, whatever it is. So we wanted to try this as well.

[00:19:53] This first experiment, from a couple of years ago, was kind of like the first achievement experiment. At the end of the registration process, we just said, "Hey, invite your Facebook friends to Duolingo," and there was a button to invite them. This got us a few hundred installs per day, but Duolingo gets several orders of magnitude more app installs per day just through organic means, so this wasn't going to meet our mark. The reason for this seemed obvious to us: we didn't actually have something to offer. Unlike Uber, which gives you $20 for your free ride, we didn't have anything to offer at the time within the app. We were literally just asking people to refer their friends.

[00:20:38] Fast forward a year, and now we have a new referral experiment, and this time we have something to offer users, because our product has changed. This time it's gems, which is the in-app currency that we have, that you can now spend real dollars to purchase. So we thought, okay, now we're going to offer this. It has monetary value, it's a good incentive, people will invite their friends and they're going to earn gems. But yet again, this didn't work out. It was a little better than the first experiment, but again nowhere near where we wanted it to be.

[00:21:06] This led us to a new hypothesis. We knew two things. The first is that people are most likely to refer their friends right at the beginning of their journey. It's basically within the first day that most people want to refer others, because they're excited about the product. The second is that gems take a few days for you to understand as a user. They're not immediately valuable or useful to you.

[00:21:33] Fast forward one more year, and we are having a retrospective. This is something that you might be doing at your own companies, but at Duolingo, retrospectives are a time when a team will present on a previous project that either completely exceeded expectations or fell short of them. They're pretty useful because they take the siloed information from that one team's head and disseminate it to others, so that they can either not make the same mistakes, or if they're going to work on the same project, they have a big head start. It's during this retrospective that we realized, okay, the product has changed yet again. We now have a reward that is immediately meaningful to users, and that's Duolingo Plus. So this experiment is actually running today on Android: it offers Duolingo Plus for free for every person that you invite who signs up. Hopefully it works out. If you have any ideas about this, please tweet at me, or see me after the presentation, and let us know what we should be doing.

Summary and a story about Irish

[00:22:37] So that's revisiting. Just to summarize one more time: borrow, but borrow from outside your own industry. Remix, and make it relevant to your users and their particular needs. And revisit ideas that have previously been dismissed. Don't take no for an answer. Dig deeper, read the experiment report if it exists, and see if things have changed, whether it's the users, the product itself or the hypotheses that you have.

[00:23:11] I started with a story about Irish, so it feels right to end with a story about Irish as well. This is Shannon O'Neill, who was studying music education in LA. A couple of years ago, her health, her health and cognitive abilities, began to precipitously decline. She had viral meningitis. During this time, because she was a fifth-generation Irish American, she thought, okay, I'll try Duolingo and I'm going to choose Irish. What started as just a way to work on her short-term memory and pass the time ended up becoming a full-blown passion. She got better, she's healthy now, and she ended up pursuing Irish, studying it, moving to Ireland for a little bit, and now she is actually pursuing a career in teaching Irish. On this whole journey, she said, "I'm really proud to call myself an Irish speaker, and I hope to inspire others. Learning Irish has allowed me to reclaim, or to claim, my heritage." And that's the power of free language education. Thank you.