Beyond the Metrics: How to Be Data-Informed, Not Data-Blind

Swati Goyal at UXDX Community: Beyond Data: Crafting Products People Love. Video: https://youtu.be/r1J-F9jd_lc

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why data matters at Meta

[00:00:07] Thank you for the opportunity, UXDX. I've been at Meta for over four years, and along the way I have learned how to be data-informed and not data-blind. In this session I'll be sharing some of the insights and learnings through real-life case studies at Meta. Quickly about me and what I do at Meta: I build products supporting content creators and businesses. Creators use our tools to produce great content, which in turn helps them make a living on Meta platforms. Businesses use our tools to connect with creators for influencer marketing, doing brand partnerships, running successful campaigns, and generating more leads from Meta platforms. So we are really helping millions of creators and businesses thrive and make a living on our Meta platforms.

[00:00:52] Just to put things in perspective, before we go deeper into how to be data-informed, let's understand why data is so important at Meta. There are three main reasons. Number one: with massive scale comes massive responsibility. We have over 8 billion monthly active users worldwide. We are the biggest social fabric connecting the world, which means that across the world, billions of people rely on our platforms for a variety of different use cases. There are users who come to our platforms to connect with friends and family. There are users who come to our platform to run their business.

[00:01:28] However, at this scale, even minor changes can have far-reaching consequences. To mitigate this risk, we use data to rigorously evaluate the impact of potential changes before rolling them out more broadly. By doing so, we can protect the user experience and prevent unintended changes from affecting our global community.

[00:01:50] Number two: building for a diverse user base presents both an opportunity and a challenge. Just for reference, our user base is spread across 190 countries worldwide. This user base is very diverse, comprising people from different demographics who use our platforms for different purposes. With this diversity we have both an opportunity and a challenge, as we strive to create products that are relevant and effective, meeting the unique needs of each segment. Again, here data helps us make the right decisions, helping us test, learn and iterate in a methodical fashion, where we understand the distinct needs of our users who are spread across the world.

[00:02:30] And then third: our product ecosystem is very complex and interconnected across our family of apps. We have Facebook, Instagram, WhatsApp, Threads, which means that a lot of these features interact with each other, and making a change on one feature can have consequences for another and potentially cannibalize the usage. So again, it becomes very important that we use data to mitigate this risk, and isolate and evaluate specific changes within the ecosystem, ensuring that improvements in one area do not come at the expense of another.

[00:03:03] Now that we understand the importance of data at Meta: A/B testing, which you all might have heard of, is a very, very important tool for us here at Meta to make data-informed decisions. Just as a quiz: how many A/B tests do you think Meta runs in a given year on average? You might guess hundreds, maybe a thousand or thousands. It's actually 200,000 tests on average per year. At any given time, we are running approximately 20,000 to 30,000 tests on our Meta platforms. You might have noticed that sometimes you see a feature on Facebook that your friends don't, or maybe your friend is seeing a feature that you are not seeing. It just means that you are in a test group, and we are testing an exciting feature before we roll it out broadly to all users.

Case one: gamifying reels production

[00:03:51] With that context in place, let's talk about the tactics for how to be data-informed and not data-blind. I'm going to illustrate this through the story of Jack and Jill. I think we all know Jack and Jill from our early days, from the nursery rhyme: Jack and Jill went up the hill. But imagine that Jack and Jill became PMs at Meta, and let's see how they learned to be data-informed at Meta through real-life case studies.

[00:04:18] The first case I have here is the case of gamification. Our goal was to drive reels production. We wanted to grow the production of reels content on our platforms. To brainstorm ideas to address this goal, we conducted user research and tried to understand the motivations of creators and their pain points. What we learned was that creators want to feel recognized, and they want to be incentivized for their content creation efforts.

[00:04:46] The idea that the team came up with was: what if we gamify content production? We have seen gamification with Duolingo. We've seen gamification in actual gaming platforms. But what if we bring the same concept to content production, where we start rewarding creators with badges as they hit milestones in their content creation journey? Our hypothesis was that when creators have fun producing content, they will actually produce more content on Meta.

[00:05:10] With this hypothesis we went into A/B testing, and the A/B tests revealed that reels production increased. We did see a goal metric move in the A/B test, which was great. Also, the reels created were getting good engagement from users, which means that users were watching those reels, they were enjoying those reels, they were liking them, there were comments happening, which again is a good thing. So Jack said: great metric move, let's ship. But then Jill said: hey, let's take a step back. Let's do a deeper dive to understand why.

[00:05:45] So what did Jill do? Jill gathered qualitative feedback from users on the feature. She also scanned social media to see what creators were saying in general about the feature. These were Jill's findings. Number one: creators are feeling burnout and constant pressure to produce. Creators have lives; life happens. But imagine if you have constant pressure to produce content every single day. That's going to burn you out and introduce stress and anxiety. While this was exciting in the short term and moved metrics, what it meant was that in the long term it would negatively impact creators' perception of Meta and would ultimately churn creators from the platform, which is what we did not want.

[00:06:25] So what did Jill do with the findings? She adjusted the product so that it became more of an enjoyable experience than a stress-based, milestone-based experience. She removed penalties for streaks, which means that if you miss a streak, you don't get penalized; you start from where you left off. She also replaced leaderboards with tiered progress and personal goals. What this meant was that we empowered creators to set their goals, and based on that set their journey and personalized milestones, so each creator could choose what their journey would look like. And lastly, she introduced sentiment tracking and qualitative feedback as ongoing parameters and success metrics to measure the product. The result was that creator sentiment improved, and they felt more in control and more aligned with their values, ensuring long-term value for the platform and creators.

Case two: automatically converting posts to reels

[00:07:15] Great. The second case was the case of automation. Our goal was to activate non-reels creators to reels. This goes back to the time when TikTok was emerging as a strong competitor, and on Facebook a lot of our creators were still producing long-form videos and photo posts. We wanted to understand, based on user research, the biggest pain point of these novice creators, and what UX research uncovered was that these creators lacked expertise, and they also complained about how time-intensive it is to create reels. Imagine asking a creator who is producing feed photo posts to suddenly create reels. It is time-intensive, and it is intimidating to switch to a new content format.

[00:08:01] The idea the team came up with was that we would onboard these novice creators to reels by repurposing their existing content into reels. For example, if they just published a feed photo post, we could automatically convert that to a reel and show them the power of short-form videos. Our hypothesis was that by reducing the friction in producing reels, we could activate non-reels creators to reels, and eventually they would pick up the habit, get a feel for it, and start producing more reels organically.

[00:08:37] With this hypothesis we went into an A/B test, and the A/B test revealed an increase in production from these non-reels creators, which again was great. Jack was excited: metric move, let's ship. But then Jill again took a step back and said: let's do a deeper dive to understand why.

[00:08:55] So what did Jill do here? Jill conducted manual data analysis to understand what kind of users were adopting the product. Jill manually reviewed the quality of the reels produced. She wanted to understand: great that creators are producing reels, but what is the quality of these reels? Is this engaging content that our consumers would enjoy on the Facebook platform? Jill also gathered qualitative feedback from users.

[00:09:20] Our findings were, one, we were mainly activating early-stage creators who have fewer than 1,000 followers. Second, the quality of the reels was low, for two main reasons. One, we were activating low-quality content creators who generally produce spammy, low-quality content. If you looked at their historical content production, they were producing content with minimal effort, not investing much time and creativity into producing engaging content. And we were also converting content that wouldn't make good reels to begin with. Imagine somebody posted a picture of tomatoes on their farm and we converted that into a reel. That wouldn't make a good reel. And then third, from the qualitative feedback, users were complaining about how spammy the experience felt when they did not want to convert their photo post to a reel but felt compelled to do so.

[00:10:17] So what did Jill do with these findings? One, she gated the feature to bigger creators who had more than 10,000 followers. This ensured that the feature was available only to quality creators. Second, she gated the feature to feed photo posts with five-plus photos. The hypothesis here was that the more pictures, the better the reel would be. A feed photo post with fewer than three pictures, for example, wouldn't make a good reel. And then third, she implemented a cool-down strategy to address spamminess. If users did not engage with the upsell, don't show that upsell to them for 30 days. This ensures that we preserve the user experience and don't come across as spammy. The result was that we took a hit on the short-term impact. Obviously, with all the gating of the feature, the impact came down, but this ensured a healthy content ecosystem and protected the user experience in the long term.

Case three: playlists and an inconclusive test

[00:11:12] Okay, the third case: the case of optimization. Here the goal was to help creators get more reach for their content, reach as in a greater number of views. The idea that we came up with was based on UX research. Most users complained about not being able to reach audiences in their niche interest area. For example, if I'm a creator in the fitness category, how do I grow my reach with audiences in the fitness segment? We identified playlists as an idea which would help creators package their content into themes, and then we could use playlists to improve their discovery with interest-based audiences. This is a very successful feature on YouTube, and it felt like a table-stakes experience we should bring to Facebook.

[00:12:00] With this hypothesis we went to A/B testing. What did the A/B test reveal? Across the board we saw non-stat-sig results. The number of views was not stat-sig, and the A/B test was inconclusive. Now Jack was disappointed and said: hey, the metric did not move, let's not ship. But then Jill said: hey, let's do a deeper dive to understand why.

[00:12:26] What did Jill do here? She conducted offline analysis to look at the product funnel, and did a cohort analysis to understand the adoption rate of playlists, what kind of users were adopting the product, and whether the users who were adopting the product were getting more views on their content. She also gathered qualitative feedback from users to supplement her understanding.

[00:12:51] Her findings were that adoption looked healthy. Creators were adopting the product, which means that they were finding value in it. And the creators who did adopt the product did get more views on their content, which means our hypothesis that playlists could be a tool to improve discovery with interest-based audiences was validated. The metrics did not move in the A/B test because adoption needs to scale to a certain level before you can see a movement in an A/B test. And qualitative feedback revealed that creators were actually raving about the feature, and they considered it a table-stakes feature, to be on par with YouTube.

[00:13:25] With these findings, Jill decided to actually ship the MVP. Even though the A/B test was inconclusive, she still decided to ship the MVP with a back test, to measure long-term impact and monitor for regressions, if any. She also brought the whole team together to brainstorm features to unlock more user value, and hence scale adoption of playlists, to actually start seeing impact over the long term.

Be a Jill, not a Jack

[00:13:53] So what is the moral of the story? The moral of the story is: be a Jill, not a Jack. In other words, be data-informed, not data-blind. I will leave you with three main tactics or thoughts as you bring some of these lessons and learnings into your product experience.

[00:14:12] One, always start with a hypothesis before testing. Start with a strong hypothesis of what you expect to happen, what user behavior you intend to change with this feature. Otherwise you might mistake noise for impact. Second, balance the what with the why. Metrics tell you what happened, but qualitative signals tell you why. Bring that whole picture together. And third, as a product manager, always trust your intuition. Not every problem has a data-backed answer. For example, for playlists the A/B test was inconclusive. When ambiguity persists, when data is not available, use your product intuition to make the right product decisions for your users.

[00:14:56] With that, I'll take questions, and I apologize for the hiccup earlier. Sometimes with a bad throat and illness, it takes a while to get set up.

Q&A

[00:15:04] Rory: No problem at all. Thank you very much, Swati, for sharing. There were three great case studies, so I hope some people out there have some interesting questions. Tiier [?] sent in a question for the last talk, so apologies we didn't get to that. But I'm going to jump in with a question, because one of the big things that I can see from my experience is that there would have been a lot of arguments pushing back in both of your first two examples. Engagement went up, interaction went up. So did you have conversations where some people were saying, oh, I think you're overblowing the risk of burnout, or I think you're overblowing the risk of low quality, and we should keep going with this because adoption is increasing?

[00:15:54] Swati: Yeah, that's a great question, Rory, and I think that's the reality of building products. As a product manager, you are just one data point, and there are so many XFNs who were behind building this product: there were designers, there were engineers, there were data scientists. As a product manager, I think it's very important to rally the entire team behind the decisions that you're making, and to make sure that every member of the team is aligned with what we are moving forward with.

[00:16:23] Some of the strategies that I use here: again, have a very methodical approach to what you are proposing. If you have a clear hypothesis, and then very systematically you bring in the user research quotes, for example, that adds evidence for why you think users are burning out. When you start putting qualitative numbers and evidence behind what you're proposing, it suddenly builds a lot of credibility amongst all the XFNs. So make sure that you bring evidence along. And also, data is great, but also roll up your sleeves and get into the weeds yourself. For example, in that case we actually dogfooded the reels ourselves. I was actually looking at the reels produced, and looking at the creator profiles to see what they had historically been producing.

[00:17:14] So when you get all this evidence, and also articulate how this is actually for the good of the long term, how it aligns with the long-term vision, because the way XFNs come together is that we all want the good for the product. Even though short-term impact might be compromised, we all agree that yes, we do want the product to succeed in the long term. We want our users to continue using the product in the long term. That's when everybody comes on board.

[00:17:44] And I'd also say it's not a one-way door. You can always come back to the decision. If we think that gating this feature did not help, then we can always unlock the gate. We can always undo the decision. For example, for playlists we rolled it out with a back test, so if my hypothesis that playlists are an impactful feature was wrong, we could always unship it. At Meta we are not hesitant to unship a product if we have to.

[00:18:15] Rory: Okay, that's great. It's the two-door decision-making: there's door one, irreversible, but most decisions are door two, reversible decisions. How do you balance as well, in your third example, where you were talking about the A/B test being inconclusive, so you decided to do a bit more qualitative research on it? How do you balance saying, okay, we're going to continue investing and doing more research, versus, you know what, that was actually a bad idea? Knowing when to cut your losses versus doubling down, I guess.

[00:18:51] Swati: Yeah. Again, this depends on the product, how much investment you've already made, and also how much evidence you had to support your hypothesis. For example, for playlists we had strong evidence from the industry: YouTube has used playlists forever, and it is a successful feature. Product intuition: even as a consumer, I think playlists are amazing. And then of course our user research also confirmed that creators do want this feature. So we had strong evidence going in that this feature should work. That's one reason why we were hesitant to just rely on the data and say, hey, let's not ship it, because strong evidence was there from the get-go.

[00:19:36] And second, an MVP is an MVP. It's just the little investment you make to validate your early hypothesis. I am typically hesitant to cut a product at the MVP stage itself, because it's too early to unlock the full potential of the product and grow its adoption. So if the product is still early stage, I would typically make a little bit more investment, especially with the bigger segments that we want to target for the product, before we decide to drop the ball and move on.

[00:20:12] Rory: Great. So it's very contextual, depending on where you think that feature is. So if you didn't have as much evidence going in, do you think you'd be quicker to drop it?

[00:20:24] Swati: Yeah. Again, there are so many variables, Rory. For example, sometimes it's company priorities. For the last four or five years, reels was a company priority, and still is. Short-form videos were taking over. Teens, young adults, everybody wanted short-form videos and reels. So at that point, across the board, every team would be very hesitant to drop investment in reels, even if the A/B tests were inconclusive. But let's say you're working on long-form videos, which was not a company priority at that point. Then you might cut your investment and stop further investment sooner rather than later. So it also depends on where you are in the industry overall, and where the macro trends are emerging.

[00:21:11] Rory: Yeah, brilliant. Just one other thing, and it's related to the question for the previous talk: do you have tools that you use? Because 200,000 A/B tests has got to be difficult to manage, and I'm assuming lots of them are parallel, so you're potentially going to be influencing your tests with other tests.

[00:21:33] Swati: Yeah. Typically at Meta, when we start the half, each team is allocated their testing environment: these are the users you would run tests on. That helps eliminate the cross-contamination. Throughout the half, when the team is running tests, they run tests on their own audiences. That way, if we are seeing an impact from the feature, we can be assured it's coming from the change we made, versus another feature that our sister team launched. So that helps.

[00:22:10] But again, there could be external factors that could influence what you're seeing in your test. For example, right now a lot of stuff is happening with immigration, and you can start seeing a lot of content consumption around Trump or politics. It's not that we started bringing in more content about it, but because there are macro trends outside our product that are influencing the content consumption patterns on our platform. So one, we try to prevent the contamination by allocating the test audience up front.

[00:22:54] And second, for the tools that you mentioned, we use Deltoid at Meta. It's an A/B testing tool that we use. Everybody in the team, engineers, data scientists and product managers, together look at Deltoid, and we bring the tests to formal experiment reviews. We look at our success metrics. We look at the guardrails, which is so important, to make sure that, okay, our metric moved, but did we regress any guardrails? The guardrails are a set of company metrics which cut across all the products, because at a company level we care about our user experience, which cuts across stories, reels, groups, Marketplace. So while we might be optimizing our local metric in the A/B test, we set our guardrails tight around the top-line company metrics that we don't want to regress.

[00:23:42] Rory: Brilliant. And do you have any minimum that you would consider a valid test, like your audience size? But I guess if you've got 8 billion monthly active users, you can probably come up with pretty large segment sizes.

[00:23:56] Swati: Yeah. We typically don't have that challenge. You're right, for an A/B test to be statistically significant, you need to have a decent audience size. Luckily at Meta, at our scale, we don't have that problem. But we did encounter some of that with playlists. There are some millions of creators that we opened playlists up to, but again, adoption takes time to uptick, and we were trying to measure a downstream metric. First users have to adopt playlists, and when they adopt playlists, their content gets discovered by the audiences who are interested in a particular subject, and that should improve their views. So we were trying to measure a second-order effect. That's where adoption becomes a challenge, where you have to scale it to a certain level before you can start seeing that downstream effect. But otherwise we are mostly okay. Typically, anything with thousands of users, we should be able to see good stat-sig results.

[00:25:01] Rory: Brilliant. Well, that actually brings us pretty much to time. So thank you, Swati, for sharing, and thank you for overcoming your illness and sharing your insights with everybody.

[00:25:10] Swati: Yeah, I was worried whether I would be able to recover, because once it hits, it's just so hard to come back, but I'm glad it cooperated. Thank you so much for giving me the opportunity. It was great presenting here.