How To Maintain User Focus In A Metric-Driven Environment

Stijn Eversdijk at UXDX Community: Amsterdam. Video: https://youtu.be/wM9nS1gm5oE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Metrics, vision and the sensible middle ground

[00:00:11] Stijn: Hi everyone, I'm Stijn. I work as a product designer at WeTransfer. This is one of the first times speaking at a conference like this for me, so please bear with me. Closer? Okay. They asked me to come and speak today about how we work at WeTransfer, how we create products and how we struggle with that from time to time. The topic was fairly open, so I thought about it over the last couple of weeks, and what kept coming up for me was metrics: how we work with metrics as a product development team, as product designers, as a company as a whole, and how that impacts my day to day. I've seen colleagues get very frustrated by a lack of metrics, or by a lack of vision, or the complete opposite, the metric fixation, or only having to rely on one person with a vision. I believe there is a sensible middle ground, so I would like to talk to you about that today.

[00:01:34] I work as a senior product designer. I joined WeTransfer around three years ago, and before that I used to work at a couple of small design studios here in Amsterdam and in Berlin. WeTransfer is mainly known as a service to send big files from A to B, and we've been doing that since 2009. Our biggest audience is in the creative sector, and we believe that creativity brings positive change.

[00:02:06] Over the course of the last year we've begun building a suite of tools to cater to different parts of the creative process. You have WeTransfer.com, which is for sending those big files at the end of your process. We've got Collect, which is for saving content from different apps into one place. We've got Paste, which allows you to make simple presentations. There's WePresent, which is a platform where we write about interesting stuff that people do. And we've got Paper, which is an iOS app that allows you to draw and sketch.

[00:02:58] At WeTransfer I got a chance to work on both established products, with product-market fit and a very constant revenue stream, and now, for the last year or so, I've been working on Collect, which is a completely new app. It's around a year old now, and we are still looking for that product-market fit and figuring out where we go. Over the last three or four years, WeTransfer went through a lot of change and growth, and during all that I got the opportunity to experience a lot of different situations by being in this growing company. What I realized is that different contexts ask for different approaches in how you execute product design.

Balancing the intangible with the tangible

[00:03:40] It's March, the end of Q1, and Q1 is an interesting time, because often OKRs get set for the rest of the year, roadmaps get implemented, ambitious plans get kicked off, and you as a designer or developer might feel a bit overwhelmed, like you're only a small cog in the wheel. But I can assure you that your input here can be super valuable. I assume we all strive to build products that deliver value to both the user and the business. If you abstract something like making a product, what you can see is that you have to balance the intangible with more concrete and tangible parts. You've got a vision, intuition, gut feel about what you want to make and how to do that. Those intangible parts get supported, informed and often adjusted through a much more tangible side: metrics, usage data, user research. It's a fine balance and pretty hard to get right.

[00:04:57] As a designer you will find yourself right in the middle. You have to take these inputs and translate them into something super tangible, a flow, a UI, a product, something that gets out the door. You affect that so directly that you really need to take a moment and weigh what approach you take, and at what time. So I'll tell you a bit about my experience at WeTransfer and show you how we sometimes completely missed that balance between the two, what the impact was on me and my colleagues, and how we tried to restore the balance.

WeTransfer 2.0: a big seat at the table

[00:05:32] When WeTransfer started, it was a typical example of a good idea solving a relevant problem and finding a way to monetize it without impairing the user experience. On WeTransfer.com we've combined screen-filling ads with the to-the-point utility of sending large files. We have an in-house design team, and most of the wallpapers get designed by them, and we allocate 30% of the ad inventory to highlight creative work or good causes. After a couple of years WeTransfer became profitable, but you could say it was more of a marketing or sales company than something you could consider a tech or product company.

[00:06:30] In 2015 WeTransfer received investment, and it was time for a step change. WeTransfer acquired the design studio I used to work at. I decided to join as well, and we came in as the design team. We were tasked to do a big update to WeTransfer, WeTransfer 2.0, to bring it up to date and ready for next steps. You've probably heard a lot of people advocating that design should have a seat at the table. Well, in 2016 we got a really, really big seat.

[00:07:12] As the design team we were a very small group of like-minded designers figuring it out, all while our data game was really, really weak, so we had nothing to leverage there. What we did was create a hive mind, by talking so frequently with each other, day in, day out, and getting so aligned on what we wanted to achieve and how we wanted to do it, that we could ask each other things like, "Does this design feel WeTransfer?" and all actually understand what we meant. Every decision we made during the redesign was based on intuition, gut feel, best experiences and a lot of discussions.

When metrics dominate every discussion

[00:08:01] While we were making that 2.0 version, the company went on to grow, and we were trying to become a more mature product. Leadership changed quite drastically, we even got a new CEO, and that all changed the way we worked. From this one siloed design team, we shifted to more cross-functional teams. That's nothing new, probably most of you work that way, but it was really needed for us to grow and become more mature. Initially at WeTransfer, the direction was always set by one person having a vision or intuition or gut feel, but when you grow, that's really hard to maintain and scale. So a need arose to know what the impact of our work was: what is tangible and measurable. That's metrics. No, this is all fine. Yep. That's all intentional.

[00:09:05] Of course, as we started to try and grow, it was a gradual transition from having one person with an Excel sheet in the corner to eventually adopting OKRs on every level of the company. But what you started to notice was that metrics started to dominate every discussion we had around product. You could give a perfect suggestion, "Hey, we should do X," and the response you would get was always, "But how does that move the needle for this one goal that I have?"

[00:09:48] The thing is that, done wrong, performance metrics promote short-term vision, because it's a natural tendency to put your efforts only towards the one goal that you're tasked to achieve and let go of the rest. But of course a lot of things in a product are not just one silo. It's about looking at the broader perspective. We got this completely wrong, and many people felt frustrated. They felt like WeTransfer was turning into this big, corporate, very slow-moving thing where you could not exercise your craft anymore.

Collect and the 10x acquisition OKR

[00:10:34] In the meantime I got to work on an exciting new project called Collect. WeTransfer.com is mainly used on desktop, and with mobile eating the world, we wanted to find a concept that would grow our presence on mobile as well. With Collect we wanted to do something different from WeTransfer.com. Collect lets you store and bring together content from different apps into one place. That's what it does.

[00:11:12] The thing is, we released this app and we did not yet know whether it was what people actually wanted to use, and we were operating on a very small scale compared to WeTransfer.com. When we got an OKR assigned, it was to fully focus on acquiring new users. We had to 10x our user base. We were all super new to this whole metric-driven way of doing product, so we all just accepted it, and we were mainly happy that there was some guidance from the leadership.

[00:11:58] For WeTransfer, acquisition is not that hard. We have a lot of organic traffic already, so by advertising on the right channels we grew our user base five times within a couple of months. But monthly active users, or acquiring users, did not tell us anything about whether people actually liked the app and used it in the way we envisioned. It only forced us to focus on the top of the funnel. MAU grew quite heavily, but we started to notice that very few people returned in the long term. The idea didn't seem to click. But our goal was to get people in, and we delivered on that, so everything seemed fine, right?

[00:12:49] With multiple products we also have multiple product teams, and each team got assigned its own OKR. Our biggest acquisition channel was, and still is, the download page on WeTransfer.com for mobile users. To reach the goal, all we needed to do was grow our presence there. But a different team actually owned that page, so to get anything done there we needed to team up with them. The thing is, they had their own OKR. Their goal was to have more people download files through this page, so us acquiring new users immediately negatively impacted their goal. What happens when teams don't align? You're very likely not to be willing to work together, maybe in some cases even compete. It was not a great situation to be in.

From vanity metrics to a habit moment

[00:13:45] This metric-driven way of doing product development actually led us to forget about the user and about the products we were making. We were only weighing efforts by the impact they had on the metric. We realized that this was a bad situation. The things we were chasing were only vanity metrics, which you could call hollow numbers. They work great for PR and marketing, "we have X amount of people using this app", but they tell you very little when you're building a product. So we took a step back. Why are we doing this, and what is the value of getting more people to sign up if they don't stick because the product is not good?

[00:14:33] What we also understood was that to be able to communicate with leadership, it needs to go both ways. So we needed to set some sort of metric. This time we did not want a vanity metric like that, but something that would inform us while building the product. We make productivity tools at WeTransfer, and what drives productivity? Habits. So what we did was define the habit moment, together with our data scientist, to see what was actually the moment that users saw value in Collect. That allowed us to shift our focus from acquisition to activation: users that add different types of content over time.

[00:15:29] It really reshaped our conversations with other teams. You can imagine how this makes a big difference when designing your presence on the download page I mentioned earlier. Going through the hassle of doing that was not easy, but it allowed us to restore the balance between metrics and the product vision we had, and it allowed us to focus on improving the proposition. From there we were able to take steps that are now showing traction. The metrics are not the driver, but they support and align with our product vision and can meaningfully indicate whether we are building the right thing.

The characteristics of metrics

[00:16:15] I wanted to tell you this as an example of how easy it is to get sidetracked by an imbalance in how you do product. A growing company comes with more stakeholders, more departments, more designers, different ways of doing things, different ideas on what the right thing is. But it can also make you feel like you're part of a production line, like you're not doing the creative work you were hired to do in the first place. However, metrics are a super valuable source of information, and if you have the skill, it's a competitive advantage that you need to leverage. You need to realize that it's important to use the right metrics at the right moment to support and inform your vision.

[00:17:06] When working with metrics, it's important to know that they come with certain characteristics. Those are not necessarily good or bad, but if you're not a trained data scientist, just a humble product designer like I am, it's good to know them. Metrics allow you to go wide. It's super easy to acquire insights from a big group of people. Metrics also measure things that are established and tangible. Innovation, though, is often about doing things that are not yet established. So metrics can help you point out interesting behavior, unintended or unexpected, but they will not necessarily tell you where to go.

[00:17:51] Also, a metric loses the context that actually built up that metric. Think about "feature X was requested 10,000 times". Okay, but why was that one feature requested? You lose that. And metrics can give a false sense of certainty, and that's dangerous when you're focusing on the wrong metric, like we did. One reason I see performance metrics eagerly adopted is often this seeming sense of certainty. But when they don't give the desired result, often the answer is to just add more metrics to the problem. Then you have to ask yourself whether you're measuring the right thing. Do you have the right proxy for what you want to know, and will that help you find what you need to know? Because adding more data is also adding more noise.

Driven, informed, aware, and the stage of your product

[00:18:57] Due to time I might skip over this, or just go really fast, one of the two. Rochelle King, VP of product design at Netflix and Spotify, I believe, wrote a book about designing with data, which is super insightful, so check it out. She talks about different ways of thinking about how to use metrics within your design process. She talks about the ones we already know: metric-driven, which is great for A/B experiments, and metric-informed, where data is one of the deciding factors but not necessarily the decider. And she coins a third, which is metric-aware, in which you are aware that there are many kinds of problems and possibilities, and you allow various inputs into your design process. It's a more strategic and less dogmatic way of thinking. So when you're facing a design problem, start from the outside in, and allow yourself to get as much diverse input as possible before going towards that one goal.

[00:20:13] Also, if you look at your product, you can probably see different stages for it. You can identify three stages, and those also influence what kind of metric is important to focus on and what is relevant for your product. There's pre-product-market fit, like Collect, where we are still finding the core value. There's a growth stage, where you try to acquire as many users as possible because you have that product-market fit and know what works. And there's a value stage, where you try to retain people, lower the churn and get as much value as possible. For instance, WeTransfer has product-market fit, and that's great for more metric-driven decisions, while Collect has less established behavior, so there are different expectations from people, and the vision of what we're building takes a more prominent role there.

[00:21:27] I've not talked about product vision a whole lot, but we might want to get into that another time. To wrap things up: building great products is about striking the right balance between the vision and the metrics, and it's super easy to lose one of the two sides. When you're working with metrics, consider the stage your product is in and get to the right metric to help inform your vision. Be aware that metrics come with their own characteristics. Realize there are multiple ways of working with metrics and of incorporating them into your workflow. Leave those vanity metrics at the door, and build and iterate on a vision that's shared broadly by the product team, so that they can operate relatively autonomously. Thank you very much for your time.

Q&A

[00:22:24] Host: Thank you, Stijn, for a nice talk. Any questions? There's one, together with the coffee. Coffee and the microphone.

[00:22:39] Audience: Nice talk. Okay, an example. Let's say you have your main transaction page, like a search page, and you have a registration pop-up, and you see that's where most of your users are coming from. Then your data team notices that if we show this registration modal every five seconds, we're going to get more users. They run an experiment, and apparently the people who see the registration modal every five seconds, after they see it twenty times, say, "Okay, I'm just going to register." So that increases your number. How do you combat that?

[00:23:21] Stijn: I think it's also a discussion about what you value as a product. For WeTransfer, design and user experience are very important, so I don't often get asked, "Hey, can we make the login form flash so that more people sign up?" But of course we've also had discussions about splash pages to upsell products. What we did there, and you probably come from a UX perspective already, is build a case that we don't think this is the right approach. What we then did was actually counter that problem with metrics as well. For instance, we had a landing page that would sell you Plus, where you can send bigger files. What we noticed was that there was a ten percent drop-off for first-time users.

[00:24:17] Then it was a very easy case to make: it's important that people first experience the product. Yes, of course people will sign up if you put it in front of their nose, or they'll just think you're a paid product or whatever. But it's worth having people experience the product in a certain way first, or go through this or that, before prompting them with an upsell. So it's probably that conversation.

[00:24:46] Audience: Yeah, definitely.

[00:24:49] Stijn: That's why I think it also helps to come with the same kind of arguments that they are leveraging. I am very fortunate to work at a company that has always valued UX a lot, and where there's also a dedication to making something that's less about pop-ups. That's why we have full wallpapers as ads instead of banner ads. We could earn way more money, but we've decided not to do that, and that's also a differentiator in terms of who you are as a brand. So probably not the answer you're looking for.

[00:25:47] Host: Just one question over there. I guess, for the time constraints, this is the last one we're going to take, but you can find Stijn around for the rest of the day, right?

[00:25:58] Stijn: Sure.

[00:25:58] Audience: Thanks. Stijn, first, my compliments on your presentation. I found the minimal slides with great speaker content a really good combination, especially not causing confusion by having too much shown on the screen. That really clicked for me, at least. My question: early in your talk you talked about creating that hive mind within the team that allowed you to ask questions like, "Does this feel like WeTransfer?" Could you talk a bit more about what you experienced as critical success factors in establishing that hive mind? What should we try and do to create that? Just a few concrete points.

[00:26:41] Stijn: That's interesting, because we really had that hive mind when we joined WeTransfer, and then we were split as a design team into all those different product development teams. What we started to notice is that people started becoming silos again. All of a sudden everyone was only focused on their product team and not necessarily on the whole UX of the product, and people were going rogue. For the last year I've been investing a lot of time in how we can bring the designers back together while working in those cross-functional teams.

[00:27:18] I think what's super key is to look at what kind of rituals your team has. How do you communicate? How do you get feedback? Promote feedback as early and as often as possible. And also look at how people give feedback to each other, because we also noticed that at some point people were getting annoyed for other reasons, and all of a sudden they became very snarky in giving feedback. That then turns into people protecting their work and not sharing it, and then you have this divide, instead of those people really working together. So be as transparent as possible, because I think that's what happened there, through all the conversations we had: "Hey, should we do this? Should we do that?" It was just a lot of conversation. And even if you're not in the same team, you need to try to facilitate that.

[00:28:15] Host: All right, thanks. Thank you, Stijn.