Planting The Right Discovery Environment

Soeren von Sebelin at UXDX Community: Berlin. Video: https://www.youtube.com/watch?v=FpOqylhJ1ts

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction: from producer to product nerd

[00:00:00] Hey, ladies and gentlemen, and welcome to Planting the Right Discovery Environment. This talk explores the ecosystem that the product team at sofatutor developed for enabling discovery work, and I will share our learnings from the change and why it is important to not only have the right tools, but also the common ground on which the team can bloom and bring sustainable success. I will talk about the team structure, the product work and the role of management. And this is not only directed to product managers or designers or developers, but also to the managers and people who want to make real change at their companies, also known, for example, as entrepreneurs.

[00:00:41] But let me shortly introduce myself. My name is Soeren von Sebelin, and I started my career as a producer at Goodgame Studios, which is now known for titles like Goodgame Empire and Big Farm. There I grew with the company as we scaled from roughly 100 to more than 1,300 people in just four years, and I was responsible for the success of scaling teams, product teams basically, of around 7 up to 60 cross-functional team members. Within that work, I loved the challenge of creating the most efficient process possible, while keeping the goal and certain conditions in mind, obviously. But slowly the question of purpose developed in my mind.

[00:01:31] I wanted to do good and bring value to people with my work, and that's why I started falling in love with real product work, always having in mind the value that you bring to people. So a product nerd was born. Yay. Since then I can't stop thinking like a product manager, even at work or on vacation, and all the books I read and all the talks I enjoy are about product and its value. Maybe the only things that get me off my product mind are board games and surfing, luckily, because otherwise I'd probably drive my wife crazy. And this is me in Galicia, doing a workation for about two weeks, and for my wife and me it was the best time this year. I know there wasn't really big competition, but still, only product work and surfing: it was freaking amazing!

[00:02:28] The wish to bring value to people brought me to education, since I personally experienced the flaws of the system. I still think it seems to be one of the least developed systems in Germany, next to internet reception. So after working at 1&1, wink wink, I moved to wonderful Berlin to work at sofatutor as a product owner. sofatutor is an e-learning platform providing German pupils, parents and teachers with thousands of learning videos and exercises, and we cover nearly all the content of the German school curriculum. Here you can see a little map of our exercises.

[00:03:20] sofatutor's former Head of Product from two years ago was a person I knew from before. Luckily, even before I started, I knew that a seed of product discovery would find nutritious soil to grow in. And after I recommended the book Inspired by Marty Cagan, it seemed to have had the impact that I hoped for, because after about a month he sent a colleague and me to London to a workshop with Marty Cagan. There we were able to learn more about post-agile product management. That's what we called it back in the day. It's actually not so long ago, but it feels so long ago because so much has happened. Our goal was to do top-notch product work, to learn from the best and bring this to sofatutor.

[00:04:13] I have to say that all the experiences and changes in the company that followed were clearly not all driven or initiated only by me, but rather by an amazing team effort from people with the genuine wish to make things better. I have the honor right now to walk you through some major learnings of this change, to grow a sustainable environment for product discovery. As I already said, spoiler alert, we're still far away from the optimum and we're also still learning, but I think we're at a stage where we can talk about our learnings.

Team structure: how to water your plants

[00:04:52] So let's first talk about the team structure, and how to water your plants. When I started at sofatutor, the product team and the tech team together were already set up in agile development teams. The transition was about one year in the past, and they were already comfortable with sprints and iterative work. They tried to keep major topics in each team, mostly based on the knowledge and past projects of the developers and the POs.

[00:05:28] The paths of the teams were still orbiting around something we called the responsibility map. It looked something like this. Here you can see each and every part of the platform had to be placed somewhere on this map. This is a classic example of component-based teams, and it caused a ton of discussions and attempts to somehow get it right. The result was that the teams felt like servants, mercenaries, responsible for continuously working on X, Y, Z, and not testing whether what they were working on actually made sense or made a difference on the platform.

[00:06:06] Next to this, all possible stakeholders were equally mapped to the teams, and what happened is that we tried to water every plant on our list, just like this picture. That was the background, by the way. You can imagine that we didn't have enough water and resources to really grow any fruits this way. We just tried to keep the most important plants alive, more or less. In our talks about product discovery, we decided that we wanted missionaries and not just mercenaries. Just as John Doerr preaches, people should want to make a difference with their work, feel success and failure, and harvest the fruits of their work.

[00:06:52] One of the major changes, led by our Head of Product and the management team, was to establish team missions. Those were closely aligned to the team's knowledge and the personal preferences of the team, and so they were really actionable. That was good, because then we could really deal with them right away, and that was super important. Additionally, it was important to get the legitimization from management to act and focus on those missions, and that's why it was good that they gave them to us. They talked with us about it and they gave them to us, and we felt empowered to work on this. Over the course of the following months, we gradually decided for and against components and KPIs that helped us with our missions.

[00:07:41] We were allowed to actually make bold moves, as long as we could explain our reasoning to all the stakeholders. So we learned to focus on what matters, and we communicated it properly. What happened to the responsibility map that I just introduced to you? The whole responsibility map basically turned into a contact list for stakeholders and nothing more, for when they have a request or whatever about this and that feature on the platform. And this led to the end of those endless discussions. The thought is, okay, we don't have to get it right. We basically just make it about the missions and get better at this, instead of somehow discussing the abstraction level of the items on the responsibility map.

[00:08:47] The mission of one of our teams was all about elementary school kids, and afterwards we focused our efforts extremely well on understanding the children: how they learn, what is actually fun to them, and how we can bring such an experience onto our platform. This was one success story, resulting in this overhead [?] that we also showed at the end. This success means that we now have a super fun way to do exercises for elementary school kids, all led by Pommes the penguin. Pommes is a wordplay in German, it's like fries, but it turned out that it's a very, very famous name for pets. So our product manager and that team made a very cool move to just go with that, and they actually love it. Kids love it! That's the first result that we had after changing to missions.

[00:09:55] Another example is how we managed to reimagine the whole navigation of the platform and reduce the path to the content by three big and hard steps. This is the first screen of those steps; we crunched it down to the most necessary parts. So our main takeaways were the following: transpose component-driven teams into mission-focused teams, learn as a team what to act on and what not to act on, and communicate those moves and decisions. Be bold and outgoing as a team and show the results.

Product work: let the sun shine

[00:10:46] The next area I want to talk with you about is product work. I always understood product discovery as a major change in how a product manager works and thinks. One of the biggest problems occurs when product people don't know what assumptions their work is actually based on. The moment you start realizing that basically you don't know anything, you realize that you need to learn, to see, maybe for the first time in your career.

[00:11:11] A few years back, we had very little knowledge of the actual behaviors of users, their intentions and wishes, and everything we saw came from isolated analytics platforms. Our database, for our type of product, was very superficial, and once we understood how little we knew, it felt like walking in the dark. Just like this guy here. He got a 6, and in the German school system that's a very bad thing, but he still smiles, so whatever. Probably because he didn't know better. We knew, okay, we're walking in the dark here, and we also know that discovery and plants need sunlight to grow well. I know plants also grow without sunlight, but they have no color. They do have shape, but they're missing something very important.

[00:12:04] So what did we do? First, we tried to find the best ways to do systematic generative and evaluative research. I mean systematic, so we can go over it again. I call this Learning First. We need to find the problems to solve and then understand whether our approaches to solving them are working. The basic product discovery 101, I would say. So we integrated Amplitude, Hotjar and SurveyMonkey, did tons of user tests, interviews and surveys, and learned best practices on how to run A/B tests. Here are a few examples of the tools that we basically use at sofatutor.

[00:12:49] We also use Slack. In there, we set up an automation that forwards any type of user feedback from the platform directly into a Slack channel for the company, and now whenever we do something, we learn as quickly as possible, directly from the user, where something went wrong.

[00:13:09] Let me run you through this with a very straightforward example: the quick start. With Hotjar surveys, we managed to get our hypothesis confirmed that around 35 to 40% of our users want to start where they left off in terms of navigation. This is quite a big deal. That means people get to the platform and they actually just want to start where they left off. So when we give them this possibility, 40% of our users get directly to the content super quickly. That's what we want to grasp. That's the value we want to bring.

[00:13:47] After gathering a lot of ideas on how to solve this, we ran some usability tests via Lookback with users recruited via Testing Time [?]. One solution seemed to work, and with this we recently deployed an A/B test supported by Convert, now with Amplitude. We realized that the metric is not getting better, but, according to the A/B test guide, even significantly worse. Since we know there is a big potential, we will have to iterate over it again. But just imagine: if we had taken out one or two of those steps, we probably wouldn't have acted on it at all. We wouldn't know that there is this potential. We would probably just leave it on the platform, even though we know that it makes things worse.

[00:14:32] This feedback loop that I'm talking about is the sun. We need to know what to do, we need to know where to go. Never before were we able to make such quick and well-informed decisions, and we never felt closer to our customers and users. Everything is measurable, all the doubts of naysayers could be debunked, and we stopped holding ourselves back from really making a change and bringing innovation to the table.

[00:15:07] Something we need to take care of right now is making sure everyone who should know about those insights is really up to date and can understand the reasons behind decisions, and is also able to take the right decisions. So working with reports and presentations, workshops and documentation is actually key. For this we need to take extra time and invest as much love into this as we put into gathering the data and forming insights.

[00:15:34] A few tools that seem to work for us: the opportunity tree. That's just an example of it; I didn't fill it in, but I think you get the gist. We also built user story maps with the team, and when we do user tests, we make them hypothesis-driven. We tried customer letters, which worked well, and we also have dedicated weekly meetings. Especially the latter now seems to be the core of functioning discovery work and alignment. It helped the team keep talking about the why rather than the what or how, because the what and the how more or less already have dedicated spots: sprints and Scrum. Just talking about the why never seems to really have a spot, so we just put one in there, and we called it Discovery Lab.

[00:16:36] The main takeaways are: discovery, learning first. Create valuable feedback loops, make your learnings transparent, and actively talk with your team about the why.

Management: the gap that grew

[00:16:52] And now let's get to management. When we came back from the workshops, we started our tour through the company. We gave tons of presentations to people, telling them what product discovery is, what it means, how we imagined it would be at sofatutor, and what our work would cover once we got the water running. We talked to the development team, to the product team and the design team, to management, to whatever stakeholder you can come up with. We wanted to make sure that everyone in the company understood this mind shift.

[00:17:30] In the beginning, it seemed to be very successful. People gave the thumbs up, and they were like, yeah, sounds cool, go for it, whatever, you know. You can see two presentation slides. The first one is the original from when we gave the presentation of what product discovery actually is, and the second is what we called the core team: what core functionalities you need to cover in the teams. Like this, we tried to make sure that people knew what was going on.

[00:18:07] Unfortunately, we realized that there was a big gap growing between management and the product teams, and it was leading to a lot of miscommunication and issues in the strategy process. If we hadn't done anything, things might have got messed up big time. First we needed to understand what caused this, and we came to the following conclusion about product discovery. Yes, I know, it seems like something we could have known before, but knowing and realizing are two different things. Clearly we heard it from Marty Cagan and from all the books that I and my colleagues read, so we could have known it. We thought it was only something that drastically changes the work for product teams.

[00:18:59] After our tour, no one was aware of what it actually means, effectively and practically, for their work and how it was supposed to change, so that discovery can actually bring sustainable success. And most of all, management didn't know about their participation. Their hard work is absolutely necessary; they provide the soil for the seeds to grow in. So yes, you need to get them on board, and this is something that we had right. We gave all those presentations. We had them on board. So what was actually missing? We should have planned clear action items for them and for us. We should have talked this through. I don't want you to make the same mistakes, and that's why I'm trying to be as clear as possible about what those action items are supposed to look like.

Providing nutritious soil

[00:19:56] We're in that process now at sofatutor, and bear with me: it's nothing that gets done within a month or two. It's a process, and you also need to acknowledge the process.

[00:20:07] Number one: you need a product vision. I know you've heard it already. Most of you probably heard, when you heard about product discovery and so on, that this is important or that you need it. And I'm going to make damn clear right now: you need it. If you don't have it, in the long run you're going to run in the wrong direction. So you need to realize how important a sustainable product vision, and the discussion about it, is. This builds the actual foundation of any discovery work. See it as the soil to grow your environment in. If there isn't the right composition of nutrients, hardly anything will grow, or the wrong plants will. And sure, it's super hard to create a good and compelling product vision, I know that, but it's absolutely necessary.

[00:21:00] Design a clear reporting process. That's something that I learned the hard way. How and when do you share insights? Who challenges them? What types of insights are to be shared? And how are you, as a company, learning together when a flower or a plant dies, like a badly performing experiment? It gives important nutrients to the ground for the next plants to grow on. They dissolve or whatever, and then the ground has a lot of nutrients, and the next plants can grow on it. Make sure you're building on top of that. It sounds weird, but honestly, don't shy away from the idea of writing a whole paper of learnings and insights. Sharing insights is a lot of work, but it's freaking worth it.

[00:22:00] Provide a clear strategic intent or business goal, and this needs to be absolutely transparent. Management and the teams, or at least the product managers, need to work through them together. Is it adding customer value or capturing value, also known as innovation and optimization? Is it cycle time? Is it ideation or growth? Fitting your vision, obviously. And if there is no such thing as a clear strategic intent, in the best case product discovery brings success that you don't care about, and in the worst case it simply goes nowhere.

[00:22:37] Last but not least, set up a fitting type of metric. I'm saying fitting because you need to find out what's the right way, and you need to do that collaboratively and transparently. If you don't have this, the team gets confused at some point, for example about whether retention or activation or adoption or task success are the keys to success for the company right now. Figure out if it's pirate metrics, the HEART framework from Google, or the North Star metric from Amplitude. It depends a lot on your business and on your organization, but make sure you find the right one, because you need to work this out together. If you remember the sunlight in the forest, there's only so much space where it shines, and just like in the market, you need to make sure you can grow in exactly those spots.

Three learnings to take home

[00:23:36] Let's shortly go over what we've learned today. It was probably a lot, and I want you to go home today with three major learnings. Number one: how to water your plants. Transpose component-driven teams into mission-focused teams, learn as a team what to act on and what not to act on, communicate your moves and decisions, and be bold and outgoing as a team and show the results.

[00:24:05] Number two: let the sun shine. Discovery, learning first. Create valuable feedback loops, make your learning transparent, and actively talk about the why with your team.

[00:24:19] Number three: provide nutritious soil. Formulate and discuss a product vision, design a reporting process, provide a 100% transparent and clear business goal or strategic intent, and work out metrics or OKRs in the teams.

[00:24:38] I hope you all will get it right too, make the best out of your own mistakes, and then learn together to grow on top of that, so that your teams can actually make the change that your company and your users are longing for. Thank you for your time, and I hope to see you later today at 6:00 pm Berlin time for the Q&A.