Hypothesis Driven Development

Arnstein Teigene at UXDX Community: Oslo. Video: https://youtu.be/_u1-pRQwQ6E

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Ten years trying to understand what product management is

[00:00:00] This is just based on my experience, so it's not the Bible.

[00:00:28] Just a bit about me. I've been working in product development. I think it's ten years ago when someone came over to me, and this was before Schibsted, before office of Berlin[?], and asked me if I wanted to be a manager. Sure, great. And the next question was, what was that experience[?]. I was really naive, and I've spent ten years trying to understand what product management is. It's funny to see, like Elizabeth talking about the tools, I got that question the other day as well: what kind of tools are you using as a product manager? I don't know. I really don't know. There hasn't been one tool that I brought with me, just a lot of theory that I seem to remember, stuff that seems to be true over time. I'm going to go through that.

[00:01:20] And I try to be mindful of not letting the tools get in the way. We use Jira. Jira seems to get across a lot of different companies, and there's this difference I learned about the other week, that you have epics and then you have stories. I was very confused about that, so I was googling, what's the difference between an epic and a story? And it was a forum online where someone's going, I think you'll find that a story is different from an epic in the way that made the name[?]. That's not important.

[00:01:57] I think what's important is the way I define my success, which is what happens in production. I keep thinking about that, the impact you make in production. That's how I want to define my success, and that also defines my approach. So I'm just going to share some thoughts on that.

Where I've worked, and why I joined Documaster

[00:02:22] I'm currently at Documaster. I used to work at Schibsted, where I worked with the lovely Sweat[?] down in Barcelona. We worked together, I can confirm she was trying to combine a master roadmap of 29 roadmaps from different countries and cultures, and that was in a matrix organization, so that was very interesting. But that was millions and millions and millions of users, consumer product.

[00:02:50] FINN.no, most Norwegians have heard about FINN.no, also consumer products. I launched the FINN apps, went from not having apps to having apps. That was part of what I did along with a lot of things there, going from desktop to mobile. And then before that I worked in Opera, where I was part of launching the Opera extension technology, also to, I don't know, about 30 million users.

[00:03:25] But now I'm working in something a bit smaller. How many people have used Vipps? How many people have heard about Documaster? A few. You guys don't care. The reason I jump ships, the reason I joined Documaster: one, I know the guy who started it. And two, at Schibsted and FINN and so on I was part of defending a market position, and I wanted to be part of taking one. That's an interesting switch. And additionally it's B2B, so that also changes a lot of things. But there are some fundamental truths in how you approach product development that are true for both.

Elon Musk, and what hypothesis driven development is

[00:04:07] So who's this? Elon Musk. Do you know what this photo is of? What's he looking at? He's looking at this rocket, he's looking at the Falcon Heavy. This is an awesome image, because he's looking at the Falcon Heavy having been successfully launched into space. And he went from absolutely nothing, knowing nothing about rocket technology, no people, and he started SpaceX. I thought he'd bought something and then expanded on that, like he did with Tesla, but SpaceX he started from absolutely nothing and started recruiting people from that domain, and then he does this.

[00:04:49] It's an extreme example of a crazy vision, and then also followed up by crazy execution, which I think is a little bit what Vipps is as well. But it brings the question, how does he approach that, and is there anything you can learn from that? I think so. In this presentation I wanted to talk about identifying an opportunity, articulating a solution, and minimizing the risk of running off a cliff, of complete failure.

[00:05:35] So, hypothesis driven development. What is it? If you google hypothesis, it will say something like, a proposed explanation for a phenomenon. So that's helpful. Before jumping into the details of that, if you haven't worked with product management, this is what I show people. Adam Nash has this really nice and simple explanation of what product management is. First you have to identify what game you're playing, then you have to keep score, and then you have to know your next three things. If you're not super confident about the next three things you're nailing, then you're in a good position. If you're really unsure, then you have to go back to number one and number two. But it's a lot about measurement.

[00:06:18] And if you have a complete brain shutdown and forget absolutely everything, I really want you to remember this as well. This is a thing I've kept with me in my luggage, so every time I go to a new place I will ask this question: how good are we at these different things? I've started in teams and departments and seen that we're really good at execution but we're horrible at insight. So what we're executing, what we're launching into production, is 95% meaningless. That means we have to hire people and staff up and become really good at those three areas.

[00:07:09] Insight and execution, and that's basically how you execute on hypothesis driven development. You can change this sentence a little bit and say, well, I'm observing something, I'm seeing a problem, now what could be a proposed solution for that? It's not that hard, I think, but the context of it might change a little bit.

Incremental improvement or radical change

[00:07:27] This is Reed Hastings, who is very quotable. Netflix is known for this pivot, going from shipping out DVDs to just streaming. This happened years and years ago, and when he got asked that question, you're really known for that pivot, he said, it's not a pivot, the core business has always been the same, this is just a different channel, a more efficient one. But I think you have to look at what you're doing and think, are we incrementally improving what we've got, or are we at a place where we radically need to change?

[00:08:22] Most of the time you're just incrementing what you have. And some of the time, like when you're trying to take a market position, everything is up in the air, everything is in flux, so there's just so much uncertainty there. And then you've probably seen a business model canvas or a lean canvas, and now and then, depending on the company, you're told to fill them out.

[00:08:52] When you have a hypothesis, you have to ask yourself, who's this for? Who am I articulating the hypothesis, or what we're doing, or the roadmap, for? Now with Documaster we're hardly describing this in detail, because we don't need to. At Schibsted we had to describe it in detail, because we had to build trust within the organization, like we heard about earlier. So sometimes we fill out these little bad boys. And then sometimes it would be, you know, that's a great idea from leadership, and you go, oh my god, I have to deal with this.

[00:09:29] So that happens. What do you do with that? You could just run with it, because you can build an app. We're 100% meaningless, but let's build an app. I think they have all these statistics for startups, but they don't have good statistics for internal projects at scale. And I recognize this: now when I'm in a startup and we're trying to grab a market, this is a nerve-wracking thing to do, because failure is an absolute nightmare. But failing a project at Schibsted is like, didn't work out, okay, next project. Still, I think the failure rate of internal projects is probably way higher.

[00:10:27] It's the projector. Yeah, I was just going to show this video. No, I'm just — okay. Well, I was actually going to show how we structure it. It's very similar to what Elizabeth was talking about before: how we structure the definition of a problem, what the solution hypothesis is and how you execute. This was the approach that we took at Schibsted, and it's just a super generic one, there are no surprises. But it's really building up the expectations of this stuff.

[00:11:13] Sorry, it goes on sleep mode and there is nothing we can do. It's just stupid. It's broken. No, no, we've tried that, we've been doing that, it's still — alright, okay, well, I'll explain it.

[00:11:33] Audience: And it needs to warm up now.

[00:11:35] It needs to warm up as well, yeah.

[00:11:37] Audience: Just to put you under extra pressure.

[00:11:41] Yeah.

Articulating the problem, and how you would measure it

[00:11:43] At FINN we had this thing where, how do you describe your observation? That's one. So you probably want to standardize that. At FINN it ended up being a verb, in that you have to describe a problem, and that's how we prioritized everything across FINN eventually. The entire organization had a massive list of problems that we had identified in the market. And even up to the leadership level they had to vote on different problems, and then the solutions were more down to the teams to do something about.

[00:12:23] This might be a little bit of a negative way to view it as well, everything's a problem. You could articulate it as needs, or as an opportunity and so on. But I really find that over these ten years I keep going back to problems for articulating it, because it's very, very easy to communicate internally as well.

[00:12:49] And then, once you articulate a problem, here's a trick that I also do. I always start thinking about how would I measure that? How do you measure that problem? Because you need a way to measure that problem in order to measure the solution of that problem. That's a whole genre of its own, about how to test and measure and so on. But it's so helpful. Mentally it crystallizes everything once you start thinking about very concrete metrics.

The FINN observation: 20 seconds at the intersection

[00:13:25] For example, at FINN we observed — first you have the observation — at FINN we started up, this was back when smartphones started becoming more and more popular and then Apple launched their App Store. At the same time you could go, oh yeah, that's incredibly interesting technology. But then you start observing that every time we went to and from work you'd see people at the intersection waiting on the bus, just looking at their phone. So that's an observation. And then you start thinking, well, how much time do they have? That's probably a 20-second session. And our technology is optimized for a desktop experience with a big screen. So you whip out your phone and go, okay, this is pretty horrible.

[00:14:17] And of course then you have to start thinking, what's the reach of this observation? How many people are affected by it, and what's the impact and the confidence and so on? This is very, very tangible. You obviously look around you: impact is huge, confidence high. And what's the effort to fix it? That's more of an internal question.

[00:14:43] And then there's an important last question that I like to ask as well, which is, what's the consequence of not doing anything? I think that's very, very important in an organization where there are a lot of feature requests coming from management. It's like, okay, what would happen if you didn't do anything about that? That's a mind question.

[00:15:07] So back to FINN, what's the consequence of not doing anything about that person having 20 seconds at the intersection? Well, there's the obvious risk that if someone else were really good at that, and you could multiply that with everybody that you saw, that's a lot of sessions. So if someone became really mobile friendly and did a marketplace, that's a massive risk. So we were like, holy shit, we need to launch something where you can go from zero to looking at an ad in seconds. And now we started talking about how to measure the success of this: how many short sessions, and how long it takes from the start until you have an ad up on your phone.

Solution hypotheses, user stories and job stories

[00:15:53] So that's the proposed solution. I'm not going to spend too much time on the proposed solution, because articulating the problem is pretty straightforward, and articulating a solution as well, having a solution hypothesis, that's also pretty straightforward.

[00:16:09] User stories, I don't do them, because there are just too many assumptions in them. As a user, what does that even mean? Of course you're a user of the system. And I can back up, or I can, I want to do this. It's just a massive assumption built into the user story that I don't like. There's this thing called job stories. I like that a little bit better, because it takes you, when I'm looking for something I think I want to, so it gives you a context and the need and then the outcome. So that's better, that's a step up, and I like those because they're easy to communicate as well.

[00:16:53] But I still keep going back to problems. So it's a problem, and the solution would be, what if we just enable blah? That's also really easy to communicate and get people aligned on. So that's not hard either. None of these things are hard.

Testing, measurement and leap of faith assumptions

[00:17:07] But then you have the testability, the measurement. This is where, okay, so you've articulated a problem, I have a rough solution hypothesis, now we need to figure out if it's worth doing or not. And this is not hard either. None of these things are hard. There are so many frameworks and people are arguing about the best way to approach this, now it needs to be quantitative data and quantitative data is better. But it's just about collecting data points, and it doesn't need to be scientific.

[00:17:41] It's just about, what defines the success of this thing is what happens in production. So you just need to have enough data points to be confident that if I invest x amount in this it will be a success. That's what it's all about. Google Ventures has this design sprint, so if you just want some thoughts and ideas about one way of doing this thing, just go to, I think it's gv.com, design sprint or something like that.

[00:18:09] But one thing that I keep thinking about is, what's a leap of faith assumption? This is something that a lot of people skip. It's like, oh yeah, so we have this problem, and now if you just give them the app that will be a solution and, you know, world dominance, it will be amazing. But you have to ask yourself, what needs to be true in order for that person to have that app in your hand? Well, they need to have heard about you. Okay, what does that cost? The average marketing budget of going from zero brand to a well-known brand is always hundreds of millions of kroner. So that's a leap of faith assumption, that that would just magically happen, that you've solved solutions somehow, or that users are actually interested in solving the problem. There are all these leap of faith assumptions that I think is useful to just write down, because this is where it normally fails.

Faking it, showing it to people, and what Documaster is

[00:19:13] And then, this is such a cliché, having a Steve Jobs photo there, but I think this is also so true. I've done so many user interviews and user testing, and focus groups and so on, and they're extremely valuable. But I think the most valuable thing is when you show people something, rather than just asking for an opinion. There's a book called The Mom Test, and that is, if you ask your mom, is this a good idea, your mom will always go, yeah, that's amazing, that's great. People have this tendency to give you good feedback because they don't want to disappoint you. So I think it's better to just try to fake it and show people.

[00:20:06] Documaster — I think I forgot to tell you what Documaster is. What we're trying to do, one observation that we're seeing, is that there are so many companies where documentation is important: law firms, shipping, oil and gas. Documentation is important, but when we've been talking to those companies it's a mess. We're looking at what they're using, and it's folder-based, like SharePoint, Box, or these enterprise SAP solutions, and it's a complete mess. So we've been asking ourselves, why isn't there a Dropbox but focused on enterprise documentation? Dropbox is designed for consumers. Why hasn't someone done a really user-friendly alternative to all these enterprise solutions for documentation? So that was the quick version.

[00:20:55] But in order to understand the problem, you have to get out of your office. I think we talked to about 100, 150 companies. This is B2B, and the first question was, how the hell do I collect data points? At Schibsted you don't feel it, it's easy, you just go out into the streets, like with Vipps as well, just go and talk to a random person. I was like, have you used Vipps? And he's like, yeah, yeah. And then you just ask questions. For me it's, how do I talk to someone in oil and gas? I didn't even know anyone in gas.

[00:21:26] So the first thing I did was bring a guy on board. His name is Lars, and he is fearless. He just calls and he's like, hey, yeah, we're building this new thing, I have to show it to you, it's awesome. And he's driving me along and he's like, what we're showing, that's your game. So we had so many awkward meetings, but we did at least, I think we did about 50, and then we went, there's definitely something here, we can see a pattern here.

[00:22:04] And then, okay, so let's start. And I worked together with Swanini[?], who is also here, on, let's start faking a solution and I'll start showing it to people. And sometimes they'll be like, no, this is not interesting, but every now and then they'll be like, oh crap, this is really interesting, how far are you away from having a product?

[00:22:22] On the right hand side is the actual implementation. So this is what do we call it, MLP, minimum level product, that we shipped this week. So you can see that eventually it turned out being pretty good. But yeah, faking it, and showing it to people, I think that's the way of doing it. And you have to remember this: if you ask three people, like your date, that's not enough data points. You have to get really uncomfortable and just ask hundreds of people.

Nothing interesting happens in the office

[00:23:10] So that's it. I wish you could have seen some of these memes I've been over. But this is a fundamental thing that I hold true always. It's so easy to default to the office chair, go to the office and get caught up in these forums that debate the difference between epics and stories in Jira, for example. I'm going to reply to that.

[00:23:32] I don't know if you've heard about NIHITO. It's just a silly term that athletes[?]: nothing interesting happens in the office. Just remember that what you put into production is at the mercy of your insights, and that insight is not going to come from inside the office, that's going to be outside. So the more you're outside, the more you're outside your comfort zone, the more insight you're going to start basing your solutions on. So you just really have to do that.

[00:24:02] Go APE is another acronym: analyze, plan, execute. And that's panace[?]. You just keep doing that in iterations. So that was it. There's nothing hard about this, there are millions of tools. I think, based on ten years of experience, the hardest thing is —

[00:24:29] That was it, that was the last one.