How To Experiment In A Zero Failure Environment

Tom Alterman at UXDX USA. Video: https://www.youtube.com/watch?v=KijptnCRlZY

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Experimenting where failure is life or death

[00:00:00] Hi there. It's a true privilege to be speaking to you here as part of UXDX. Today I'd like to tell you a story of how I learned to do lean, experimental product development in an industry where failure can literally be a matter of life or death. To quickly introduce myself, my name is Tom Alterman, and I'm on the product team here at Asana. For those of you who don't know, Asana is a work management platform with a mission to help humanity thrive by enabling the world's teams to work together effortlessly. I've never worked with a company as mission driven as Asana before. It also has an incredibly strong culture of collaborative product development and experimentation, and I'm not just saying that because they pay my paychecks and feed me delicious meals every day.

[00:00:47] But most of my recent experience has actually been in the construction industry. I fell in love with construction back in 2015 when I joined a startup called PlanGrid. There, our founders were the first to see how the form factor of the iPhone and the iPad could revolutionize how work got done on construction sites. They taught us to focus on building beautifully designed apps that foremen and superintendents would fall in love with. We had the highest rated construction app in the App Store, and at the time I left we were being used on over 1.5 million projects across 100 countries. In 2018 we were acquired by Autodesk for 900 million dollars.

[00:01:29] While we always wanted to deeply understand our customers' needs, we had a challenge that will be familiar to many of you, and that's how to ship great products quickly when they have to work perfectly first time. Much like getting ships to water, there's very little opportunity to revert.

[00:01:52] I'm sure many, if not all, of you have read this book. If not, I strongly recommend you do; it's really defined the last 10 years of how we build products and companies. In it, Eric Ries talks about how to get things to market fast using experimentation, with a key tenet being "fail fast, succeed faster." You may have also seen this quote from Reid Hoffman: "If you're not embarrassed by the first version of your product, you've launched too late." And there's Facebook's famous, if not infamous, mantra of "move fast and break things."

[00:02:29] All of these ideas always really resonated with me, but another thought also came to mind, and that was: I'm building tools for these people, and they don't want to hear about anything failing, ever. Their safety and their livelihoods depend on the tools that they have at their disposal being 100% reliable and dependable. This has long been a challenge in construction. Any change can be incredibly risky, so people prefer to stick to tried and tested methods, even if they're incredibly inefficient. This was the world in which we, as a relatively scrappy startup, were trying to build new tools to help them get their work done faster, easier and safer.

[00:03:11] You may not be working in construction, but I'm sure many of you are working on software that your customers rely on every day, and you know they never want to feel like guinea pigs for your experiments. If that's the case, this talk is for you.

[00:03:27] I'm sure you're all paying incredibly close attention, but for those of you that have attention spans as bad as mine, I'm going to give you the quick TL;DR up front. If you want to do experimentation in these environments, I recommend building a cross-functional team that truly cares about the customer, learning about the problem together, and treating everything as an experiment: not just the product, but the process and even the roles you're going to have on the team. Do the least amount of work needed to test your hypothesis. And finally, when failure is not an option, use humans to power the first functional MVP.

Submittals, and why they matter

[00:04:10] Let me tell you a couple of stories about how I learned this through my own blood, sweat and tears. Both of these stories revolve around an absolutely insane part of construction called submittals. What are submittals? In the United States, you need approval from the architect for every product and method you are going to use to build a building. This means every brick, every door, down to the actual nuts and bolts, needs to get sent to the architect and approved before work can continue. This means thousands of documents that need to be collected, reviewed and approved, on time and correctly. And if that sounds crazy, it's because it absolutely is.

[00:04:58] There are two things to remember about this. Firstly, everyone truly hates it. It's hard, tedious and stressful for everyone involved. You're filling out spreadsheets for days, you're keeping track of millions of emails, and the paperwork is endless. In the best-case scenario, if one of these is missed or late, the work will be delayed, your company could lose tens of thousands of dollars, and you could lose your job.

[00:05:24] That leads me on to the second important thing: these are incredibly important to get right, because in the worst-case scenario, buildings fall down and people die. That's not an exaggeration. This diagram shows a single bracket that was built incorrectly in a hotel in the 70s, which caused a walkway to collapse, killing 114 people, all because no one spotted this seemingly small mistake on one page of a hundred pages of a submittal document. It's an important process.

[00:06:03] The founders came to us and asked, "Can we build something in PlanGrid that would fix this process, or at least make it not suck as badly?" Obviously this couldn't fail fast, or at all. Being a scrappy startup, we had some pretty fun constraints that will be familiar to many of you. We had four engineers; actually, two engineers, and the other two were joining later. We had six months to get from idea to something that we could sell to customers, and the future of the business was dependent on this, so don't mess it up. All of this required us to rethink how we were going to build the product, and this is where we started to get experimental.

Learning the problem together

[00:06:49] The first step was learning about the problem together. We needed to truly understand the process to figure out how to tackle it. We did this with interviews, site visits and shadowing, and everyone on the team spent time with users, and we really mean everyone. This photo here is us on one of the visits, and this includes engineers, research, product, design, QA; all of us were there. We really got to know them: what are their days like, what are the boring parts. All of the submittals process takes place in spreadsheets and emails. Our users thought it was weird that we wanted to watch them do paperwork all day, but we learned so much from even the mundane bits of their day.

[00:07:33] As we went, we built up a picture of the process and pain points. After each conversation we'd list all the questions we still had, vote on them as a team, and then change our script for the next set of conversations, until we truly felt confident that we understood enough of the process to propose a solution.

[00:07:53] To reiterate, everyone was involved, and this was incredibly important for us, because we felt everyone needed to be bought in and focused on these customer problems to really have that focus in the time frames we had. We knew there were trade-offs to this. One of our engineers, who was incredibly talented, was more interested in systems work than in working on this problem, so he left to go and do great work elsewhere in the company. It was more important for us that everyone was bought in, even if it meant being an engineer down for a while.

[00:08:27] So we understood the problem and built a great team that cared about the users, and this helped us flesh out a few key initial hypotheses. Now we needed to test whether we were right or not.

Ranking hypotheses and the concierge MVP

[00:08:43] You're probably familiar with the standard iterative loop of product development. You come up with an idea, you build and launch it, and then you learn from it and iterate from there. We had an idea, but we didn't have the time to build it, so we needed a way to shortcut directly to the learnings so we could iterate faster. We needed experiments.

[00:09:08] We had all these hypotheses from our research, so we used a similar process as before. We sat down as a team, listed out all the ideas we had, and ranked them by risk: what's the likelihood that the product will fail if we're wrong about this hypothesis? Then we thought about the cheapest way to test each of them. The mindset we had was: be lazy. Don't do any work that doesn't help us de-risk the riskiest assumption.

[00:09:37] Sometimes the test was simple, like a survey. Some things we could test traditionally, with clickable prototypes, concept tests, those kinds of things. In the end we were left with the hardest thing to test: could we build something to replace spreadsheets and email? Everyone's process was a little different, and dummy data and prototypes weren't really resonating. At some point we realized that the only way we were going to learn more was by giving users a version of the idea to use with real projects and real data.

[00:10:13] This led us to the idea of a concierge MVP. We needed to learn if our hypothesis was correct, we needed real data, and we needed a hundred percent reliability: no buildings falling down. And we needed to do this quickly. So how do we validate a fully functioning product without building the machinery behind it? For those of you who've seen the show Futurama, you know how the robot Bender refers to humans. So we created Project Meatbag. We would be the machinery. We would sub in humans for the technical parts of the product and test it without writing any code. Our plan was to slowly replace the people in the process with code over time, as we validated ideas and built the product in parallel.

[00:11:04] We found five customers that represented our target audience, and we sent them a pitch saying, "Hey, we've got this great new submittals tool. It's going to effortlessly handle your submittals process, track input and give you a report every week." And of course, we were the tool. Each of us took one of the user projects and did all of the data entry and all of the processing. Users sent us their existing spreadsheets and forwarded us all the email responses and files that came in.

[00:11:32] This was v1 of our product: a pretty simple Google Sheet mirroring the system that we wanted to build, populated manually. We moved all their data into our template, we kept track of everything that came through email, and we looked at the files to figure out status. There's some basic automation here, you can see colors and highlights, but fundamentally it's a spreadsheet. From there we continued to automate things in the spreadsheet as we learned what the users needed: a lot of conditional formatting, automatic dates, data validation in the columns, really anything to reduce our workload. But by and large it was still a spreadsheet maintained by our team.

[00:12:14] The critical thing here is that, for our users, this functioned like a product, and a fairly magical one at that. They sent us data and it appeared. They didn't have to do any of the things we knew were pain points in their existing process.

[00:12:28] Then we set up a dashboard driven by these spreadsheets. It was there to provide insights and next steps, a critical hypothesis of our product. It updated in real time, driven by a bunch of queries; really, Excel scripts were the first code written for our product. This was our first major breakthrough that showed us we were really on to something valuable, as a lot of our users were genuinely shocked by what they were seeing. So much so that they thought we were wrong: there was no way there was stuff that was six months overdue that was so critical. But when they investigated it, they realized it was true, and they had potentially millions in cost overruns at risk because these things had been missed.

[00:13:14] Another thing we did was have a Photoshop template that we would manually populate every week, to send them a report that they could put their logo on and share with architects, owners and other stakeholders to keep everyone updated. This was another really valuable element that we learned about. These three things, the spreadsheet, the dashboard and the report, were all pillars of the solution that we wanted to build.

[00:13:43] Throughout this process we met with the users every week. We went to see how things were going and got the feedback, and because all of this was manual, we were able to take that feedback and immediately iterate on the format and the data, creating that tight loop we were looking for. Now, obviously, people are going to love you doing their job for them, but we needed to turn this into a software product, not a concierge product. So we had a few constraints: we made sure we only did things in a standard way for all our customers, and we only did things that we knew we could automate in code.

[00:14:20] How did this work out? The outcome was over 600 submittals tracked across five projects for four months. This included a half-a-billion-dollar highway interchange in California, where they estimated that using our process saved them several million dollars. This was way more than we thought we would do in the process, and we learned so much from it. We were able to validate our product solution and quickly iterate while real customers were using it. This meant that before a single line of code was written, we knew exactly what needed to be built and how it should work. We also built deep customer knowledge and empathy on the team, which meant everyone was focused on the details of the problem.

The clone question and the submittal log

[00:15:06] That led us to something we didn't even expect. During our customer discovery process, one of our engineers spotted an opportunity that set us out on a completely new journey as well, so let me tell you my second story, about that. Something you need to know about construction is that if you ask people there, like Ahmed, what they hate about their job, they'll think about it and say, "Nothing really, I love my job." They are wildly accommodating people. Ahmed was telling us this having just walked us through one of the most terrible admin processes you've ever seen, which he has to endure every day.

[00:15:48] After a bit of experimentation, we finally landed on a question that helped him and others reveal what their true pain points were. We'd ask him, "If you could clone yourself and convince that clone to do the part of the job you liked the least, what would you have that clone do?" This opened up the floodgates and helped us see what we should really focus on to improve their lives. Next time you're trying to figure out how to improve someone's productivity and you feel they're being way too accommodating, I really recommend asking this question.

[00:16:20] Almost unanimously, they mentioned something we hadn't even thought about before. At the start of every new construction project, someone gets the incredibly crappy job of having to read a three-to-six-thousand-page specification document and type out every single submittal required, creating what's called a submittal log. This is incredibly tedious but vitally important work. On average, working 10-hour days, it takes someone two weeks to complete. It's a task so hated that it's often given to the person who annoyed the boss most on the previous project.

[00:17:02] Once we'd heard this a couple of times, one of our engineers said, "Hey, this looks like a solvable machine learning problem. I don't think it'd be that hard to do." But we had a challenge, because we had no ML expertise on the team and a crazy tight deadline to meet as it was. This was outside our initial assumption of scope. It felt like a huge risk, but also a massive opportunity.

[00:17:26] So we wondered, can we test whether customers will find this valuable? Luckily, we'd already worked out a process to answer that question. Enter the meatbags. I sat down with the engineers and wrote out a human-readable set of instructions for how to extract submittals from the specification doc. We set the constraint that these instructions had to be simple and able to be followed by any competent human with no prior knowledge of construction. They also had to be instructions that could be easily replicated by a relatively simple machine learning algorithm.

[00:18:03] We then reached out to a few customers and said, "Hey, we have this fantastic new tool that extracts all of your submittals for you. Just send us your specs and we'll send you back a spreadsheet. Oh, it's still an early-stage product, so it might take a couple of days to get back to you." Then we just hired some data entry contractors to do the work. This was really cheap: a couple of people at $20 an hour. They were simply picking up specs sent to an email address and responding with a spreadsheet of the results.

[00:18:32] The response was incredible. We literally heard shrieks of joy from people when they saw the results and realized they'd just got two weeks of their life back. This was really a double whammy, because not only did we validate the idea, we also learned incredibly valuable insights on things like precision versus recall. It turned out that people were okay with us listing more than they needed, but they were not okay with us missing any important items. We also got labeled training data as part of this process, all the stuff that was essential for us to be able to build and train the machine learning algorithm. All of these insights and that data allowed us to build the product faster and be way more confident that we had something the market really wanted.

[00:19:24] What were the results? With only four engineers, we were able to learn, experiment and launch a production-ready solution in six months. This was a paid add-on to our core solution that from day one accounted for nine percent of the company's revenue, evidence that we'd built something that the customers really needed.

[00:19:46] One of my favorite things about this was that our product was featured in a list of the 13 most interesting advances in construction technology that year, alongside this robot that builds walls. What I love about this is that the robot is an incredibly complicated piece of technology solving a relatively simple problem for humans, whereas we built some relatively simple technology to solve a very complex problem for humans, and initially we actually used humans imitating machines to power it. That made us feel very smug.

What we messed up

[00:20:22] Now, this wasn't all sunshine and rainbows, so I want to share a few of the things that we messed up, so that hopefully you can learn from them. The first was that we didn't have an exit strategy from our concierge MVPs. The idea was that we'd start tracking submittals manually and then quickly replace ourselves with code as the engineers got up and running: maybe one month, two months max. Then, as things do, they got delayed, and we were left hand-cranking this process for almost four months.

[00:20:56] We didn't want to leave our customers in the lurch, as these were mission-critical processes that we'd promised our customers we'd support, so we were basically doing two jobs at the same time. It was particularly stressful for Heather, who was supporting that half-a-billion-dollar highway interchange project. So if you're going to do this, think about the exit strategy. If things get delayed, or you decide not to pursue this product, how are you going to do right by yourselves and your customers?

[00:21:31] Another thing was that we did a great job of involving the engineers in the early research, but once the concierge MVP started, we decided they needed to be focused on writing code and we'd share the knowledge with them later. At that point we could already see the cracks starting to form in terms of customer empathy, and that led to mistakes that cost us many weeks, which could have been caught way sooner if the engineering team had had first-hand experience of managing a customer's submittals. If we could do this again, we would have had the engineers spend at least a couple of weeks being the human in the loop for those concierge MVPs as well, as that would have saved us a huge amount of time in the long run.

[00:22:19] Lastly, we only brought in marketing and sales at the very end of the process, which meant it took much longer than it should have for them to understand the value propositions, and therefore how to position and sell this in the market. Having at least the product marketing manager involved in the discovery and experimentation phases with the rest of the team would have saved us a huge amount of coordination effort and rewrites when it was time to develop and launch the go-to-market strategy.

Recap

[00:22:50] To recap, if you want to do experimentation in zero-failure environments: build a cross-functional team that truly cares about the customer, learn about the problem together, treat everything as an experiment, do the least amount of work needed to test your hypothesis, and use humans to power the first functional MVP.

[00:23:16] The last point I'll leave you with is that experiments come in many shapes and sizes. You can do this at any stage of development, be it a new idea or something that's been in the market for a while. I encourage you to think about what it is that you're trying to learn next, and what's the least amount of work you need to do to learn it.

[00:23:37] Thank you so much for your time. Please feel free to reach out to me with any questions or comments, or if you'd just like tips on how to implement this for challenges of your own. I also offer free coaching to people who are just starting out or looking to transition into product careers. If that's of interest to you, feel free to reach out to me any time by email or on Twitter. Or, if you want to join me here at Asana, you'd be very welcome; I'd love to chat with you about that as well. Thank you so, so much, and hopefully see you in New York.