The Product Trust Gap

Sam Bradley at UXDX EMEA. Video: https://youtu.be/adcBv53ixlM

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

The Expedia timer: my first experiment

[00:00:08] Thanks so much. What I want to talk about now, which is a bit of a continuation from the personalization talk that we started with, is this gap in trust that I see with product. I'm going to start with a story. The very first experiment that I ever ran as a product manager was at Expedia back in 2014. It was a flights checkout page where we ran an experiment to put a timer in the top right corner.

[00:00:40] This was a huge winner, actually. This far down into the funnel, it's very, very challenging to actually change someone's behavior. So we ran the experiment, we had a pretty strong conversion winner, multiple millions of dollars of revenue that product teams tend to bank when you annualize it. And so we went to product review, shared, everyone celebrated, great. We rolled it out and moved on.

[00:01:01] A few weeks later, we started to get feedback from consumers, either through the site feedback at the time or calling into CS, which we had not actually analyzed before making the decision to roll out. And what happened was two things when consumers interacted with this. One, it wasn't real. This was a timer for 20 minutes. When it expired, it essentially just disappeared. If someone just left the tab open, it didn't do anything. But the people that did interact with it expected that this was maintaining this flight for them. Very similar to Ticketmaster, other sites where you have these tickets briefly, and we're going to hold them in this amount of time.

[00:01:45] All Expedia is doing in this case is calling into global systems for booking inventory. And at the time in the 2010s, about 4% of the time there would be a price change when you go to book. So we had convinced people that they have this amount of time to make a purchase, and then the people that actually then had a price change were irate at us, because this was non-functional and it didn't do anything to build up confidence or trust.

[00:02:09] The signal problem here was the conversion is an incomplete story. Very similar to what we were talking about, very often we are able quickly and easily to measure things like conversion, a two-week test, how often is someone booking, how many new accounts you get. But what that does not capture is the friction that you created, and the pain that someone may have had, and the frustration that they had with your experience. Whether they booked this one time is one thing. Are they ever going to come back is completely another. And how do you actually start to measure that?

The cycle of distrust and the Edelman data

[00:02:43] What I want to do is talk a bit about just this notion of trust. There's some very interesting data and trends that I think broadly relate to everything, and then we'll go into a few case studies that I've gotten to work on around a very light framework that we've been able to work on.

[00:03:01] To start out, there is a book from 2010, if anyone recognizes this, from Dan Ariely. This book is Predictably Irrational, and Dan has a chapter all about trust in this, and it's called the cycle of distrust. Dan's point is that people are not trusting by default. Trust has to be earned, but it really does not take much to break that trust. Even a few bad actors in the entire market can ruin the trust of this market or experience or product type.

[00:03:36] Think about a few examples. Crypto, granted, might be more grifters than not, but I think our collective societal reaction to crypto is based on what you see in the news, people going to prison. How much trust is there that this is a benefit? Even Airbnb, it's changed the relationship with so many communities on how people interact and take holidays, that there's a lot of frustration with every other company that serves the same space of vacation rentals and holiday homes. Actions from just a few can really shape how we collectively think about a relationship with the business.

[00:04:14] The Edelman Trust Barometer is something that comes out every single year and it talks about broadly where trust is as a topic. We are entering, I think probably no surprise, this insularity phase. Edelman this year talked about how 70% of people essentially don't trust people like them. So we're all going into our echo chambers broadly. This doesn't necessarily relate to anything with business, but we know we are not trusting of people with different ideas and interacting.

[00:04:43] This room is made up of lots of people from all over Europe and beyond, as well as locally. There's a lot of breakdowns in the Edelman Trust Barometer by country. I think we would guess that many of the countries in Europe rank very low on being outwardly trusting of both institutions and other people. We just know that there's a lot of headwinds, as you're working on products, of what is someone just inherently going to default trust, especially if it's from another country.

[00:05:15] Germany ranks very low, second to last, on trust of international companies. I stand in Germany representing PayPal, an American company. I think it's an important lesson to just think about where are you building products and to whom, and what is going to be the assumption of what you're starting with.

[00:05:33] One final data point from the US, this is two years old. Trust in America with big business has hit essentially an all-time low. So over the last 25 years on the chart, there is just a decrease, an eroding trust of big business, major corporations. Some of these inflections are notable points in history over the last 25 years, but essentially what it's summarizing is that trust overall is eroding. So we don't trust big business, we don't generally trust anyone, it's easy to break that trust, and we're going further into our little echo chambers and cycles where we don't necessarily want to interact with anyone else.

Reviews: the trust signal that flipped

[00:06:12] That's super challenging to build products then, obviously. I think a very common trust signal that we have always collectively used within products and building products and experiences is reviews. I was able to work on reviews at Expedia several years ago, which I found super fun. I'm a review skeptic. I hate filling out reviews, and I like building products that I wouldn't necessarily want to use.

[00:06:36] At the time, reviews were the lifeblood of travel. It was a stat that I used, I think, in every single user story that we wrote for a while from 2014: 88% of people trusted online reviews as much as a personal recommendation. It was essential, especially if you think about vacation rentals or even brand new properties on hotels. How is someone going to make the decision that they trust this? If they see reviews, they're going to be a lot more trusting of what they see.

[00:07:06] This is completely flipped. Over the last five years, it's upside down. It's essentially now less than one in two are going to trust this as much. And I think there's been a rise of reviews being more skeptical. There's been articles coming out of people who have paid for reviews. The entire market of it, I think, has left this challenge of, is this the right signal for people?

[00:07:33] What reviews are trying to accomplish is essentially, is my experience going to match my expectations? And so if there's a lot of reviews on a product, I think what people are after is, I want to find someone that I think is like me and the experience I want to have, in order to make this decision. And so the very specific granularity that I want is something I want to be able to control. Obviously, an hour ago I talked about filters and sort, which I think is a big part of what I want to be able to see: things that are exactly like me.

[00:08:05] We have this rise now of these AI summarized reviews. This example is Tripadvisor. This is The Savoy Hotel, which is in London, very iconic hotel, nearly 8,000 reviews. The summary doesn't give me any information and it's very, very positive. I've had a conversation with multiple people today about AI and how complementary it is, I think, in most situations and most ideas that you go give it.

[00:08:34] The same sentiment analysis that is put on these, I question what is the value of what we're getting out of this. If there is a mix of reviews in any product, is it going to over-anchor on being positive because it is here to sell you? And you have to dig further in to go find the granularity and find what I would say is, I want to see people like me. I want to see someone traveling with a small child so I can make my own decision instead of trusting the summary.

[00:09:02] And this isn't just a problem with Tripadvisor. Obviously Amazon does this and many, many sites have started to do this. An example from this last weekend: we were traveling in the north of England. I was looking for a place to find breakfast. And Gemini gave me this gem, which is that reviewers find this cafe excellent sandwiches, great selection of sweet treats, friendly staff. Other reviews mention the staff can be super unprofessional and food is tasteless.

[00:09:29] So the challenge is, what is it actually doing that's solving a problem? AI is getting very, very easy to use and incorporate into your products. But is it helping us have more trust in the products that we are building, helping us help people trust what we're trying to convey to them, over what we've had for the last several years?

A framework: drivers and detractors of trust

[00:09:52] Let's step back into a framework. Over several years with Expedia, we were building this notion of a trusted marketplace, and I worked very closely with a researcher to define, how would we actually measure trust? I just laid out all these headwinds. This is super challenging, but how do you actually compare trust as a metric when you're then looking at overall conversion?

[00:10:14] What we did was start to look at what are the drivers of trust that someone would have, agnostic of the company and product. On one hand, you have just the core basics of, is it secure? Can I trust it? Is it reliable? And that moves further into, do I have a connection with this brand? Do I feel that there's a human involved, or is this company looking out for me? Is there a benevolence feeling of, I think they're going to go out of their way to do the right thing here?

[00:10:41] And Sarah, who now actually we get to work with again, defined trust as the willingness to rely on others in the face of risk. Depending on the product you're working on, there might be a pretty low bar of what someone has to get over to purchase a sofa, but this could be a lot bigger depending on using a new site or deciding if you trust it or not.

[00:11:01] On the other side, there are many, many detractors of trust. A few examples that we started to incorporate into how we're measuring interaction with this marketplace: from the minor side, feels very transactional. I never actually talked to a person. I was able to rent this property, check in on my own. There's not really this feeling of reciprocity, maybe. The further up this goes, it could be bad intent. It could be, this site is wildly unreliable. And then worst-case scenario, your information is stolen and you get hacked.

[00:11:35] In the US this is essentially a constant. We just get a note that says your information has been hacked, here's two free years of security tracking now. The more and more that this happens, we become a little numb to it, but I think broadly you are not going to return to some of these sites and use them if you have other alternatives. This is going to break your trust in the product and in the company where this experience happened.

PayPal: built on security and reliability

[00:12:00] Let's walk through a couple of examples. Obviously we've talked about PayPal several times. The entire thesis of PayPal over the last 25 years has been on trust. On PayPal it is very much on the side of, this needs to be secure, it needs to be reliable. That was what online purchasing was built to do. So if I happen to be on what seems like kind of a dodgy website buying a gigantic skeleton, because I have a five-year-old and it's debatable who's more into the skeletons, I don't know if horrorshop.com is the best site to use, but I feel okay when I see PayPal, and I know if something goes wrong or I don't get this product that PayPal will have my back. And I'm at least giving my information just to PayPal, not necessarily this site that I haven't ever used.

[00:12:45] Now, PayPal has expanded with this idea into other products. If you think about peer-to-peer, I need to settle up with someone, there are hundreds, sometimes thousands of people with the same name. Can PayPal help me find the right person? Whether that's someone in my contacts or I can send them the link, or other ways that we are always building products to try to simplify the trust that they need to have in the job that they are trying to solve. This is actually a better case of, some people have photos, some don't, but how do I find the right person?

[00:13:17] And I mentioned this earlier, but PayPal is now expanding into lots of other adjacencies of businesses. So what else can PayPal offer consumers without ever breaking that security and reliability? People expect that their information is always safe and that they're always in control of what they're able to do.

[00:13:37] And then at the same time, PayPal is looking at opportunities like expanding into a rewards program that we've launched in several markets, being able to buy and sell crypto, apply for credit, use buy now pay later. So there's a lot that is us using the information that we have with consumers to hopefully represent products that are correct for them and a good fit. We talked a lot about this an hour ago, of how do you let someone have control over that or opt out of that use? It is something we're always designing and trying to improve, the access within the app quickly to have more control over that.

Stitch Fix: benevolence, connection and the gray sweater problem

[00:14:15] On the other side, I very quickly mentioned an example from Stitch Fix. I'm going to go a lot deeper into that. Stitch Fix as a company is on the other hand of these drivers of trust. The entire company is built around this notion of benevolence and connection. For anyone that isn't familiar with the company, it's based out of the US. There were other markets, I think it's now down to just the US market.

[00:14:36] It is essentially one of these companies that takes an incredible data model, onboarding experience, collects everything they need to know about you, and combines it with a human stylist to send you clothes at home. You get five items in a fix, a box. You try on, decide what you want, send back the rest, and are only charged for those items that are there. But this feeling of someone did this for me is core to the thesis of Stitch Fix.

[00:15:04] I worked at Stitch Fix several years ago, and a lot of what we were building on top of is this feeling that people have, this connection that they had with their stylist. You have a team of stylists who have access to inventory, and based on what you describe you want and are after, and your previous experience, they recommend new clothing pieces for you, and then you decide what you'd like to keep.

[00:15:26] The interesting challenge and tension with Stitch Fix was that the data models were actually better at this than the stylists. The association was always with, oh, I have a stylist, I've interacted with them for four months, six months, a year plus, and they always know what I like. But more and more, the company has moved to data models that are selecting the clothing items, even not having a stylist interact at all, and sending you the box.

[00:15:55] So when it goes right, great. The stylist is still getting the credit. You still feel this human connection. But when it goes wrong, you are blaming this human connection. You feel like, how did you ignore exactly what I wrote in this? There would be notes we'd get of, I don't need anything specific, but no jeans, please. And at the time, and this is crazy to say with the rise of generative AI over the last two to three years, but the model that was grabbing what we'd send as a preview could not read that free text. So someone might say, please no jeans, and they'd have a preview of 10 items that was made up of six or seven pairs of jeans.

[00:16:32] Because jeans always do well, and we generally are more willing to buy additional pairs. And like any product, we might not have made up someone's entire closet, so you know a handful of items that someone liked. You don't know what else they have. There's no ability to match those pieces. And so what the model is trying to do is get to decent revenue and keep rate. What is the likelihood that someone's going to keep an item?

[00:16:58] One of the problems that we worked on that ended up being a trust eroder was what we like to call the gray sweater problem. Over time, somewhere between four and six, seven fixes, which is the subscription that you get every single month, everyone ended up going to essentially the same problem that we had coined as the gray sweater problem. Because it turns out we are all quite basic and all have the same style, and the models eventually recommend over and over a gray sweater or jumper.

[00:17:32] And the reason is because these are very likely to be kept. It's a staple in a wardrobe, it's not overly expensive. It was very, very common that this would be kept even if you had had something like this in the past and kept it. So the model would recognize patterns, and I'll speak for myself but I think it's very common, if I find one thing, I will then go try to find other colors of the same thing if I like the fit. And so this pattern would be picked up by the data models and you would have a recommendation to be very similar, or maybe a different color.

[00:18:05] But gray sweaters ended up dominating after a certain point, and so we saw this churn model happen right around five, six, seven months. One of the ways that we were trying to work on this is expand someone's style. Essentially ignore what the recommendation engine is telling you and push them into something else. Start recommending them yellows and different experiences than they would have had. But the tension there is that those experiences aren't necessarily optimized. So we're still driving churn occasionally on some of the tests we were doing, because it was not optimized for what the experience was.

[00:18:40] But the problem is you get to this point where you keep getting similar clothing items, they're fairly bland, and then you are blaming this human stylist that you think is ignoring what you have told them and what they should know you're actually interested in. Even though, because everything is so optimized, the inventory available to a stylist, the amount of time they have to grab the items, all of it is optimized with different models. A lot of times this was just cornered into something that you couldn't control and everyone ended up with the same experiences, and eventually churned out.

Front-line teams, and what AI does to them

[00:19:15] One of the great things that Stitch Fix had, and still has, is an incredible customer service experience. In these scenarios, when this would happen, and someone feels like they've been ignored by a human, if they called in, if they emailed, there was an incredible team that was local, very, very engaged, and would make it right almost always.

[00:19:38] This became an example that we saw, and I've seen this example in multiple companies, where your front-line teams are essential in any building of trust. In the perfect scenario, nothing happens that's wrong, you have a transaction with a company, there's no issues, and you never have to think about contacting them. Great, no returns. But that's fairly transactional. You haven't really had to go out of your way, other than they have an item, I have a need, we exchange this, it's a transaction.

[00:20:08] What we've found in a lot of research is that when something goes wrong is actually an incredible trust-building moment. It's friction, there are issues, someone gets frozen out of their account, someone's account gets hacked, the wrong item is sent. All these things can happen, we know, but depending on how your team handles it, how the company that you work for and how your CS team is able to fulfill this, has a lot to do with building loyalty. People are very, very willing to give people the benefit of the doubt if they can be made whole. A huge part of this that we found is these moments are actually trust builders if handled well.

[00:20:51] But that obviously leads to an interesting scenario that I think we're going further and further into, which is, what is your CS department made up of, and how much of it is still talking about humans? This is an example from Virgin. I've unfortunately had this several times recently, where you have to play the game with a bot to try to get to a phone number or a person to interact with you. And not all of these are great. I'm sure they'll get better over time. But you start to think about the products that we all interact with every single day, and how many of them are driven by AI, and we don't necessarily have an ability to control what we're seeing when we have a problem and when we have a need.

[00:21:31] One of the biggest things in terms of building trust, and the case studies and anything that we're talking about, that I've started to think about is: we all work in technology, we have an interest in this. I would argue we are all early adopters. Not everyone is. And so as we are building products and designing products for consumers, not everyone is excited about the tooling and what AI can do for us. I kind of add myself into that camp as well, where we'll embrace it, but I do think that there is this dystopian version of it that's just challenging and full of friction, of what someone was able to do easier without it now being optimized.

[00:22:09] And the last thought I'll leave here is, as we think about building products, trying to measure trust, things that are successful and not, we have a unique moment right now where there's as many as four different generations that might be using your product. You could have Gen Z all the way up through baby boomer depending on what the product is. Do you think all of those generations get along and hold the same beliefs and expect the same thing out of products that we are building, in the companies that we work for, on either saying everything and taking a stance or saying nothing?

[00:22:45] Do we think that even within the same family everyone necessarily agrees on what we're building, and what the role of AI should be, and how much of your data should be used to tailor experiences? I do not have a good answer on that, but I think it's a very challenging thing to think about, of, are you clear on the exact target market we're after? Is there an important stance to take with the role of AI, and at least disclosure and being transparent of what is built into the products and what control someone has over their own data and the experiences they see? That's it. Thank you.

Q&A

[00:23:27] Host: Thank you very much, Sam. I was personally really impressed and surprised with the change, the drop in trust in customer reviews from 88% to...

[00:23:41] Sam: To less than 50, yeah.

[00:23:44] Host: That was something I was not expecting. So let's look at a few questions from the audience. How do you measure trust apart from ratings? What KPIs, what metrics are you looking at?

[00:23:55] Sam: Yeah, I mean, this is still a very inexact science. I think a lot of what we've done is consistent quantitative surveys. You know that people get into friction within your experience. Is there an outlet where they can share information? As well as long-term studies. There are still people who are more willing to use your products and give you feedback, and so this isn't necessarily the wider population, but a consistent measure of, what was the experience of this? Did you have friction? What did this lead to? Would you use it again? I think it's just kind of a constant thing. And then we're really looking at those drivers and detractors and just trying to get almost to a metric. But it's case-by-case in terms of the product.

[00:24:44] Host: Right. And also what we were discussing earlier, what you mentioned about the disengagement. So, stop using a certain feature, or using it less and less because I don't want this here or I don't need this here, and people opt for something else.

[00:25:00] Sam: I mean, in a window of, you haven't seen engagement from someone for four months, it depends on if I work directly on the product or not. I would argue, oh, they don't have a need, or no, they have probably left. And I think what we're stuck with then is looking for patterns, because someone is not, on exit of not interacting, telling you exactly why.

[00:25:19] Host: Try to understand why, yes. In a world with increasing expectations by leadership regarding the speed of delivery, great one, any tips on how to successfully lobby for longer lasting A/B tests to ensure long-term effects of changes?

[00:25:33] Sam: So many opinions on this. Yes, we can collect a lot of data quickly, but I one time had a CFO tell me, if he believed everything a product manager told him, none of us would ever have to work again. So I think there's this notion of, you take this snapshot in time of your data and then you are making this claim of, extrapolate that out forever and nothing else has changed and I'm sure that this is consistent.

[00:26:02] Sam: Holdout groups is a big way that we've done this. It is a hard argument to say, let's run this test, or this product, for six months before we fully commit to it and roll it out. We had this argument when we launched this reviews product in the UK recently. What we've started to do is look at a holdback group on a feature level. Feature level works really well because they can still accomplish the task. You're not holding back value from the consumer, it's just, hey, we've improved search. You can still find it, but this one group maybe is now this collection of experiences that is separated, and what it does over time is start to show you what might have been noise and what is actually beneficial. So one way is holdout groups.

[00:26:45] Host: Thank you. How do you manage the conflicting needs of building customer trust versus the business wanting to be more efficient with AI? You gave some great examples in the talk.

[00:26:56] Sam: Yeah, I wish I had a good answer. I generally go back to this notion of storytelling. AI can do everything, but just internal storytelling: can you describe the pain that someone goes through of using the product? I think it's easy to skip and just look at data and just have constant decks talking about what could be an addressable market. And even if people tell you not to, just methodically talk about the pain that we are putting our consumers through to accomplish their tasks. It's not a story of revenue when you're doing that. I think it's a story of, go make them feel the pain, so that we drive collective empathy of, we need to fix this, instead of just using AI to be faster.

[00:27:41] Host: So put them in the shoes of the end user. If a company violated user trust, like timers in your example, what can you do to regain it? What do you say? It is so easy to lose it and so much harder to get back.

[00:28:01] Sam: I think it's variable to the person. Some people are going to be willing to try again. I think a lot of this has to do with, do you have a moat of what your product does, or are there other examples? If you don't like what you've gotten out of Zalando and you then go start shopping at other marketplaces, and I'll use ASOS, I'll use everything else, pretty easy. If you have this situation of, I've kind of invested everything into this company, it's a pain to now go open new accounts. I think there's going to be more willingness if the company has this acknowledgement and then is working to make it right. So it's inexact, but I do think that trust can be rebuilt depending on the response.

[00:28:42] Host: Depending obviously on the product that you're using and the needs you have for that. Love that you have the gray sweater problem[?]. Great one. So, last question.

[00:28:52] Sam: I am as basic as anyone. That's yeah.

[00:28:56] Host: You are, like, proudly average. So, last question. Not everybody needs AI, we are all in agreement here. What would be your advice on handling pressure from management and investors to implement AI everywhere and in every corner of the business? Because sometimes pushing back doesn't really work.

[00:29:16] Sam: I think most things internally in companies are narrative problems. If we just could rely on the machine, if everyone's agent just does this, then obviously there's no need for any of us to interact. I am massively skeptical of that, because I think most of this work is in these margins of, it's not perfect, and just because we can run a quick test doesn't mean it's going to work and that all of this will be here. So I think some of the speed of AI is helpful, to prove this is what AI was trying to do, this is what it recommended, it's pretty inaccurate. But broader than that, I just think what we have to do is go back to this idea of, let it be a tool that helps us accelerate the obnoxious parts, but we stick to the thinking and the empathy and building for consumers. Some of that fight is just going to always be there.

[00:30:09] Host: Thank you very much.

[00:30:10] Sam: Thank you.

[00:30:14] Host: Thank you very much, Sam. Give him a round of applause, everyone.