When AI Knows Too Much: Personalisation That Builds Trust Vs Breaks It

Sam Bradley, Pallavi Modi, Mihaela Draghici at UXDX EMEA. Video: https://youtu.be/40RLkRAG4e8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introductions: when personalization felt off

[00:00:07] Mihaela: Thank you very much. Welcome. Yeah, you can have a seat. I'm going to start the discussion by asking you to quickly introduce yourselves, and also with a first question, which is: can you think of a moment where personalization in a product you used felt off to you, and what were your expectations in that situation? What would you have expected differently? If you can start, Pallavi.

[00:00:42] Pallavi: Yeah, absolutely. Hi everyone, thanks for coming in and spending time with us. I'm Pallavi. I'm director of product at Zooplus. If you do not know Zooplus, it's a pet e-commerce company in 26 countries in Europe. We serve about 12 million active customers every year. I'm responsible for customer acquisition, and of course the retention part of it, and personalization plays a very strong role over here.

[00:01:11] And to answer your question, Mihaela, it's a bit personal. When I joined Instagram many years ago, the first recommendation Instagram had was my ex. Where I was like, how do you know that? [laughter] Exactly. And I think the question is not so much that personalization is creepy. The question is whether this is the right moment and context for the customer who's going to be consuming that information; that's what makes it creepy. So yeah, we'll discuss more about it today.

[00:01:48] Mihaela: That was definitely not the right context in your case. What about you, Sam?

[00:01:52] Sam: Hey y'all, I'm Sam. I work in consumer product at PayPal. Over the last few years I've led different parts of the app experience, and most recently I am now leading our core European markets across everything for app. There are many, many examples of when I think personalization goes too far. Everything that comes to mind of parents and my wife and non-technical people thinking their phones are listening to them, I think is very key. But recently we were working on a country move and had talked about the need to get a painter for our house. And as far as we believed, we hadn't typed that in. And then I saw a recommendation for a painter in a banking app that I use. So I think there's both what I would associate with not having given the information to that specific tool, as well as just being way too quick on exactly where our heads were. Which I think just shows the depth of the value that a lot of this can get to.

How a good recommendation becomes a trust problem

[00:02:57] Mihaela: Right, that's a very interesting perspective. And connecting to what you just said, in some cases consumers do not always trust or understand how companies or products know so much about them. So there is, yes, the angle of someone listening to us through our phones, but there are also other contexts where information is being collected. How can a good recommendation become a trust problem? Aside from this example that you just offered, can you think of other situations where this becomes a trust problem?

[00:03:33] Sam: Absolutely. I think it has a lot to do with the job that the consumer thinks they're doing. If you know you're giving your data and that data is going to then give you value within the tool, I think that's fine. If you are shopping on Zalando and you're aware that the more information they have about you, the different things that you've bought in the past, the better and more customized the recommendations for you, great. I think if you are in something like banking or medical, or different experiences where you do not expect to see this, and you see that they are using your data to recommend products, even when it is a fit for you, then I think it can break trust quickly: wait a minute, why are you using this data? And how often are people reading the full terms and conditions and what we're agreeing to with our data? I think it just stands out that we've been so inundated with it. You are just moving on and selecting and not even thinking about what the outcomes might be in that case.

[00:04:34] Mihaela: Right, thank you. What about you, Pallavi? When do you think relevance stops being helpful and becomes suspicious?

[00:04:43] Pallavi: Yeah, in addition to what Sam said, and it's really the right framing as well, the customer's job to be done is the most important factor.

[00:04:52] Mihaela: So the context in which...

[00:04:53] Pallavi: Exactly. And the intent. Sometimes it's just better to get out of the way of the customer as a form of personalization, rather than more and more, based on what the mental frame of the customer is when they're having that interaction with the product.

Signals that personalization is breaking trust

[00:05:13] Mihaela: From a product team perspective, what signals would you look at to inform yourselves and the teams that personalization is becoming a trust problem? Thinking of your products, or products that you have come across.

[00:05:31] Sam: Me? Yeah, I think there's a lot of data that we are trying to stitch together, because there's an imperfect story on where this might be coming up. If I ask everyone to raise their hands if you think your company sends too many emails, I would expect everyone listening is going to raise their hands. I think there's a nature of us so often pushing information to consumers. So seeing an opt-out is one signal, I think, but on a push notification that's complicated to stitch together. Did you opt out within the app of a consumer product, or at your OS level, where we actually then don't get that signal?

[00:06:11] A much more targeted way would be thinking about whether they have gone and changed the data controls within the app. And most consumer products I've ever worked on, it is not really the area we invest a lot in. So those controls can be fragile, not very up to date, typically a framed web view even if you're an app. And so when you see people both trying to get to that and calling into CS, trying to take away some of the access that a company is using, obviously that's a big, loud signal in that case.

[00:06:46] Mihaela: Thank you. What's your perspective on this, Pallavi?

[00:06:49] Pallavi: Before I answer what the metrics are, I would say we need to ask ourselves: are we playing the short game or the long game as a business? Because personalization is not there to improve today's conversion rate. I think that is the most important perspective. When we run experiments and we're running A/B tests, they're usually a window of two weeks maximum, and then we suddenly look at it: oh yes, conversion rate has gone up, clicks on recommendations have gone up, let's roll it out. But what it doesn't measure is, did we lose the trust of the customer over a longer period of time? And those metrics are different from immediate conversion metrics.

[00:07:30] Hence we need to look at, as I was mentioning as well, what's the unsubscribe rate? Are we seeing people engaging more with the self-service area in My Account, to look at where they can take away permissions? Are they calling the call center? And now we do have very strong GDPR in place: are they trying to look for that information within the My Account area? So I think there are other signals that become important. But one important thing that we as humans do quite often is, when we do not like certain things, we disengage. And disengagement should also be a metric for longer-term personalization, and not just short-term conversion improvements.

[00:08:18] Sam: Can I add on that?

[00:08:19] Mihaela: Yes.

[00:08:20] Sam: I think exactly as you're saying, we are looking for these signals in terms of the loudness of someone calling into CS or going and trying to control that. Most of the time, they're just going to stop engaging. And then how do you measure whether they no longer have a need, or they don't currently have a need, versus they're completely done with your product? You don't have a signal of it, other than eventually you fall into this lapsed and churned consumer.

[00:08:44] Mihaela: Yeah, so basically you're looking at the reduced engagement, or completely zero engagement, correlated with the loss of trust, and using it not as an immediate short-term metric but actually as a long-term view of what's happening to your product or to the business.

Industries where data use crosses a line

[00:09:05] Mihaela: You made the point that not every company has the right to use every piece of data. But even if they can, are there specific areas or industries where you think using data for specific personalization can become crossing a line?

[00:09:32] Sam: I think probably, as with the initial example with Instagram, depending on the stickiness of a product, there's a lot we will put up with. I think the recommendations, especially on social media, can get so powerfully strong, but we're probably more willing to accept them, because you accept, okay, I'm scrolling, they understand who my network is and the connections. There are so many other products that collect our data that we don't think about. A big one that comes to mind, that I remember reading about, was the Weather Channel, which is a huge app that collects all data across your app by default.

[00:10:08] Mihaela: That's something completely new I've learned today.

[00:10:11] Sam: So there are these two sides. There are ones where I think you clearly know that it's happening. I think TikTok is slightly frustrating with how much they know about me, but at the same time I use the product, so I'm accepting of that, and there are at least granular controls that in theory I could go and control. The ones that are just silently building this picture of you and selling off your data, I think it's way harder to know unless you go searching for it. And how often products end up pushing out updates on privacy and changing your defaults, I think, is another huge problem. There's been this trend of training AI with, now, oh, we're going to use all of your content from LinkedIn, all of your photos, all of this information, unless you go and explicitly tell us this new thing you don't want.

[00:10:58] Mihaela: You touched on a point that I want to bring back later, regarding users proactively searching for information or the products showing it. But I'll leave it for later, because I want to hear Pallavi's perspective on specific industries or specific products where using the data collected becomes borderline, crossing the trust line.

[00:11:24] Pallavi: I might have a bit of an extreme view. I think every industry has that problem. It comes down to how you're going to use that data, because we are all collecting data. And highly likely none of us would know what data we already have from the customer, where to locate it and what to do with it. We have started treating data with a FOMO, which obviously is quite problematic. That leads to this downside effect of how this data will get used should there be a leak. Even if you do not have a bad intention, it is happening.

[00:12:05] But to be very specific to your question, there are those industries. Health is that space. The news that you're consuming is another area. Your health trackers that you're wearing: I'm wearing this ring, and everyone here is wearing a ring or has an app, and we are always giving this information. Which is still fine, once again, if this information is being used for the context. However, now imagine that this information starts being sold to, say, insurance companies, who are then stacking up your insurance premiums, or it gets tracked as surveillance by countries to impose laws. Then that is when it becomes problematic. Data in itself is not bad, but what you do with that data is what makes it really, really bad and creepy. And hence I feel that we can't put some industries in the nicer bucket and the others in the not-so-nice bucket just yet.

[00:13:07] Mihaela: So everything is in one bucket for now. [laughter]

[00:13:09] Pallavi: Unfortunately.

How much to explain in the product

[00:13:11] Mihaela: Going back to what you also mentioned, Sam, about users' awareness of how their data is being used, or where they're being served personalized recommendations, ads, or different flows in their interactions. I know even from our previous conversations that you strongly see it as essential to show in the product where this is happening, what the privacy controls are, and what options the user has. What specifically should a product show to the users, and where do you think too much explanation creates friction or frustration, or even leads to the loss of trust? And what makes it enough explanation to give the users some information about what's going on?

[00:14:10] Sam: Yeah, absolutely. Even in the best-case scenario, if it is a shopping experience or something where the personalization recommendations are very good, I think it's still very necessary to explain where this came from. AI makes this very easy, but it's been fairly easy with a data set for a long time, if you know a little bit about a consumer, to say: because you liked this, we're going to recommend you this. But if you leave out the context of where this recommendation came from, I think naturally people, even if it's perfect for them, do not trust what they're seeing.

[00:14:44] So we've played around with a lot of tests just explaining exactly where something came from. There is a look-alike version of this: people who also shopped at this store might now like this other one. And then there's, I think, the more personalized case. When you share more of what's under the hood, what is actually powering this, you've got to have tight controls, I think, on the taxonomy: what is creating these collections or this headline, what fits into that, is that going to be completely accurate?

[00:15:13] The other side, I think, is that you are exposing that we are tracking your previous purchases or interactions in order to recommend. By exposing where this came from, I think naturally people will identify: well, wait a minute, I don't want you to do that, I don't want you to look at this. And I think especially, there's lots that PayPal is now expanding into, different types of businesses, and where that business can go. Similarly, and PayPal's not a bank, there are banks that are doing a similar thing, and a lot of banking has gotten into that. So whether you want it or not, you will see offers and ads in your banking apps, and there's not necessarily control over what they read versus don't.

[00:16:01] So I think when you push more information out there, I would argue it's going better for the consumer overall in the relationship, but it also might lead to opting out of the use of your data. That's the friction, I think, from the product perspective: share more information, bet that the longer-term trust is there, but we probably have a smaller addressable base of who will actually leave this on. That would be the tension point.

First-party versus third-party data

[00:16:26] Mihaela: Right. A question to you, Pallavi. Talking about this difference between, let's say, first-party data that the customer is providing directly, or third-party data coming from other sources that is being used: how does that change the trust dynamic between the user and the product?

[00:16:51] Pallavi: Very good question. If we forget for a second about customer and business, and we really think about us as human beings: you and I might know things about each other because we have shared those. You have found that out because we interacted. You probably have not found that information out from stalking. The same thing happens in a business context as well. When a customer is giving information to a business, there is an inherent value exchange happening: I give you some information, and in return I expect a better, easier, more relevant experience.

[00:17:35] When that information exchange has not happened between the customer and the business, the value exchange equation is quite unbalanced. And that is where the trust erosion starts happening, where you're thinking, wait a minute, how do you know this about me? If I'm watching crime documentaries on Netflix and then Netflix recommends one more to me, sure, makes sense. But if I've never watched it before, and then suddenly there are additional recommendations coming because I'm reading a book about some crime that happened, then it's a bit creepy.

[00:18:11] Mihaela: It's kind of visual [?].

[00:18:11] Pallavi: Yeah. I think that is where the difference comes in, and this also goes back to how the organizations are using this data. Do you have the right to own that data? And how far apart is the data from the customer, which is also a critical factor. What I mean by how far apart the data is from the customer is: am I giving you the data through explicit information that I'm giving by entering a form, clicking on certain links, et cetera, or are you buying this information from third parties? Which creates a friction point for the customers.

[00:18:52] Sam: Can I build on that?

[00:18:52] Mihaela: Yes, go ahead.

[00:18:53] Sam: In the rewards space, which I worked in, there was some due diligence we were doing on a company, specifically looking at personalization. The model that this company had was that they would reward consumers, essentially give them money, for sharing more information. The exchange was: give us more personal information, and we'll give you more money towards gift cards. So by default you know a whole lot about this person, but the consumers were very unaware both of what information they were letting their data go to, and of how long that access lasted.

[00:19:32] One of the tactics, and it was explicit, they were aware that they were giving this, was that you could sign into your Amazon account. If you churned out of this app, essentially used the money available to you and then never went back, there was no actual account deletion. They didn't actually have the ability to delete an account, so it would track your Amazon purchases in perpetuity. Even if you hadn't interacted with this company for two years, they were still tracking everything you do. So it's not just your address and your personal information, maybe how much money you make; they are actively continuing to use this information that you agreed to, because you don't go and read the terms. So I think there are a lot of ways that this can go poorly.

[00:20:13] Pallavi: Yeah.

[00:20:14] Sam: Regardless, yeah.

What product leaders should ask before saying yes

[00:20:15] Mihaela: It definitely could go wrong. I have one last question before we move to questions from the audience. Thinking from a product leader's perspective: if the team says AI can make this experience more personalized, therefore more profitable, what should product leaders be responsible for before saying yes to this idea and implementing AI personalization at scale? What should they consider as a priority?

[00:20:42] Pallavi: I would think about what the customer problem is that we are trying to solve. That's really the first job to be done as a product person. And how is this AI personalization going to make it easier for the customer? And the third important question to be addressed is: does it create customer value, or business value, or both? Because if we are leaning only towards business value and no customer value is being created, this is probably not a great idea, and we're optimizing for short-term gains here.

[00:21:19] And in the end, I think the most critical thing for us as businesses to continue to evaluate ourselves against is that what we are creating has to create long-term relevance. If that relevance is broken, if it's not happening, whatever metric we are optimizing for today becomes irrelevant, because trust itself is a growth metric that companies should leverage and measure. So yeah, that's how I would look at it and recommend teams look at it. Thank you very much. Sam?

[00:21:53] Sam: Yeah, I think it has so much to do, exactly as Pallavi's saying, with what the intent of what they're doing is. A very quick side story: there was this phenomenon a few years ago on TikTok called North Sea TikTok. It's still out there, but essentially there were these gigantic ships in the North Sea crashing into waves, with this very scary music playing in the background. And it exploded, to where everyone was watching this terrified, myself included. A large ship in the ocean is the most terrifying thing to me. So I watched it longer than I would have, and so the AI, without realizing, is just like, I'm going to serve up Sam a whole bunch more ships crashing and sinking.

[00:22:36] So I think there's this nature of: what are you trying to achieve? You could let AI go crazy, but is that what we want? Do we need to personalize this any more? We've used a lot of examples of selling. I go back to: yes, personalization can be helpful, but I think what a lot of people want is filter and sort, and their own ability to control this. So you've got to be really controlled, I think, on what you're allowing it to do.

[00:23:00] Mihaela: That's a great point to end with, actually: giving the users that control to build the trust, so they feel that they can trust the product.

Q&A

[00:23:07] Mihaela: I'm switching to questions. I don't know if you can see the questions on the big screen, but it would be great. Thank you very much. Let's look at the first question coming up from the audience: how much do you track about us that we don't know about? [laughter] Who wants to go first?

[00:23:31] Sam: I will take that. From PayPal's perspective, financial services is so tightly regulated that it is very, very easy to identify what we track about you. Purchases are obviously there, but any use needs e-consent. So I think there's a lot that is controlled. Across the internet, I would not recommend going deep into it, I guess. I think it depends on the state of the company and the industry, but there is a lot that's the Wild West.

[00:24:01] Mihaela: What's your view on this, Pallavi?

[00:24:04] Pallavi: Well, as I said, everyone who's collecting data is doing it with a FOMO: we should not miss this click. So anyone who says we are very careful is probably embellishing the truth a little bit. But I do believe that the regulations in Europe are so, so strong that we have more control in cleaning up the data as well. So, yeah.

[00:24:37] Mihaela: We talked a lot about different contexts where we get personalization, and here's a really good question coming up: when do we not need personalization? Can we live without it?

[00:24:48] Sam: But follow me now, figure out what both of us can travel at one point [?]. I think we would go through this in the past: trying to recommend someone the exact type of hotel that they booked six years ago, and there is no understanding of whether that was a business trip and this is now you by yourself. Was that when you were in your early 20s, and now you're traveling with a child? I think it's personalization without constant use. For any purchase that is not constant, I think it's challenging to claim that you know everything about them and you know exactly what will be there. And then if you have consumers that are trying to unpick and get away from these recommendations, that's the challenge, if we're guessing at the personalization you want.

[00:25:30] Pallavi: Yeah, I think we have come to a place where personalization is inherently there. And the best personalization is when you don't even notice that it is there, but it's just so easy and everything is flowing and you are enjoying yourself, and suddenly two hours have gone by on Instagram doom scrolling. So it's really hard to say when we do not need it. But as businesses, I would say there are key moments in the customer journey that we can identify and say, okay, here we back away. If the customer is talking to the call center, or in the pet care business, if say someone suggests "my pet has passed away, this is why I don't need more food," this is a very good moment to back away and let the person be, and maybe send them flowers, rather than saying, here is a 15% discount when you get your next pet. So there is some basic sensitivity involved as well.

[00:26:28] Mihaela: A really good example of what not to do, yes. Could you share a real situation in which you challenged a feature or rolled back a feature that seemed intrusive to customers? How did you identify it as problematic, and how long was it live before it was taken down?

[00:26:49] Sam: In about an hour I have some other similar topics to this. But a few years ago I was working at Stitch Fix, and there was this experience where fairly reliably you could see that someone may be building up towards new clothing for being pregnant. There was a test run where that was inferred, and people had a very big problem with that, as you would imagine. Luckily in that case it was pulled down after quick feedback. But I think there's this nature of: the data tells a clear story, and even if that story is there and accurate, people don't want it, and they want more control.

[00:27:31] Pallavi: I don't have a specific example from my experience, but I do remember one experience that someone had shared, very similar to what you mentioned, Sam. This person was using a period tracking app, and they logged that they were pregnant. However, after a few months they had a miscarriage, and because they were going through the trauma of the miscarriage, they did not update the app. And after nine months the app sent a "congratulations on your baby" message. That was such a heartbreaking moment, of course, for the person, but also a very good sign for the app to treat disengagement as a signal to stop interacting with the customer. I really hope they took it off. But that's a very good example of the trust erosion that we were discussing earlier as well.

[00:28:29] Mihaela: And how quickly that can...

[00:28:30] Pallavi: Yeah. Yeah.

[00:28:31] Mihaela: ...can be gone. If you added a personalization button to your apps, would people select that? Sorry, a "no personalization" button.

[00:28:43] Sam: We're in a room full of designers. If you put a button on anything, you can get people to click it; I think we know that. [laughter] But absolutely. I haven't worked on something where it's this extreme, but I do think there are controls, like default sort, give me it by price, or give me more control. People will absolutely go and find that, and the more you make that easy for them to do, the more they will get into their own control.

[00:29:09] Pallavi: This seems like a very industry-specific topic as well. I think I would definitely click on no personalization on news, for example. I think there was a question similar to this, about a lot of polarization happening because there's an information bubble that we often fall into, like crashing ships. That's a very good space not to have personalization in, and to really have information from conflicting viewpoints as well, to really...

[00:29:38] Mihaela: Have the diversity.

[00:29:39] Pallavi: ...in the information you receive, rather than being fully focused on building one message.

[00:29:47] Mihaela: Is profit the primary driver pushing data collection toward dystopian outcomes?

[00:29:54] Sam: Yes. [laughter]

[00:29:56] Pallavi: I mean, that's a bit of a rhetorical question.

[00:30:01] Sam: Possibly the easiest question we'll get, yeah. [laughter]

[00:30:03] Mihaela: One-word answers only.

[00:30:03] Sam: Absolutely. Yes.

[00:30:07] Mihaela: Do you think that consumers should be paid for providing all the free data, since companies are earning from it?

[00:30:17] Pallavi: Yeah, yeah. [laughter]

[00:30:18] Sam: I actually know some people in the States that used to work on a product built around data and selling, and then started a new company where you can essentially get a share of what a company is making off of you. I'd have to check how that was doing, but I think there are people starting to play in that space, where it's about giving you more control. And I think with the rise of technology being constant, something like this can happen. If we just refuse to engage, then there's going to have to be a different way that companies get access. So someone should build it.

[00:30:51] Pallavi: Absolutely. Getting discounts is another way of getting payment, no? And we willingly do that. We do give information: if you sign up, you get 10% off. Sure, we're doing it. So I think this is already happening in different shapes, ways and forms in different markets.

[00:31:09] Mihaela: I guess it's also a way for people to get more awareness, and maybe even start asking these questions, or asking for these benefits. Because the mention here is also about dark patterns, where you're opted in automatically and you have to opt out and you're not being informed. So people should also try to get themselves informed and see how they can benefit from certain things, or even ask for it directly. I think we are good for time with our questions for today. I really want to thank you. This was a useful and really interesting perspective from both of you. Thank you very much, and I wish you a great rest of the day.

[00:31:53] Speaker 1: Thank you, Mihaela. Moving on to the next topic. [applause]