Human Centeredness At a Time of AI Amplified Business Metrics

Pamela Mead, Ricardo Marquez at UXDX EMEA. Video: https://youtu.be/018Hq2IBUu8

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Unintended consequences and the speed of AI adoption

[00:00:01] Pamela: All right, good morning, how's everybody doing? We weren't sure if everybody was going to be here and it was going to be full, or if it was going to be empty, so we're really excited to see everybody here today. Since we've already been introduced: I'm Pamela.

[00:00:14] Ricardo: And I'm Ricardo.

[00:00:17] Pamela: Today we want to share with you some of our thoughts around human centeredness in a time of AI-amplified business metrics. The reason we phrased it this way is we're going to talk about how to engage in what we see as inherent challenges to some of the practices that Rory was just now talking about, at a time when we are being asked to engage with and adopt new technologies faster than we can think about the consequences of them.

[00:00:56] May I have yours? Yes. There you go. Here we go. It gets us to a topic that Ricardo and I have actually talked about before, which is unintended consequences. We first talked about it before the pandemic, when we were both working in mobility. Ricardo was working with Ford on self-driving cars, and I was working with Delivery Hero and spending quite a bit of time on the urban impact of some of our scooters and mobility tools. We also spent some time talking about the impact of social media and some of the things that we really didn't foresee. Today we really want to be talking about what we anticipate will be unintended consequences that we can't quite grapple with, with the emergence of large language models and generative AI.

[00:01:46] AI has been around for quite a long time, but it seems to have come full force this spring with the launch of GPT. In a poll of a thousand businesses, to understand what they think about AI and where they see it, 67% said they envision AI to be an influential driver for their innovation practices this year. It's here, and it came on us really quite quickly.

[00:02:18] Before we really get into some of the material we want to share with you, we want to ask the audience some participatory questions. Rather than raising hands, which is impossible for us to see, I'm going to ask you to stand up if the question applies to you. How many of you were asked to integrate AI in your roadmaps two years ago? If anybody was already working on AI two years ago, can you stand up, please? I see one person, a couple of people. Oh, cool. If you stand up, it's easier for us to see you. And then how about in the last year? In the last year, please stand up. Please stand up, it's good, it's morning, we're still fresh. All right, how many of you have been asked to work on AI or integrate AI in the last six months?

[00:03:16] All right, cool. This is roughly what we anticipated, especially in the last six months. AI has really come to the forefront. Thank you very much, you can sit back down. It's really here, and it's really coming with quite a bit of momentum, and the velocity with which we're being asked to adopt this technology is part of what really concerns us. We want to come back to a question we're going to bring up a couple of times, the question we wanted to ask ourselves and ask you: what is within our power to work responsibly with gen AI and large language models, at a time when we're being asked to adopt these technologies faster than we can anticipate the consequences?

The tension between human centeredness and business metrics

[00:04:01] Pamela: The reason we care about it, I want to come back to Rory's point about our human-centered practices. I think this is why we're here. We all care about developing products that make a difference to people's lives in a positive way, that also impact the business because they really are effective and better than the norm. But this human centeredness is a bit at risk when we're working at a time when, through the fast adoption of AI, we're being asked to actually be less thoughtful, because we don't have as much time. And we're doing this while the businesses have expectations that we can drive business impact in efficiencies and also drive more revenue. This is where we see the tension between what we care about and the forces that are at work from the business perspective.

[00:04:50] What we do see, and I think this resonates a little bit with what Rory was saying, is that even though we care about human centeredness, our agility, however much it's practiced in all of our permutations, is really forcing us to think about work in two-week increments, to focus on MVPs and launch happy paths. But we rarely ever have a chance to go back and actually do it right, or fix the things that we launched. This is where we see the tension, and we see it accelerating now with AI, because here the expectations are so high on the business side that we will be driving efficiency through automations, and that we'll be affecting the business positively through our innovation practices leveraging this technology. This is the tension that we want to keep addressing when we come back to the question we were asking earlier.

The scale of AI investment

[00:05:37] Ricardo: Yeah, so just to give a little bit of context of why we are in this acceleration, let's look at some numbers to see how global this trend is. It's not only about this room. In 2022, according to the Stanford AI Index, the global investment for AI-related projects was almost $92 billion, and according to Goldman Sachs it's going to reach 200 billion by 2025. It makes sense, because the revenue that came from those projects and other projects that were already going hit something around 51 billion last year, and it's going to increase exponentially between 2025 and 2027. And in terms of not only corporate businesses but also startups, there are 14,000 registered US startups related to AI, and 58,000 globally.

[00:06:33] You can see how this whole notion of a huge sector of the industry, in many areas, being all in to adopt AI is something that is only going to accelerate more. What that means to us is that we are going to feel more pressure to create those MVPs, more pressure to create those happy paths, and to apply faster launches to market. But at the same time, and Pamela talked about the tension with business, there is a tension with the speed of technology that you're all aware of. It is because the technology is delivering and solving some incredible, very complex problems that we haven't been able to solve before.

Three cases: AlphaFold, facial recognition and recommendations

[00:07:18] Ricardo: I wanted to walk you through a couple of examples that connect to these unintended consequences that we want to talk about, and that are very well known today, so you might have heard of them. The first example that I wanted to share with you is that of AlphaFold, which is a database developed by Google's DeepMind. AlphaFold is solving a problem that is, I don't know, 50 years old, something like that, of understanding and cataloging protein structures. Understanding these structures is really good for pharma, because that allows them to find new products and new treatments and launch them in market. AlphaFold started with 100,000 structures known to science back in 2016, and in just a couple of years they actually jumped to cataloging something around 200 million protein structures using AI, which is basically close to the number of proteins that are known to science today.

[00:08:16] What that means for products is that there was this startup, "cical health" [?], that partnered with the University of Toronto. They were trying to find a new approach to cure a type of liver cancer, and it only took them 30 days to find the first viable prospect to drive to experimentation, which is an exponential acceleration from the months and years it has taken to get to these candidates. We believe that this is an example of how the technology can be used responsibly, because even if the acceleration is exponential, the scientific and technical teams on all sides of this conversation have tried to keep to the processes set by the governments and set by the scientific industry.

[00:09:00] But not all applications, and you have all heard different points of view in the news, not all applications of AI are free of problems, and facial recognition is one that has been not only controversial but also very well publicized. One of the cases we wanted to bring to you was this of Porcha Woodruff, who is suing the Detroit Police Department for wrongful arrest. What happened was that the police department used their facial recognition system using a mugshot from 2015, and they used this information and proceeded with little corroboration to arrest Porcha.

[00:09:46] They failed in the steps that they were following, both from the system, not looking to see if there was more recent data, for example her driver's license picture from 2021, and the police took that information and didn't corroborate details. For example, when they looked at the footage from the robbery that she was accused of, did they identify the suspect as pregnant? Porcha was eight months pregnant. If the human side of this story, plus the system, had taken into consideration verifying the data, checking the data, then a mother and three kids would not have gone through a very traumatic experience like they did.

[00:10:34] Another example that we wanted to bring to you, yes, is one of recommendations. This is a heartbreaking case, a very unfortunate case, and it's very well publicized. Chase Nasca, the kid in the picture that you see here, 16, died by suicide in February last year. There was no apparent evident cause for this action. His mother started looking at the For You feed that Chase had in his TikTok, and started looking at endless hours of content that was sad and depressing, and that Chase was consuming on a daily basis. This couple that you see here is suing TikTok for wrongful death.

[00:11:24] Because of these cases and other similar cases, journalists, for example The Wall Street Journal, started doing a study using bots to understand how long it would take for a bot to replicate the type of feed that Chase had. In the study they found that the bot only took 36 minutes to turn that feed into 93% content that was depressing [?]. Now you can imagine the impact that this can have on a teenager who is in crisis, for example. I think the story goes that Chase broke up with his girlfriend, and there is this combination of factors that gets us to a very complex conversation on how to keep our teens safe while at the same time growing at the exponential rate that TikTok has been growing.

Patterns behind unintended consequences

[00:12:21] Ricardo: Let me do a quick recap. We started talking about how there is a tension between the business and being human-centric, and that this tension actually creates what we call oversimplified product practices, where we end up with MVPs that we might not be able to come back to to improve on, and happy paths that we have to create really quickly to launch to market. This has only been increased by the promise of the technology around automation. When you think about these examples that we just shared, and we looked at a bigger sample, we wanted to bring you these three because they highlight these patterns that have to do with unintended consequences, which we believe we can start addressing, or at least take into consideration, when we're creating new products.

[00:13:15] The first factor is what we saw with AlphaFold. There is definitely an increase in speed to get products to market, and the systems, because they're self-learning, they're autonomous, can actually get to scale really quickly. But when they're used with good processes and with good rules, they can be brought to market in a responsible way. In contrast to that, it's really hard to talk about these decisions, or have these considerations, when there's only one metric that dominates all conversations and decisions, and in terms of TikTok, one of the core metrics is watch time.

[00:13:55] And then if you think about the case of facial recognition with Porcha, there was this whole human and system understanding: if we bring datasets that are not current, if we use models that are not inclusive, and we combine that with not checking the data, then we can create a lot of hurt for people who are not deserving it. Last but not least, one thing that I wanted to highlight in the case of Chase is that it is necessary to test the products over time, because what seems to be a bias in the algorithm to bring that sad and depressing content is something that cannot be tested after just one use. That's something to take into consideration.

[00:14:38] These are not new topics. We have heard about these topics over time, but it is critical for us to really, really consider them, because the machines are accelerating and actually starting to surpass those capabilities, and making decisions that, as we go forward, are going to be harder for us to track at the level of ability that we have today as humans.

Data: where it comes from and how it is checked

[00:15:03] Pamela: Thank you for those examples, which are quite heartbreaking. We want to come back to the question we asked earlier: what is within our power to work responsibly with large language models and generative AI? We wanted to reduce it to three things that we think we all can start to really work with, in a way that allows us to start to anticipate what could be unintended consequences, and really build towards a future where the positive power of artificial intelligence can be leveraged.

[00:15:35] The first one we want to start with is data, because all of the language models are really fed from data. It's really the power. As Ricardo already pointed out, it's really important for us to look at what the data is, who created it, how old it is, is it the latest. We can start to engage with that ourselves, with what is the data that we're using. At SumUp, we use Azure from Microsoft for some of our models, but we use only internal data for training these models.

[00:16:07] Then the second point is to think about the biases. Do you see any biases in the results that you're getting that you should be careful about or aware of? We work with people who check the results of some of the reports we're generating, for example for risk and fraud purposes. We leverage AI for automation purposes, but we still engage with the data and the reports to ensure that what we're actually getting back is accurate and doesn't reflect unintended biases.

[00:16:39] And then the last one is that there are some biases in industries. If you're working in an industry that might be historically driven by biases, how do you really anticipate that and work with that? There's a particularly interesting example that I want to share in the lending business, which, especially in the States, I think has been suffering from certain types of biases. There's a company called FairPlay. It's a startup from 2020, and they have introduced a new concept which is quite brilliant, which is fairness as a service. What they actually do is, when you sign up with them, they check your models and work with your models and train your models to remove biases that might be inherent in them. The whole purpose and mission of the company is to ensure that the risk for the business continues to be appropriate and low, but that the biases that have been driving decision making around lending, or even financial services, are removed and minimized, so the customers can also really benefit from a much fairer practice.

[00:17:43] The second one is Bard by Google, launched in the spring as an experiment. It allows anyone to use Bard to come up with some creative answers to creative questions. We all know that some of the ChatGPT technologies are still hallucinating, or creating some information that's not accurate. So they introduced a "Google it" button that comes back and says, "Yes, we've actually verified this, this looks pretty right, that's what we see on the internet," or it gives you an alert that says, "You might want to check it, because we found something different." It's a very good way to have this extension, to be able to continue to do due diligence on what we're getting back, what we want to believe is fact, a bit like the police department, but that we should really check and verify is accurate.

Inclusivity: team makeup, co-creation and testing over time

[00:18:36] Ricardo: All right, the second way that we suggest to engage is to talk about inclusivity. We all have been talking about this for the past couple of years. It's an important topic, and we are very aware of it, but we think it's critical, for something that Pamela already mentioned and that I want to highlight again, which is that the system builds on our biases. Bringing these voices to product development and product creation is very important right now.

[00:19:06] That's why our first recommendation, when we think about inclusivity, is to think about the makeup of your team. Bringing folks from other parts of the organization, or bringing folks from other projects that can help you, can be a way to gain perspective. They don't have to come as full-time contributors. They can be advisers, or they can be challengers to your team. Make it more casual, but also have a good cadence when you bring these other voices to the table.

[00:19:34] The second recommendation that surfaces as a pattern is to think about co-creating with internal or external customers, when it has to do with customer support or internal systems, especially when AI is applied to things like managing resources, resource allocation or payroll. We have all heard the complaints from the ride-share drivers, how they call the algorithm their digital boss, which tells them where to go, how to work and how much they're going to get paid when they're using the client for their activities. What some articles refer to is that when you don't have a boss to actually talk to about the factors that can create problems related to resource allocation or payments, what happens is these people get together with their peers and, through creativity and collaboration, fix the problems that this algorithmic manager cannot fix. That's an opportunity for us to explore.

[00:20:39] Last but not least, we have talked already about testing your product daily, and doing it over time to find those biases that cannot be found in a single use. As examples of inclusivity, we wanted to highlight May Habib of Writer.com, which is on the Forbes AI 50 list. This company, and her vision as a CEO, has created these stats that you see here for the makeup of the team: 60% of the leadership is made up of women, and 62% of the staff is made up of women, people with different genders and races. She also looks at factors like socioeconomic strata, which creates a real diversity of voices, and that gets translated into products.

[00:21:33] But it's not only the makeup of the teams that I wanted to highlight. Same as with FairPlay AI, there are also tools that are coming out that might help us gain insight into the data. Sony AI just published a paper on how they want to expand how to analyze skin color in visual recognition, to move from just lightness and tone to include a second factor of hue. This addresses not only the differences between different racial groups but also the changes that happen with age, which creates a lot of perspective when we are trying to apply this to a product.

Thinking ahead beyond the core use cases

[00:22:14] Ricardo: Last but not least, extending beyond core use cases and scenarios, the functional part of the product, is very important to understand the human dimension, if you recall the examples that we brought to the table. There are different factors that we wanted to highlight for you to take into consideration. The first one came from a 1930s author, Robert Merton, who is one of the fathers of sociology. He says that there are factors that constrain our teams from looking at long-term consequences, and they're here, and I'm going to read them because I remember them all: ignorance, short-termism, values, fear, error, and assumptions that people make. We think that now, with AI, speed should also be a factor.

[00:23:09] If you think about that, plus the motivations that companies but also individuals can have in terms of power, control, including and excluding some groups, that helps to create a much richer picture of the context in which the products will be created. The idea is to be able to have these conversations with the teams, capture those cases, and gain a point of view that we can bring back to our corporations on cases that we haven't considered before.

[00:23:40] As an example of a cool tool, one of our favorite tools to bring out these cases, there is this Tarot Cards of Tech deck that an agency in Seattle, Artefact, developed. These cards contain really hard questions that are really useful for teams when they're trying to start product development. For example, who or what gets displaced when your product is successful? Or what happens when a bad actor takes over your product when it's going to scale? We really recommend it, and we'll share a link along with the slides.

Wrap-up: engage now

[00:24:17] Pamela: To wrap up, we want to come back to the question: what is within our power to work responsibly with this new technology? We covered three areas. One was our responsibility in working with the data and checking the data, and there's quite a bit that we can actually do to make sure that the data that's fundamentally driving our products and services is accurate, correct and unbiased. Inclusion is really important, and there are a variety of ways, which Ricardo explained, in which we can supplement the inclusivity of our teams, so we can bring different perspectives, either of our audience, our customers or even our colleagues, into a conversation. And the last one, which we also really believe is a muscle that we need to start to practice as product teams, is to think ahead. Even though we are working towards MVPs and we're talking about happy paths, take the time to develop a practice with your teams to think about other ways in which these tools that you're creating can have negative consequences.

[00:25:30] To close, I want to come back to something really fundamental that we talked about. We believe in human centeredness, yet we're challenged by the speed of development. While this is all true, and we're working at this tension point between the demands that are put on us and the speed at which we need to be working, we want to believe, and we want to encourage you, to engage now, in a human-centered fashion, with the tools that we're suggesting or any others that you may have, so that we can together create the future that is actually quite powerful and possible with artificial intelligence.

[00:26:11] Have I said everything? We debated this quote quite a bit last night. We had very different perspectives on whether it was a good one to close with or not. Did I capture it?

[00:26:20] Ricardo: I think the word that you need to remember is "engage." This is the time, and we are those people who are engaging with this technology. It's our responsibility to create the tools and the processes that will get us to the next step, to use it responsibly. Thank you very much.

Q&A

[00:26:40] Host: All right, so let's talk about consent. Whose data can we use? As an example, did we ask Porcha if we can use her photograph in the presentation? And where does the data go, and how can we make sure it doesn't harm?

[00:26:52] Ricardo: Yeah, that's an ongoing conversation. In Porcha's case, or I guess I should say in the data that the police departments use in the United States, there is a provider that is connected to 3,000 different police departments in the country, and they're getting sued because of that point of consent of data. They have been doing a lot of scraping from social media to complete and train the models, and we know how that is already biased. There is definitely a push to make this consent clearer, and it's an ongoing conversation, because we hear how often people refer to how the legislation is too slow compared to the development of data. What I believe, and I think Pamela and I were talking about this last night in our discussions, is that raising the cases, bringing the information and actually communicating what we see is very important. It's not only about creating the product, but also telling these stories that bring clarity to what we should do.

[00:27:55] Pamela: In the case of SumUp, I brought it up. Of course, we're in Europe, so we very much focus on being GDPR compliant, which is why we only use data that's internal and stays internal. And with the models that are coming in from Azure, we are really spending a lot of time making sure that there isn't any kind of unintended bias introduced through some of this external data. Staying really focused on the internal is one of the ways we can certainly drive the efficiencies that companies are looking for, but we're not introducing unintended consequences, and we're not leaking the data from our customers.

[00:28:29] Host: Certainly, great answer. I saw a really fun question in here about whether we feel like we need to upgrade our techniques when it comes to qualitative and quantitative research, now that there are great advancements with LLMs and AI.

[00:28:44] Pamela: Could you repeat the question, please?

[00:28:46] Host: Yeah. I was reading, and I was like, wait, I don't see it. Do we feel like, as designers and product teams, we need to upgrade our techniques when it comes to qualitative and quantitative research, given the big advances we have in LLMs and AI?

[00:28:59] Pamela: Yes. I think the qualitative and the quantitative is where we need to invest more time. Historically, you've worked in design for a long time, and I feel like that's become very reduced to, again, some very basic practices. We need to reinvest in that interaction between what we know quantitatively and how we balance it with the qualitative insight, and really have them work quite closely together. We definitely see in our practice, and I think in yours as well, that there's a real acceleration of looking at research and research data and quantitative information to actually help drive some of the decisions.

[00:29:37] Ricardo: Yeah. One of the topics that, because of time, we couldn't address was this whole topic of metrics and metrics development. There is a whole opportunity for us to start developing what are the things that we're going to track that might bring some balance to those metrics that the businesses are driving.

[00:29:51] Host: Exactly. Do you also see AI actually being a part of how we conduct research? We've seen some companies utilize AI to automate synthesis, things like this.

[00:30:00] Pamela: Hopefully. I'd love to have our head of research, "Arno" [?], actually answer that question, because he believes that there's a lot of potential, and we've been super excited about it. But I think we're a ways off from it being effective. We're going to continue looking into it, but it's not yet, and I think I'm speaking the truth, Arno, at a point where we can really rely on AI as a way to accelerate all of our knowledge building. There's quite a bit of interesting studies. I even heard at a conference last week about the lack of emotion, this human dimension, when AI models are basically just doing pure logic around language and words.

[00:30:44] Host: Certainly. Got time for one more question. I saw one earlier. When it comes down to individual power, a lot of the things we're talking about maybe are at the manager, director, executive level, how the teams are structured. But as an individual, what are techniques we can apply to make sure that AI-powered products are more ethical? What are our own tools?

[00:31:03] Pamela: Yeah, but this is what we wanted to bring up in the three examples that we shared. We know, for example with the data, we're being asked to use data. So what can we do? What questions do we need to be asking? What due diligence can we actually be doing in our own work to continue to drive some level of awareness and engagement? We tried to keep all three of our examples at that level of responsibility, so that it isn't an abstraction, or just something ordained by executives, but something that we can do in our teams on a day-to-day, weekly or some other kind of cadence that works for your teams.

[00:31:43] Ricardo: Yeah, that first step is to gain more awareness. How do you change your next study plan for research? How do you actually bring in these voices that we were talking about? There are small steps that are very meaningful, and that is going to open the road for actions that fit your company and fit your team. But gaining that awareness and taking it into consideration, that's the first step that we are recommending through the presentation.

[00:32:08] Host: Fantastic, thank you. If you have any more questions for Pamela and Ricardo, please find them. Let's give a big round of applause to Pamela and Ricardo. Thank you.