Building for Everyone: Iterative Innovation

Christopher Patnoe at UXDX Community: Building for Everyone: Iterative Innovation. Video: https://youtu.be/gP1uC7QCNug

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

A failed opera singer who found accessibility

[00:00:07] My name is Christopher. I lead accessibility and disability innovation, but I'm not the average Google nerd. Pardon me. I'm actually a failed opera singer. I studied classical music in school and realized in my mid-20s that I wasn't very good. So, because I was born and raised in the Bay Area, I found myself doing tech.

[00:00:27] I worked for 10 years. I worked on the first iPod and the first iTunes, the towers and Final Cut Studio. Then I went to Sony Ericsson and made phones for a bunch of years, and even went to Disney Mobile and made video games. But I've been at Google now for 14 years, and it's here where I learned about accessibility, because my product wasn't accessible.

[00:00:48] I was the lead technical program manager for Google Play Music, and we had a test engineer come in, turn on VoiceOver, and I heard "button, button, button, button." I said, "What's that?" And she said, "Well, this is Google Play Music for someone who's blind." I said, "Well, gosh, that's stupid. How do they use it?" She said, "They don't. That's why I'm here." That was my introduction to accessibility and disability. Never thought of it, never heard of it, but couldn't look away. And that took me into the career that I have now.

[00:01:15] People ask me, why does Google care about accessibility? Well, I'm proud to be able to say that's part of our mission statement, which is to make the world's information universally accessible and useful. And we take this seriously, not just in the digital world but also the physical world. For example, did you know that Google Maps can tell you if a particular restaurant has step-free access, or if there are accessible toilets or accessible parking? We have this knowledge of the world and we take the responsibility seriously to share it where we can, and where we can do it reliably.

Disability affects everyone

[00:01:49] Because disability really affects everyone. For the 1.3 billion people in the world who are self-identified, for them, say people who are blind or people who are deaf, assistive technologies like screen readers and captions can be dramatically life-changing. But in reality there's a lot of people who don't identify with disability, either because of stigma or just because of lack of understanding. Like for me, I didn't realize I was dyslexic into my 40s. I just thought I was garbage at writing and spelling and stuff like that.

[00:02:18] But then also there are the friends and families of us. Say, for example, going back to that wheelchair idea I mentioned a moment ago, if you have a person in your family who's in a wheelchair, you're not going to go to a restaurant that has steps, because it's not a place they would want to be, it's not a place that they're being made to feel welcome. And the same thing is true digitally. We all want to participate with brands that we identify with, that have the same values that we do. So a brand, a product that's more inclusive will have more people using it, not just the people who need to use accessibility tools, but also those people who will support them and love them.

[00:02:55] But in reality, every single one of us, all 8 billion plus of us, will have some kind of permanent, temporary or situational disability, and a thoughtfully designed product will meet you where you are, no matter who you are, where you are, or when you are.

Three methods: leverage your strengths, nothing about us without us, start with one

[00:03:12] So for the sake of time, I want to just go into a couple of key methods of how I've seen companies do this, like Google.

[00:03:20] One, you want to leverage your strengths. Do what you're good at. And these days everybody's talking about AI, so yes, I'll be talking about AI. I'm very sorry.

[00:03:28] Two, nothing about us without us. This is probably the most important thing. You want to build with and for someone, never just for. Only someone who's blind, for example, can tell you what it's really like to receive yet another bank statement that is not accessible, and how you have to deal with it: either suffer through it, or sacrifice some of your privacy and have someone else read it for you. So this firsthand perspective is profoundly important.

[00:03:57] Three, if 8 billion people sound kind of scary, it is. Start with one. Start with one person. Start with one feature. Start with fixing one bug. The important thing is that you get started, because there's no such thing as perfectly accessible to everyone. What you're developing is a process to become more accessible and more accessible, more inclusive and more inclusive. You're building a process. You're building progress, because there's no such thing as perfection.

Captions at Google, from 2009 to today

[00:04:26] So what I want to do is show you how we've done this at Google. Now mind you, we didn't think of this back in 2009 and think this is how we're going to do it. I'm looking back through history and realizing these are the things that we did in real time to create this process, specifically around captions.

[00:04:43] Going back to that 2009 date I mentioned, YouTube had its very first experiment in this intersection of AI and captions. And frankly, it wasn't that great. But what we learned by partnering with the deaf community was that something that's not great is significantly better than nothing. So we continued to invest, we continued to invent and bring people together. And today we have 67 different languages, from African languages, Indic languages, Asian languages, and we have these captions that are quite good. Some languages are very good, some languages need some work, but even something that needs some work is still better than nothing. So today we have 67 languages in YouTube, for free, for everyone.

[00:05:28] But then we took the same stack that was once offline, like YouTube, and turned it into real time support, and now we have 87 different languages for Google Meet. So we can provide real-time captions and we can provide real-time translation. We can go from one language to another. In fact, in some languages we can even do real-time dubbing. So you're speaking English to someone who speaks Spanish, they can actually hear it in Spanish, because we can do some translation and vocalize it for you. It's amazing how these things work together.

Live Transcribe, and the tools in your hand

[00:06:05] But what I want to talk about is the tools that are in your hand, because these have the greatest opportunity for impact. So we have a wonderful application called Live Transcribe and Notifications, where we can provide real-time transcription in over 120 different languages with online and offline models. If you can't speak or can't be understood, you can pop up a keyboard, tap a message, and we can see what you want to say.

[00:06:31] But by partnering with the deaf community, we realized that sounds matter a lot. We have the sound notifications part of the application, so we can tell you when someone's calling your name, or when a sound around you has been triggered, say a baby crying or a car alarm, or even when dinner's ready, because we can train the AI on what your kitchen timer sounds like, so we can tell you when it sounds like dinner's done. And it was invented by these two guys, Dimitri on the left and Chet on the right. I'll let them tell a little bit about the story.

[00:07:19] Dimitri: I completely lost hearing when I was over a year old. Technology provides tremendous opportunity to help people with a disability. I'm a research scientist at Google and I'm developing speech recognition technology. Speech recognition accuracy finally became so good that I could fulfil my dream for my whole life, to use speech recognition to communicate with people.

[00:07:58] Dimitri: Hello.

[00:07:58] Server: Hello, how are you?

[00:08:00] Dimitri: I'm fine. And how are you?

[00:08:02] Server: I'm doing good. What can I get you today?

[00:08:03] Dimitri: I would like... Live Transcribe is an application that runs on Android. It provides transcription of speech for deaf and hard of hearing in real time.

[00:08:20] Chet: Dimitri worked with the speech team to develop that technology, and Google has made strides in this in the past couple of years, but the app itself is what I've been working on. I'm not an Android engineer. I saw an opportunity for something useful and dug into that, and it kind of evolved naturally.

[00:08:43] So here's technology that was invented by a person who was deaf. So it's start with one. It's do what you're good at. It's almost the entire thing in one application. It's brilliant.

On-device models and captioning the world around you

[00:08:56] But I mentioned that we have these offline models. For those people who may not be familiar, online models are the things where you send the data off to the cloud. There's going to be a little bit more latency, but a little more accuracy. An on-device model runs on your phone, so it's a little bit more secure, a bit less latency, but not quite as accurate. Depending on what your needs are, you can choose one or the other.

[00:09:17] But when you have an on-device model, you can do some really interesting things. For example, we can take some voice notes or videos or social media posts and we could provide captions for you, because that gives you a more equal experience. So we built this into the Android operating system.

[00:09:37] What's really neat is with the latest large language models we're starting to add context to what these sounds are. It's in English now, but we'll be rolling out more languages over time. So we can use this contextual understanding of our multimodal AI to understand when someone's yelling, or someone's sighing, or recognize the sounds or the music that you're hearing, because these are things that are important. We're going beyond the string, beyond the words, into some of the feelings that go behind it. So we're able to provide this kind of information in a thoughtful way, and we're experimenting with this. This is really exciting new stuff that's just coming out.

[00:10:15] But I do have to say that this is also very contextual. It's also very cultural. So somebody that is yelling in, say, Italian may not be yelling compared to someone who's yelling in American, because people in Italy speak with a little bit more gusto. So this has to be done thoughtfully and carefully. But it's exciting to see these things start to come out.

[00:10:39] But again, when you have this on-device model you can do some really interesting things, like use it to talk on the phone.

[00:10:46] Matthew: I hesitated for a few seconds and I realized the enormity of it, what it really meant. I'm 55 years old, and for the first time I was able to call my son.

[00:11:04] Narrator: This is Matthew. He likes to swim in cold water.

[00:11:07] Matthew: After a swim, you feel alive.

[00:11:10] Narrator: Like really, really cold water.

[00:11:13] Matthew: I also swam across the English Channel.

[00:11:15] Narrator: But that's another story. Matthew is deaf.

[00:11:21] Matthew: People think if I've got a hearing aid on I can hear everything, and that's not true. I lip read most of the time. I have two boys and they're both a very important part of my life.

[00:11:38] Narrator: This is Matthew's youngest son.

[00:11:41] Harry: Hi, I'm Harry.

[00:11:41] Narrator: He lives in Hong Kong and plays rugby.

[00:11:45] Matthew: I really want to keep in touch with him. Before, I needed to communicate with him through Zoom or WhatsApp. I would try and lip read him, but it's really hard because it's not 3D, it's a flat screen. When I got the Google Pixel phone that gives live caption, Harry was the first person I phoned. It was incredible, and we were talking for about half an hour. All of a sudden he said, "Dad, do you realize this is the first time we've talked on the phone?" I was speechless for three, four seconds, and I realized what it really meant. I'm 55 years old, and for the first time in my life I was able to call my son.

[00:12:27] So that's really cool. We're using technology to bring people together, families together, over thousands and thousands of miles. And these are not the things that will necessarily get you a job or help you go to school, but these are the things that make you want to live the life that you have. It makes your life worth living. It brings the joy into life, and that's what accessibility can do.

[00:12:52] So we have this on-device model. And you notice, or maybe you didn't notice, that Matthew was verbal. Not all deaf people are verbal, or can be understood. So we had this idea of using text to speech to be able to provide a voice. We added a feature called live caption for calls, where you pop up a keyboard and type in a message like WhatsApp, press send, and we use text to speech to generate a voice for you. So you can use the phone just like anybody else, whether you're deaf or non-verbal or have MND or ALS. These neurodegenerative diseases stop people from being able to speak, so with a technology like this they can use a phone just like everybody else.

Wearables and sign language

[00:13:37] What's really exciting to me now is where we are going in technology, and wearables is profoundly exciting for me. This is what I'm super excited about. I've been working on this for over 10 years, this idea of using a pair of glasses that can provide the world to you. So if you're blind, you can use the cameras and describe it to you. But if you're deaf, you can have inline captions.

[00:14:01] And this exists today. This screenshot I have here is from about five years ago, a prototype that we had, but it's starting to ship for real today. In fact, Android XR will be coming out later this year and we're going to have Google Translate and live caption and all these native experiences built in, bringing people together. And this wearable keeps your phone safe and keeps you looking at the other person, because I don't know if you noticed when Dimitri in that video was talking to someone, he had to look at the phone, not look in the eyes. So now if we have these captions in your glasses, you can look at the person and get all the non-verbal information as well. This is super exciting for me.

[00:14:42] And now we're getting to the point where the AIs are getting robust enough that we're starting to be able to do sign language. So in May of last year we announced a thing called SignGemma, and my hope is in the next few weeks or months we can finally ship it with a quality where you can do sign language transcription. Not interpretation, but transcription. Meaning, for example, if you were deaf, you can go into a coffee shop, sign into your phone saying "I'd like a double cappuccino," hold it up, and you're speaking your native language and they can see it in text.

[00:15:14] So this is exciting, and in the long run we'll have bidirectional communication where someone could speak to you and you can see an avatar. We're not there yet, but again, progress over perfection. We're making the technology better and better over time. So sign should be out in the next handful of weeks to months, and I'm really excited what people get to create with this, because this is going to be an open source model, again starting with English and adding other languages later.

Recap

[00:15:38] So to recap, there's some things you want to think about. One is leverage your strengths. Do what you're good at. And in today's context, we're talking about AI. Two, nothing about us without us. Always design with and for someone, never just for someone. Three, start with one. Start with working with one person, or start with building one feature. Start with fixing one bug. Because only by doing this can you start to understand the nouns and the verbs and develop the insights necessary to be able to create something really meaningful.

[00:16:07] Because again, this is a journey you're going on. You're going to continue to make progress. There's no such thing as perfection. No technology will be perfectly accessible for everyone, so don't make that the goal. But in all reality, the most important thing is that you just get started, because if you don't, none of this is going to happen. And when you do, be amazed where the technology can bring you.

[00:16:36] Video: A look can say a lot. A look can be honest and true. A look can speak for you. Say what's on your mind, or in your heart. A look can say what's best unsaid, and find the right words without having to say a word.

[00:17:14] Video: It's great to see you.

[00:17:16] Video: A look can say a lot.

[00:17:29] Video: How are you? Technology that's built to help. Thank you.

Q&A

[00:17:43] Host: Excellent talk, Christopher. Really showing the value of the work that we do, not just turning out another product but really looking at the improvements that you can make in people's lives. So I really enjoyed that and I hope everyone out there did as well.

[00:18:00] Christopher: Thank you.

[00:18:02] Host: So yeah, if you're watching and you have any questions for Christopher, please do write them in on whichever platform that you're watching on. But I'm going to open it up, because there's been a lot of improvements I would say over the past few years with new laws around accessibility, and now that it's required and you have to make sites accessible. How do you feel that is going, and do you think it's going well enough? And I'll have a part two of, are these LLMs just going to blast straight past that?

[00:18:35] Christopher: Yeah, it's a subtle answer, and I want to also be slightly political about this. I think the presence of the European Accessibility Act, and the fact that there's teeth, really moved the state of the art in many companies around the world to make their product more accessible. But we're at a point now where, pardon my Americanism, it's like, show me the money. The challenge we have now is that the market authorities across the EU weren't really ready for it yet. So only now are we starting to see some action. We're starting to see people do things. So we had this weird six months of hurry up and wait. I think it took some of the wind out of the sails, but we're now starting to see Sweden and Netherlands and other countries starting to take this seriously and starting to make some noise. So I'm cautiously optimistic that we're going to see more and more teeth, which causes more and more work. So I think it's going to be good in the long run, but we're not there yet. We're still at day zero, day one kind of thing.

[00:19:44] Christopher: Now, to your second point about LLMs, they will help but they won't solve. I think what's going to happen is AI technologies, not just LLMs, but think about text to speech and speech to text, and the ability to create automatic alt text and automatic captions. What these will do, they will lift all boats. So people are able to use some websites more than they would without. But we still have work to do to have a good experience. To your point earlier about having broken software, it sucks if you can't use it.

[00:20:24] Christopher: And what's going to happen, I truly believe, is we're going to get to a point where the platforms, Mac, Windows, Android, iOS, etc., will help a lot of people do that stuff better. But the companies that want to actually compete for the business, go back to that first part of my presentation: the brands, if you want the people to come to you, you want to show that you care, you want to do the work. Those brands will do better and better over time. Whereas everybody else just kind of stays in stasis. Stasis that is a better quality than we have today isn't bad, but it's not going to make your business grow. It's not going to really move your customers to love your brand, love what you're building, or change their lives. And we as designers and we as technologists here, we want to make a difference, and just letting things be good enough isn't going to be good enough for long.

[00:21:16] Host: Excellent. Yeah, it always reminds me, we had a talk a few years back at one of our main conferences and it was a blind person online navigating a website, and yeah, it was, as you said, eye opening, because it was a cinema and they were like, red seats are occupied, and he's going, brilliant, how do I know which seats are occupied and which ones aren't, because the screen reader didn't tell him red versus green, it was just a coloring distinction.

[00:21:46] Host: I guess I'll play devil's advocate slightly, because I really think particularly the wearables is going to be a game changer for a lot of people. Is there, because with wearables and that kind of always on recording, is there a privacy concern, or how do we address that concern and still enable people to get that kind of real-time translation?

[00:22:15] Christopher: This is an important question and I don't think anyone has solved it. But what I can say is I think it's important for us to have a base understanding, a base experience, that is good enough to have value. And then if we provide extra value by signing in, telling us who you are, we can provide more value to you and develop more trust. I know a lot of people, for example, who won't use the Meta glasses because of Meta's reputation. But a lot of the disabled community, the blind community, love it because it gives them something that they wouldn't have without. So they're willing to take that risk, whether it's a real risk or not, this perceived risk of that kind of intrusion.

[00:22:56] Christopher: So I think it's important for technology companies to provide good value and make it worthwhile for someone to give up a little bit more of their privacy, because there's value in it. But it's also up to us as technologists to minimize, and make it easy to get rid of your data, so you can take it away. So if there is a concern you can protect yourself. All of those things have to be real. So we provide value, we make it easy for you to step away and take your stuff back. It has to be valuable, it has to be meaningful, it also has to be thoughtful.

[00:23:31] Host: Great. We have a question come in from the audience, I'll jump to that now. So this is from Victor de Jesus Oliveira, and he said, first of all thank you for the amazing talk Christopher. What do you recommend for young designers who want to learn more about accessible development and disability? Do you have any sources or materials you recommend, especially when WCAG can be overwhelming for beginners?

[00:23:56] Christopher: Yeah, a couple of things that I would think about. Both Microsoft and Google have some excellent materials. I think from a design perspective, Microsoft's materials on inclusive design are probably a little bit better than ours. So I'd say take a look at Microsoft's inclusive design. They just recently did a revamp, adding some stuff around neurodiversity. So take a look at those. They have a great history of doing this kind of work well, and I'd start there. Also take a look at what we have as well. We have some really interesting things on Coursera. So there are places to get good information.

[00:24:30] Host: Okay, brilliant. Yeah, it can be sometimes daunting when you look at those more official spec-driven documentation.

[00:24:39] Christopher: The thing is, specs are just written by people, and we're trying to design for people broader than just the people who built the spec. This is why the inclusive research is so important. This is why talking to real people, instead of just the smiling profiles you have when you're designing something at the very beginning. Real people don't always smile, and having them tell you that something sucks is really a gift, so we can actually try to make it better.

[00:25:04] Host: Excellent. And on that feedback, here's another question from the audience. Are there any online or in-person communities for accessibility professionals that you would recommend?

[00:25:14] Christopher: There's a neat group called CAN, the Champions of Accessibility Network. They're out on LinkedIn. I'd say take a look at them, it's free. There's also the IAAP, and this is actually good for the previous question as well. The IAAP, the International Association of Accessibility Professionals. They have some lovely courses as well, and they even have some certifications that you can take. I don't know if they have design per se, but it's a great place to take a look at the community there. Between those two, I think you can get quite a lot of information and a community of people who are all focused on the same thing.

[00:25:47] Host: Excellent. And one thing, I'll just say we've probably got time for just one last question. I was just surprised by that first product example where you showed there was just two people working together on it, the live real time translation...

[00:26:04] Christopher: Live Transcribe.

[00:26:05] Host: Live Transcribe, sorry. Is that the normal kind of team size, very small, kind of empowered teams building amazing products?

[00:26:14] Christopher: In this case it was a hackathon. They submitted Live Transcribe as an app as a hackathon, and it won. So we have some examples of literally two guys in a garage building up something that has real impact. If you look at Guided Frame, which is a really neat feature we have on Pixels, which allows someone who's blind to be able to take pictures, starting with a selfie and then on the other side, this also was a hackathon, but was done by about six people. But the inspiration came from the son of a Googler who was blind, who wanted to be able to take a selfie while doing sports.

[00:26:52] Christopher: So the best ideas come from the community. This is a problem that we have, can you use technology to apply towards it? And then it takes a bunch of different kinds of philosophies. You need researchers but you also need developers and you need the community. So in the Live Transcribe part we had the community through Dimitri, the engineers through Dimitri and Chet, and the technologists. So that was unique, and I don't think you normally see something as impactful starting with two people. Normally it's a handful of people of mixed roles, where you have research as well as engineering as well as community and design, because that's when you try to create the full understanding and build out a high level roadmap of where you want to start, what's the next most important thing, etc.

[00:27:42] Host: Excellent. Yeah, hope there's many more hackathons in future and we get to take advantage of some great products coming out of it.

[00:27:49] Christopher: They're my favorite. Absolutely.

[00:27:52] Host: Excellent. Well, that brings us to the end of our time. So I just want to thank you once again, Christopher, for sharing your story with everybody out there, and I hope everyone enjoyed it as much as I did.