Strategic AI Integration in Engineering Teams

Keyvan Azami at UXDX USA. Video: https://youtu.be/XhnO0wSYSPE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Who we are and the problem we were solving

[00:00:00] I'm Keyvan Azami from Google. I'm an engineering manager and I focus on solving business problems for Google's own day-to-day operations using ML. I lead a team that's been doing this for quite some time, eight years or so, before GenAI was in everybody's minds. What I'm hoping to do today is just walk you through some of our experiences and hopefully let you take away some insights about how to apply ML in day-to-day life, how to deal with the uncertainties that it brings about.

[00:00:38] I might not bedazzle you with the greatest in ML. You can get that from Google I/O, Cloud and all those places. But I find that there's a different approach to applying ML to real world solutions that sometimes, that muscle hasn't been developed in all of our teams so far. Nobody gave me a clicker to move the slides forward. Ah, there it is. Thank you, appreciate it.

[00:01:19] Our mission: applied AI research as enterprise competitive advantage. What we do is work with Google Research, specifically Google DeepMind, and look at what are the relevant capabilities that they're working on that could enhance our business functionality. We work with different pillars at Google, finance, legal, HR, technical support, etc., and find those solutions that could make sense. I'm going to walk through a case study that we worked on a few years ago and talk you through some of our processes, some of the challenges that we came across and how we overcame it, and hopefully this would be interesting and useful.

[00:02:10] Just to set the scene, Google, as you might guess, has billions of users across their products such as Search, Maps, YouTube, Cloud, etc., and we have, I think as of the last financial statement, 180,000 employees plus x amount of extended workforce. These folks generate millions of inquiries every year for IT support: systems, hardware, software, etc. What we want to do is get people answers faster and make both our support staff as well as our Googlers more efficient. The self-service automation goal is to identify critical self-help areas and make this as intuitive as possible, as accessible as possible.

[00:02:59] With this project my team was tasked with surfacing relevant support articles to internal users when they raise a ticket. They raise a ticket, don't wait. If I can find a solution for it automatically, make a recommendation, link them to the help article and let them get the problem solved sooner than later.

The ML life cycle, ideation and prioritization

[00:03:23] Our general ML life cycle is a little bit different to traditional software development. We start with ideation and prioritization, data acquisition, data exploration, which is a big, important phase for us, and I'll walk you through the details of that. And then we have prototyping. You might see some interesting turn of events here, where we don't actually follow a linear path to get through this, because the nature of ML is iterative and there's a lot of experimentation that goes on.

[00:04:00] Ideation and prioritization. What is the business problem? We start with that problem. I know it's very tempting to think that the answer is generative AI no matter what the question is. Well, start with the problem. What is the business problem that you're trying to solve? Assess that on its own merit and look at the complexity of the potential solution and the possible impact. We lay out the different ideas, we lay out the impact and lay out the complexity, and we start focusing from the top.

[00:04:30] Article suggestions, so finding support articles relevant to the problem at hand. What's the business impact? Tens of thousands of hours in efficiency gains for Googlers, for support staff. Is ML a good solution to the problem? Do we have enough data? What kind of data is it? What type of problem are we trying to solve? What's the risk tolerance for the users, stakeholders, the support organization? What happens if we get the answer wrong?

[00:05:01] That's an important question, because ML is probabilistic. If somebody says I need 100% guarantee that you give me the right answer, I can't do that project. We have to be able to deal with that uncertainty in the process. And can we measure success? Are there metrics available so we can know whether we've actually achieved the goal? And the answer was yes.

Data acquisition and data exploration

[00:05:26] Data acquisition. What do we do? We looked at our support data. We had millions of resolved tickets. About 10% of them were linked to an article. Somebody raised a question such as "I can't remote into my machine," and they waited a couple hours, and one of the support staff came on board, saw the ticket and responded with the link to the article that solved the problem. We want to automate that 10%. That's tens of thousands of tickets that could be solved.

[00:06:00] What data would we want? We want validated responses. We want to look at the tickets that were solved and had an article linked to it. Finding those is really critical.

[00:06:10] Data exploration. We spend quite a bit of time here so we don't jump into the solutioning. It's like, what are we actually looking at? We claim that we have these tickets and we claim that we have these articles. What do they look like? What is the relationship between them? My team at Google, we use Google Colab quite a bit. There are a lot of other solutions out there, Jupyter notebooks, etc., where you can pull your data and have different illustrations to help you understand the relationship between the features that you're trying to map to each other.

[00:06:45] This is where I think this is more than an engineering task. This is where we find UX researchers come into play, where our product teams can come into play. It can help identify these relevant, important capabilities that we can draw from the data. The tickets: titles, descriptions, links. And the articles will have again a bunch of text, titles and descriptions. So it's a big language problem that we have to solve for. And obviously we know that there is this issue that we can only solve 10% of the tickets.

First prototype and disappointing results

[00:07:23] Model prototyping. We have some data, we've looked at this and we're like, okay, let's start with a hypothesis and see if you can solve it. A typical approach is something called a dual encoder. You take the two things that you're trying to map to each other, the tickets and the articles, and you build a model that codifies these two documents such that if they're related to each other, the code between them will be very similar. Details are not that important in terms of how that works. You can always look up dual encoders; there's tons of literature out there. But we start with this: let's see if the solution works.

[00:08:06] We start with even a very simple algorithm, TF-IDF. Again, look it up if you're interested, but basically look for frequency of meaningful words in the documents that you're looking at. And then we can move on to more complex stuff: embeddings, contextual embeddings, etc.

[00:08:27] We run this model, we train this model and we look at the results. Precision 40%. Out of all the tickets that the model recommended an article for, only 40% of them were actually correct. The other 60% recommended the wrong article. And what's worse, recall, 15%. What does that mean? It means that out of all the tickets that could have had an article associated with it, we only recommended 15% of them with an article. So we missed out on 85%. The results weren't great. But this is where the iterative nature of the work comes out.

[00:09:13] You start asking, does the model not perform well because of my features? Do I need more information about the problem that I'm trying to solve? Does the model not perform well, but there is room for experimentation? Can we try other techniques? We started with dual encoder and some language techniques, but are there better solutions to this? And what's worse, did I even think about the problem right? Have I framed the problem correctly?

[00:09:44] And then here's where you're like, uh oh, I thought I was moving forward, but no, we go back and forth. This was the life cycle that I described, and we actually work like this. We go back, we look at the data, we look at the idea again and say, okay, how can we improve this? How can we make this more appropriate? Again, this is where the interaction between engineering, product, UX becomes quite important, because that problem framing stage is where you're trying to understand how you're solving the business problem.

Reframing the problem

[00:10:18] Ideation and prioritization again. Focus on the impact we're trying to achieve. What are we trying to do? Do we really need to suggest from the long tail of articles? We've got a lot of articles, tens of thousands of them. Do we actually care in the first instance that every one of them is looked at for each ticket? Maybe not. What we need to do is focus on an 80/20 approach. If we get 80% of the tickets resolved with the most common articles, that's still a win. Can we simplify the problem?

[00:10:54] This is where we reframe the problem differently. Instead of finding an article, we say actually we're going to classify it. We're going to say here are the most common problems, and for a ticket we're going to say which class, which type of ticket, which type of article does it relate to most. Different method, different technique, and focusing on returning the right article.

[00:11:17] If you remember, we looked at that precision and recall: 40% precision, 15% recall. What we focus on is let's get the articles right. Even if we miss the opportunity, the ones that we do try, let's make sure the answer for those is right, so at least we're helpful and we don't send people the wrong direction.

[00:11:38] We start back in the prototyping phase. Tickets, we try to identify the articles. Let's say we find a thousand articles and we start mapping them, and we train a model to try to guess which article would be most relevant. Language modeling, as we talked about. We start getting better results, not perfect but better: 50% precision, 25% recall. We're getting more accurate, still not good enough.

[00:12:06] So what's wrong this time? Does it not perform well, but is there room? We're still short of the goals. Time to debug. We start engaging with the stakeholders, the partners. So then again, back to the iterative nature, bring them back in, let them see, let them weigh in on the problem and the solution.

[00:12:27] We show them some examples. A ticket comes in, the model says X, and in the data actually there was no article that would have been appropriate. So the feedback is, that seems like a mistake. We do the same thing again. They kind of understand it. We do this again and they start seeing that actually there's a labeling issue. All the data we were looking at initially, looking at tickets that were resolved and there was an article, it wasn't always correct. There were some false positives in there. There were some tickets that were linked to the wrong article, which we didn't anticipate initially.

Labeling, better results and productionizing

[00:13:10] So we go back. Data acquisition. Here we work with the partners and said, okay, what if we actually run a labeling exercise? We take 2,000 tickets, we take a bunch of them and then have your expert staff label them. Make sure the articles are correct, make sure they're linked correctly, so that when we're training a model it's learning from correct information. The old adage of garbage in, garbage out. So we're going to clean that up, and surprisingly, even with 2,000 tickets you can make a dramatic improvement.

[00:13:48] We start prototyping again with this labeled data and we try a bunch of different ideas, and we start seeing significant improvement. We get to 80% precision. That's really good in the ML world. We can work with that. And recall is improving, but that's okay. We're like, get 43% of the problems solved that could have been solved, and with 80% precision, that's a solution that the stakeholders can work with.

[00:14:16] But it's important to engage them at that level, because ultimately when it gets wrong, somebody's going to complain, "hey, you sent me the wrong way," and they need to be comfortable that the benefit is worth the occasional pain and escalation that may happen.

[00:14:36] And then the hard part: model productionizing. You spend all this time looking at data, building the model, but actually to make a solution, put it in production and make it an industrialized solution, you need a lot of different pieces, and the ML code is a small part of it. We have all this apparatus that you have to put together. Obviously larger organizations will have their own third party solutions, cloud solutions across all the typical vendors, but this is where you have to put it together and make it work as real software would.

[00:15:18] Monitoring and maintenance. Now here's the problem. We looked at tickets at a point in time. What if things changed? COVID happened and the problems were different. People were complaining about different types of issues. They're working remotely, there were much more remote-oriented questions and issues. So you start seeing actually a drop in the quality of your model. It doesn't often find the articles that it needs. It starts sending people down the wrong way more often. That precision drops.

[00:15:49] So you need to be able to measure this. You need to have tools, dashboards, reports, whatever it is, some way of metrics. You don't need to do this on a day-to-day basis, but at least on a quarterly basis, if not more frequently, so you can have a feel for whether the solution changes. In traditional software development you have regression when you change the software. Well, here you have regression even if you change nothing. The data distribution in production changes, so you've got to keep an eye on that. That's what we call model drift, and you'll have to have retraining. You could have continuous training cycles that will help you.

Takeaways and other considerations

[00:16:30] Takeaways and other considerations. Have a structured approach to machine learning projects. It's very tempting to say, hey, just give me some data, I'm going to start prototyping this. This is where I think product teams are quite important, because you can ground the engineers a little bit more as well. Think about the problem, frame your problem correctly, think about the different approaches that you could take, think about the users.

[00:17:00] I failed to mention how much of our focus is on human-in-the-loop process. How do we make sure that this automation is not people versus machines? It's people with machines. It's an assistive technology, not an independent automation.

[00:17:24] Be ready to iterate and ask questions. That's okay, and we can fail fast, by the way. Those diagrams that you saw, be okay with getting to a point and saying, well, you know what, the precision and recall is just not good enough, we're going to abandon this project. The sooner you do that, as opposed to sinking many months of iterative work only to get marginal improvements. And celebrate those failures. In my book they're actually not failures, they are experiences that you've gained. We have developed high tolerance for that type of thing because we want to encourage experimentation, but we want to fail fast so we don't waste too much time.

[00:18:08] Leverage the right tools for the job. ML has different tool sets, so look for them, research them. There are a lot of good opportunities. The space is improving quite fast as well. And look at the impactful problems that are fit for machine learning. The first question is, can I solve this problem with a rules-based approach, with heuristics? If you can, that's where you should start, not with machine learning. That should be the next step. Develop a baseline so you can build on top of it.

[00:18:40] So that was our case study. We have a few minutes. I'm happy to answer any questions if there are any, but also folks can always reach out offline too.

Q&A

[00:19:02] Host: Questions, right on time, that's perfect. We're seeing something weird here, so we've got to look. There, here you go. Awesome. There's no question submitted online. Anyone in the audience have a question?

[00:19:23] Audience: Thank you for sharing your work for this article surfacing ML project. Can you share how you work with the UX designer, what was good and what can be improved?

[00:19:37] Keyvan: Great question. The question is, how do we work with the UX designer in this project? Think about it from a user perspective, whether it's a Googler or a support agent. What do they see? The questions for us were, when they raise an issue, how do we reach back out to them? How do we provide the feedback? The questions were like, do we send them an email? Do we help the agent and then have the agent validate before we send that? But that kind of created some delays. Do we add the article directly into the ticket and hope for the best? Those were some of the considerations that we were working on with our UX partners.

[00:20:19] Keyvan: There were future phases of this project where we had more real-time interaction, for instance when they're actually engaging an agent in real time, where I think the user experience was a little bit more dynamic. So thinking about how do you surface relevant articles as the chat goes on. Here we talked about a ticket mapping to an article, but if you have a live chat and constantly the model is monitoring the conversation and says, ah, now I understand what they're talking about, it's probably this article, then how do you surface that was definitely a key aspect. But really thinking about how do we want to expose the user to potential uncertainty was a consideration for the UX team.

[00:21:03] Host: Great. One more question from the audience.

[00:21:06] Audience: Can you elaborate on the rules-based approach and creating a baseline? We have an opportunity to apply machine learning, but my junior employees don't quite get what creating a baseline and rules mean.

[00:21:21] Keyvan: Thank you for the question. Let's say we had, I mean, this problem may or may not be appropriate, but the idea being, how would you describe the solution in traditional ways, with if-then-else? If you see this type of issue, if you see the following words in the ticket, do the following thing, choose this, recommend that, or show them a list of potential articles. That would be a rule, a heuristic that you could apply, because that's what the human nature would be when they see that issue. Try to replicate that in code, and that will give you the baseline.

[00:22:06] Keyvan: The baseline here would be that, and how effective is that? Obviously without ML here it would be hard to match free text between articles and tickets, but if you were looking for words and trying word matching and things like that, you could probably get somewhere, and you could look at the quality of that, the precision and recall of just that rules-based approach, a very traditional software development approach. You probably will find that precision may be like 20% and recall maybe 50%, because you probably will try more. That kind of gives you that baseline to say, okay, if I apply ML, can I get better than that, and how much better for the effort? That's how I think about it, business rules. Hope that helps.

[00:22:57] Host: Great. This is a good question that I kind of like. How do you make sure your structured approach is effective? Do you validate your approach before you start the project?

[00:23:10] Keyvan: Yes. There is a little bit of experience from doing one or two projects that you develop. But by structured, I think even just looking at some of the blueprints that I illustrated, the steps that we take, and thinking about your project in that way. When you're trying to plan a project, it's kind of hard when somebody says, how long is it going to take? How long does it take to build a machine learning project? I don't know. I really don't know, because I don't know what's in the data, I don't know what will work.

[00:23:46] Keyvan: What I can do is say, okay, the following are my next activities. I'm going to look at the data and I'm going to take three weeks doing that, and I may have to extend it, but at least I can have an estimate for the following task. You can break down your project that way and then gradually develop confidence around some of the abilities. If you've done a project with language recognition a couple times, the third time you may be able to have a better approach to estimating it. But by just breaking down the phases of it and trying to time-box it, it will help you manage expectations and set some goals for the team that are working on it.

[00:24:28] Host: Great. At what stage should the UX designer be getting involved?

[00:24:34] Keyvan: Very good question. I think UX designers need to be involved from early on, actually, the early phases. It can be sometimes tempting to say, okay, people are talking about data science and models and algorithms, I can back away. But a recent project that I've had, we had early involvement, but very quickly we focused the conversation on engineering, and then with the product that has developed, we've missed the opportunity to think about the UX aspect of it, and so we're now backtracking.

[00:25:10] Keyvan: So I think early on, making sure that before you jump into what ML can do, you're thinking about that experience and how would that work, and just assuming that the model is not going to be perfect. The model is going to have x% precision, so how do you surface that uncertainty to the users? You can't hide it from them. You can't say to the user, here's the answer, I know this is the answer. You have to be able to leave some room, and that's the job for product and design teams to figure out how we want to manage that risk. So I think early on is the answer.

[00:25:47] Host: How early? Day one?

[00:25:50] Keyvan: Day one, day two, something like that.

[00:25:55] Host: I love it. I think when I hear that from developers and engineers, that just makes my day. Love it. Maybe one more question. Was 80% of improvement in accuracy just from labeling?

[00:26:12] Keyvan: The labeling improved things significantly, but if I go back, I don't know if I can see it here anymore, but we actually looked at some language models. This project is from a few years ago, so we used the BERT language model that helped us quite a bit. It had a much more sophisticated ability to understand the context of those articles compared to the initial primitive TF-IDF approach. So there is a little bit that you can gain, well, a lot that you can gain from improving the techniques that you're using, especially nowadays with very sophisticated language models. You probably can do even better. But the labeling was important because that was your ground truth. That was the thing that you could go back to and make sure that helped drive the precision.

[00:27:05] Host: Excellent. Okay, Keyvan, thank you so much.

[00:27:07] Keyvan: My pleasure.

[00:27:08] Host: Give him a round of applause.

[00:27:10] Keyvan: Thank you, appreciate it. Thank you.