Setting up for success with ML

Ekaterina Garbaruk Monnot, Maya Bogdanova, Dana Aonofriesei, Adam Bermingham at UXDX EMEA. Video: https://youtu.be/77otZcqxFwU

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Should this product use ML at all?

[00:00:00] Adam: Well, I guess we just want to dive right in. We have 30 minutes, which I'm sure will pass very quickly. I've had the pleasure of meeting you all before; we've already had a bit of a chat, and I think we got very excited about doing this in the UXDX forum, where we're thinking a lot about products, the really big view on ML. I think some people in our audience today will be thinking, wow, this is something that might be part of my life or my company's life over the next few years. But maybe we could start with something quite general. Not all products have ML. There was a time when most products didn't have machine learning, but now more and more products do. So how do you orient yourself in that space, and how do you decide whether ML is what we're going to do on this product? Maybe, Ekaterina, I might start with you.

[00:00:45] Ekaterina: Yeah, absolutely. I'm really passionate about this topic, and I'm glad that you asked. I think often machine learning and artificial intelligence are used like buzzwords, and the thinking about implementing them starts from the direction of, let's do artificial intelligence or machine learning and then we will find the use case. Instead, and this is also what we talked about in our talk, it's very important to think about the use case: what is the customer problem or business problem that you are trying to solve? Because all those solutions come with a cost. So I think it's super important to think, within your context, within your product, what are the problems that you're trying to solve? Maybe solve them with machine learning, or maybe solve them with better user experience, or maybe with better customer care. There might be different solutions. From my perspective, just treat machine learning as one of the tools to solve those problems.

[00:01:50] Adam: Yeah. I know myself, having come up as a data scientist and done a PhD, you come into industry and you're like, right, show me the data science problem, I'll solve it. And they're like, it's a little bit more complicated, because there are multiple ways we could solve this problem and we need to decide. And then you realize the world is more complex than you thought it was, and that's why we need that real cross-functional mix. I listened to your talk earlier, and you were talking a lot, maybe even more on the technical side, about how it isn't just an algorithm you need. You actually need to assemble quite a different way of working to traditional software engineering if you want to go down that route, and that has a lot of implications in itself for how you set up and invest.

[00:02:35] Dana: Yeah, for sure. As Ekaterina said, sometimes we can fall into the trap where we say, hey, we will just develop a machine learning model here. And besides not figuring out the use case, you also realize that you need some components of infrastructure to actually deploy it, retrain it, maintain it, keep it up to speed with new patterns. I think we sometimes forget about that part. Also, one of my learnings is that maybe our products, or our systems as we've built them for many years, are not ready to absorb machine learning models or predictions from machine learning models. I think this is also an area that requires a bit of a change in skills, or in how we look at our products. And this is an invitation not only for engineers, but also for designers and product managers, to help in embedding machine learning in user jobs and workflows.

What makes a good environment for ML

[00:03:56] Adam: Yeah, so for sure it's about those people, and there's infra, and then there's actually building the product itself, so there's a bunch of things to get right. All of you have talked to me previously, and mentioned in your talks, that there's a secret in AI: a lot of the projects are not successful. That's this open secret, because research is necessary in order to create these AI products, and research inherently involves experimentation and failure. So any company that's looking at the bottom line and looking at timelines will really want to maximize their chance of success. I know this is something that you looked at, Maya, especially from a learning perspective: what makes a good team, what makes a good set of stakeholders, what makes a positive collaborative interaction. Maybe you could talk a little about what you feel makes a really good environment for machine learning to flourish, and maybe what might not be that sort of environment.

[00:04:50] Maya: Yeah, thanks. I think the way that we've approached it was to look at the large picture and consider who the people are that are involved in decisions around making machine learning work overall. That's the part that we've really tried to emphasize: it should be everybody, along the whole process, all the business stakeholders. And they all need to have a really common language to be able to explore this as a solution to a problem, as a possibility, and have the ability to decide: is machine learning actually a good solution to this problem, or can we find other ways to solve it? So I think it's really important to provide the skill sets and the ability for everybody to take part in that initial conversation. It will save so much time and headache to talk about these things and start from a really good place from the very beginning, rather than encounter this later down the line.

[00:05:59] Adam: You mentioned something there that really tickled me. I really liked this idea of the responsibility to use a shared language to talk cross-functionally about the product. And I guess, Dana, that's sometimes a challenge for engineers, to use business language, a cross-business vernacular, rather than what they might be used to in more technical circles.

[00:06:22] Dana: Yeah, well, I think nowadays communication skills for engineers and data scientists are super important, and it's something that we value a lot. I think the industry has changed a bit. I can't say I've seen this as a challenge for us. As long as we facilitate that discussion and create the forum for stakeholders and engineers to meet, the results are usually good.

[00:06:58] Adam: I think that's the unifying thing I've heard across all your talks: just create the right forum, create the right arena, and it's actually not too hard to do the right things once the environment is correct and you have some standards and norms established, some cultural ways of working.

[00:07:13] Ekaterina: But I think, to complement what Dana said, it's also about what the discussion topics are in those cross-functional discussions. I believe in educating both business stakeholders and engineers, so that engineers know they can discuss different aspects with business stakeholders, including discussing the cost of machine learning or artificial intelligence: what are the risks, what are the challenges and what are the benefits. By having the talking points aligned across different stakeholders and groups within the company, you can have a much better result, because that also opens up creativity. If you already know that you've got the foundation across different functions, then you can build on top of that and focus on the content, rather than trying to figure out what the format of the discussion is.

Treating ML like product management

[00:08:10] Adam: Yeah, I really like something you just said there, Ekaterina, about how we structure the activity that we're trying to do: what are we trying to learn, how would we make best use of our resources? If you talk in terms like that, I think you can realize that research and machine learning development can be quite similar to product management in that sense, where you have hypotheses you want to test and try to figure out as you go. Do you have that same view? Is that how you feel?

[00:08:42] Ekaterina: I'm generally following the philosophy that many things can be thought of as product management, and in that sense, absolutely. I think with machine learning and product management, since machine learning solves a specific problem, it's pretty much the same process. We have a problem to solve, a need to solve, we have a hypothesis, and then we need data to validate it. In that sense it's very much aligned. So I'm very much in favor of using that thinking, and also learning from failures, because then you can put that learning back into the process and improve in future iterations on what you are doing, and maybe decide not to use machine learning after all.

[00:09:33] Adam: Yeah, that option could always be on the table, and you may even find yourself going back there after getting your feet wet. Maybe on that topic: how do you check yourself along the way? If you're taking a product management point of view, you might think, how do we pause, pivot, stick or twist, that kind of idea. But similarly on the ML side, your first model maybe doesn't work as well as you thought. You can suddenly find yourself with a very big decision: do we stick with this and keep going, or do we maybe take a step back? Is it standard product management practice, or something specific you would do for ML? Maybe I'll get Dana's perspective on that first.

[00:10:16] Dana: Yeah. I think in products it's a bit easier, and maybe less costly, to experiment: make something, ship it to production, test it and receive feedback. With machine learning, I think the experimentation part takes much more time and it's more costly. I don't think I have a straight answer. In our case, my teams work in the fraud detection space, so we already have a few good use cases for machine learning, and we don't feel like we have to validate our idea. Our use cases are very solid. But I imagine that could be a challenge in other use cases. So maybe Maya or Ekaterina can share how you validated your idea.

Setting metrics and aligning them with the business

[00:11:20] Maya: Yeah, I can share a little bit from the perspective of what we try to teach in the course. When it comes to pivoting, or thinking about how we adjust, I think it's really about setting up really good metrics from the very beginning of a project. At least for us, the emphasis was on how we empower everybody to understand, to some level, machine learning metrics, which can be a very technical topic, and empower business stakeholders to understand how these metrics translate to business metrics, which are not the same, and how these can be used as proxies. So from the perspective of the course that we're teaching, I think that decision lies in establishing really good metrics for your product that uses machine learning, and constantly checking yourself there.

[00:12:26] Ekaterina: And on top of that, just to add: metrics come from your decision about what you optimize for. When we designed the flow for the course about machine learning, and when I was in a similar project to the one Dana described, where you're actually implementing the model to solve a real problem, you need to decide what you optimize for and what your success looks like. Based on that you can devise the metrics, and then see across the project, across the development, whether you really are achieving what you want to achieve. Then you can have a retrospective: if you don't achieve what you're aiming to achieve, how can you adapt to that, and what do you need to do? Out of pivoting, stopping or maybe scaling it up, what is the right action for the circumstances?

[00:13:22] Adam: Yeah, I guess something I've seen before is very similar to that. If we're in a very data-oriented land, then finding a way of measuring success with metrics can be very helpful. But something I've found, and I'm interested to ask people what their experiences are, it's so fascinating, is when you think the machine learning KPIs are the same as the business KPIs, but they actually turn out not to be aligned at all. The machine learning KPIs are quite technical in some sense, and when you look at whether this actually solves the business problem, you need some different type of KPI framing around that. Sorry, I just lost my mouse for a second. Okay, there we go, it was on the other screen. I wonder, Dana, do you find you have to separate your technical KPIs from your business KPIs when you think of progress and success with your teams?

[00:14:20] Dana: We are trying to align them. Because I think if they are not aligned, there is no link between them, and that doesn't seem right. For example, when it comes to detecting fraud, we are able to say that with this new model we will be able to capture this type of fraud, in these numbers. Those are the results that we expect, and once we ship it, we monitor ourselves against that expectation. Sometimes the results are better than our expectations. So when it comes to fraud detection, I think that's very much in line with the business KPIs. But I can imagine it being a bit harder, for example, when you have models that look for sentiment.

Finding and growing ML talent

[00:15:32] Adam: Having worked on sentiment analysis myself in the past, I can relate to that. You can have the best performing sentiment model in the world, but you might not have a product. Much more is needed to understand what the application specifically is. One of the things, maybe more on the challenging side: we talked already about having fertile ground for teams and for product development. There is one role within the mix, either a data scientist or an ML engineer, which is often very important in developing an AI product, and it can be reasonably hard to find high quality candidates. Maya, have you come across that challenge before? How would you recommend people might tackle that issue?

[00:16:26] Maya: It's a good question. I would really say that almost any expertise around machine learning, whether that's data scientists, machine learning engineers, PMs with machine learning experience or researchers who are well-versed in machine learning, is hard to come by. Any of the folks with that experience are really a scarce resource right now. I think this is also why we at Spotify decided that it was a really good investment of our time to create a training, because we have amazing people internally, and we wanted to make sure that we could upskill them and empower them to use this new technology.

[00:17:10] So I think the focus for us was how we use this training to provide the skills necessary as quickly as possible and in a practical way. The way we thought about it was, we're not going to teach everything about machine learning, because that's a huge scope. But how can we really focus on our internal use cases, and empower data scientists who maybe don't have as much ML experience, or other engineers, backend engineers who can support machine learning engineers in their teams, to gain some of those skills, in order to lift everybody and move forward without only competing for the scarce resource? Which we continued to do too, but internally we decided it was a good investment to also try to upskill everybody at the same time.

De-risking expensive ML experiments

[00:18:09] Adam: Yeah, I think that's a really good idea, because then, earlier on in the process, you can have more skills in the mix to make some data-driven decisions, even without maybe your target team there, which is obviously much more efficient and allows much more activity within the company. Very related to that, we have a question from the floor, which pulls on a thread from Dana earlier, where we talked about how one of the characteristic differences between product development and more technical ML development is that research phases, iterations or experiments in ML can take a long time and can be quite expensive to execute. So the question is: what can we do before that to try and de-risk, given that it might be an expensive exercise? Dana, it's a comment on your point, so do you want to jump in first?

[00:19:01] Dana: Yeah. Well, if you have the data, you should spend more time in analytics, trying to predict what the outcome would be: try different variables, different features, more features, fewer features, and then compare the results.

[00:19:26] Ekaterina: Sorry, maybe also to add to that. I cannot help but link it back to product management: can you somehow validate this hypothesis in a cheap way? Can you create manual processes? Is it expensive? Where does the expensive part come in? Is it that you do not have enough data, or the data is not in a good state, or is it that you're not sure whether it would solve the problem? So address the biggest risk, and maybe try manual processes to validate that, or look at the analytics that can help de-risk that large expense. And then really challenge yourself whether you do need to train that model, or whether you can solve the same need in a different way.

[00:20:16] Adam: But I think what Dana said is also really valuable, about analytics and educating stakeholders to look into analytics. Sometimes people could think, oh, I'm going too fast, I'm a startup, I'm on the highway, I don't have time for analytics. But actually they're going to slow themselves down, because they leave themselves really open to not seeing what's in front of them and going down dead ends, that kind of thing.

[00:20:41] Ekaterina: I would like to challenge that, because for startups and fast-moving companies, looking at the data is at the core of it. If you are not looking at the data, you might go very far in the wrong direction. So I actually think that not looking at analytics can lead you the wrong way. But the key is to balance how much you invest in that and how fast you can move while balancing the investment.

[00:21:13] Adam: Yeah, that's exactly it, isn't it: being able to balance, and being cognizant of what you're doing, what you're investing in and the cost of that. I really like the point you made as well about characterizing your risk. From a product point of view, what are we worried about here? Is it feasibility? Is it viable? Is it a big enough market? Which part of that product opportunity are we really worried about? If it's feasibility, then maybe it doesn't need a technical exercise to verify that it would work, but another time it might be a different type of validation that's required.

Pitfalls: bias, learning culture and imagining disaster

[00:21:46] Thanks for your answers there to that question from the audience. I'm just doing a time check: it's 22 past, so we still have a small bit of time. Maybe we'll take a quick turn to the riskier side. I want to ask a little bit about what we might call pitfalls or common traps. Maybe there's something you'd like to highlight from your own experience or your knowledge of the industry, relating to things to watch out for if you're new to putting together a team or a project in AI or ML. Who wants to take that? Maya, do you want to have a go at that one?

[00:22:27] Maya: Sure. I've not led machine learning projects, so my experience would be more about how we communicate such past experience through the courses that we teach. I think an interesting topic in this area, which all of us have to tackle with machine learning projects in general, is how we work with algorithmic bias that gets fed into the data, and the different ways that it seeps into our machine learning models. In the courses that we teach, we really highlight this and spend a great deal of time talking to all of our stakeholders. It's usually a very debated topic, and it's always really interesting to have those conversations out, and again to provide the space for engineers and data scientists and designers and product managers to actually talk about this.

[00:23:39] And to really consider how different data can introduce bias, but also how different metrics can introduce bias, and all the different places where this can happen along your development cycle. That's really interesting to consider, along with what our internal best practices are at Spotify. We have an algorithmic bias team that works really hard to give us the best tools available to deal with that. So I think it's also about making sure that people are really aware of our best practices to avoid such pitfalls, and of how to reach out to teams that are experts in these areas and use those resources. I think that's really, really important. There are a lot of people who have learned things from the past, so making sure that knowledge is contained within your organization and shared around is really crucial. Because, as you said, there's a lot of experimentation, and if we are not learning collectively from it, it's a lost opportunity.

[00:24:52] Ekaterina: And I think the culture of learning is also very important. Many companies really embrace a no-blame culture, learning from the experience instead and building on top of that. I think that's also a key component: you can evaluate and then reflect on what went wrong and when. Another way you can bring people together and help them find the riskier parts is when you talk about success metrics or success criteria. What does success look like? But you can also talk about what disaster looks like, because it's also very important to be frank at the very early discovery stage, not to kill the idea, but really to be aware of what risks you can encounter. Then you can put in guardrail metrics, or check your data for bias or for misrepresentation of your customer base. Or you could work really closely with engineers and find solutions together to address those risks. Because otherwise, if you just hand it over and say, train the model, without addressing the more holistic perspective, you wouldn't be as successful.

Pitfalls in execution: data engineering and testing

[00:26:15] Adam: It seems a little similar, what you're talking about there, to the idea of a pre-mortem: pitching yourself into the future and imagining. Place yourself in the future, something's gone wrong, what is it? To try and really grab it concretely. And Dana, in your presentation, one of the things I really liked was that it got a bit more technical, down to the business end. One of the lessons you were driving home was that you're really going to need to test this stuff before it goes out. You might find yourself doing data quality testing, doing dry runs, doing betas. It's often hard to get an observation of the system that gives you enough of the guarantee that you need to release. Maybe you could talk a little about that in terms of pitfalls.

[00:26:57] Dana: Yeah. When it comes to things to watch for, I'll present the execution part. Data engineering is something to watch for. I believe it's usually underestimated in machine learning objectives. The next one is how you test it: how you enable the dry run, for how long, who is looking at the results, and then the quality checks or data validation. A risk or pitfall that we discovered is that we want more tools available to do that. So this is also something that could make machine learning objectives take longer than expected.

[00:27:55] Adam: Yeah, for sure. And anyone who's worked in the field for the last couple of years knows there are so many more tools than there were even three or four years ago to support this. Long may that continue. I have time for one quick question from the floor, and then we might just have a final round of the table. This is for Ekaterina and Maya. We talked a little about training programs and rollouts. For companies with a smaller number of employees, maybe fewer than 50, how might they adopt some of your approaches and learnings from education in a smaller environment, with maybe only one or two teams of engineers or product managers?

[00:28:38] Maya: I think the first thing that you need to assess is whether you have internal expertise to share across your organization. Most of the time there is such expertise, even if it's quite scarce. Getting those people on board, and finding out whether they can share some of their learnings and experience, is a really powerful way, because you can relate directly to the use cases in your organization. I think that's probably the best way. The second thing, if you find that there isn't internal expertise, is finding the best outside solutions to bring that expertise to your team. That's the second option, and it actually happens quite often early on in organizations.

Closing advice

[00:29:29] Adam: Finding the support that you need if you don't think you have it in house is a really important message. So, to close, maybe for those in the audience who are new to ML and are going to get their toes wet, give me one line of inspiration or advice you might leave them with. I might start with you, Dana.

[00:29:51] Dana: Yeah. We discussed at the beginning that it's important to find the use case. But if finding the use case takes you, I don't know, one year, I would recommend just trying it. Even though you will fail, you will learn so much from it.

[00:30:09] Adam: Absolutely: embrace failure and learning, for sure, and focus on the use case. Ekaterina?

[00:30:17] Ekaterina: My advice is the opposite of Dana's, finally. Think about your customer needs and your use case, and really challenge yourself whether you need machine learning. You will always be able to build that out and build the competence, but is it really the best course right now to spend your time and effort on?

[00:30:37] Adam: I think already we see the benefit of having the right language and having the debate back and forth about what the right thing to do is, the microcosm of it here.

[00:30:46] Maya: And my final thoughts will be on the balance. My final words would be: think about machine learning as something that everybody can do and participate in a conversation about. It's often taught as a very technical subject, and I think it's really, really important that everybody understands that it's not something reserved only for data scientists and machine learning engineers.

[00:31:14] Adam: Yeah, I think the thread holding all of that together is how we do experimentation as a way of working, and how we manage that in the business, not in a real top-down way, but in a way that we can manage ourselves and our teams. So I just want to close by thanking everyone: Maya, Dana, Ekaterina, I've really enjoyed the chat, and I also really enjoyed talking with you before this and learning from your experiences. I watched your presentations twice; they were great. We'll be hanging around in the Slack for any follow-up questions, and I'll keep an eye on it if anyone wants to tag me for anything. And I really want to thank the organizers as well for inviting us today. I really appreciate the opportunity.