Why So Many ML Models Don't Make It To Production?

Dana Aonofriesei at UXDX EMEA. Video: https://www.youtube.com/watch?v=KvANvzSP7lE

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why ML models stall before production

[00:00:00] Hi, everyone. I hope you are enjoying your time at the UXDX event. I am Dana, Director of Engineering at Trustpilot. I'm based in Copenhagen, and I work in the content moderation, fraud and online abuse detection space. I am blessed and grateful to be working with talented people on solving business problems with machine learning, and I'm here today to share a few of our learnings in that space.

[00:00:31] Trustpilot is a reviews platform with 800 employees based in eight locations. There are more than 100 million reviews on the platform, and a bit more than 500 websites with reviews on Trustpilot.

[00:00:55] Now, back to our topic: why so many machine learning models don't make it to production. Some say that as many as 87% of data science projects don't make it to production. I dug deeper into this topic, and in a Gartner report I found that 37% of 3,000 surveyed leaders had deployed AI or would do so shortly. I know that this number could look small, but it is an increase of 270% in four years, which shows the focus in the industry on machine learning and AI.

[00:01:43] At the same time, in a McKinsey report, I found that of 160 reviewed AI use cases, 88% did not progress beyond the experimental stage. So what is the root cause? Digging deeper into this topic, I found out that sometimes the cause could be performance or quality concerns around the model. Sometimes companies find a simpler or more cost-effective solution to their business problem, so they give up on machine learning. And sometimes lack of management support could also be the root cause.

Data: labeled, available and democratised

[00:02:32] Now let's imagine that we do have management support and a strong business case for applying machine learning. Let's see what other impediments or blockers we could experience in shipping machine learning to production. When it comes to the data, my main learning is that high-quality, labeled and available data keeps being an expensive asset. So brace yourselves.

[00:03:07] For example, you may need labeled data for your machine learning. By labeled data I mean data that has, let's say, tags referring to a classification or categorisation. If you need labeled data, companies usually have a few avenues to choose from: either to buy it, to label in house, or to label externally in case they don't have the capacity to do it in house. Each of these options requires coordination, alignment, research and quality checks. In reality, this effort could take months, and of course months of money invested.

[00:03:59] When it comes to data availability, there are usually two scenarios. Either the data is available in your company but not easily accessible because of a siloed data approach, or the data is available but, in order to be used for machine learning, it requires cleaning, processing, extraction and similar work. Either way, the effort could take months.

[00:04:35] My recommendation in this space, if you really have machine learning on the company agenda, is first to invest in data as a service. This is a recommendation that you'll find all over the internet from other leaders. It's about democratising your data, making it available to your data scientists and engineers. I found this image from PwC very interesting, where they present how democratising data and AI could look for a company. The secret here is that once you start to democratise your data, further opportunities will come, and you'll probably consider in the future democratising model training, the deployment process and all the other components.

[00:05:44] Another recommendation in the data space: if you continuously need labeled data for your machine learning, I would recommend considering embedding the labeling system in your existing products, so users label data when they're using the product. The best-known example, which you probably know, is Google Photos asking us to confirm the faces in different pictures.

Tools: MLOps and ML tech debt

[00:06:20] When it comes to tools, my main learning in this space is this. When I was thinking about machine learning, I was mainly thinking about data science and data scientists, and I believe this didn't happen only to me. In reality, in order to ship machine learning to production, more is needed: more principles, engineering skills, tools and other components. I found this view very interesting, showing how machine learning could look with those components and tools, or without them. We do have to think about retraining the model, versioning the model, monitoring, the data pipeline and other components.

[00:07:26] Google has a similar view explaining the components of a machine learning system. They mention configuration management, feature engineering, data verification and monitoring. So machine learning is not only about developing the ML code, but also about considering all the tools needed to ship it to production. The same way we saw the rise of DevOps a few years ago, we are now seeing the rise of MLOps. MLOps refers to techniques for continuous integration, continuous delivery, configuration management and other automation in machine learning.

[00:08:24] My main recommendation in this space is to define the ML components needed or desired and invest in developing those components. Think about the foundation that you need and invest in it. There are many learnings from the DevOps philosophy, so I do hope that we can apply some of those learnings in MLOps.

[00:09:02] My other recommendation when it comes to tooling is to define what ML tech debt means. Sooner or later this will come to your backlog and to your team. My recommendation is to gather your team and discuss responsibilities: what can data scientists do around tech debt, what can data engineering do, and what are the tools or practices to consider in this space? Sooner or later ML tech debt will come to your backlog.

Processes: harmonising ML and software development

[00:09:39] When it comes to processes, the main learning in this space was that, again, as I shared earlier, in order to ship machine learning to production, besides the machine learning knowledge and skills, there are more roles and skills involved. You need data engineers, configuration management and operations. What we discovered in this area was this. I'm proud to say that Trustpilot is a fast-paced environment with frequent releases, and our assumption was that machine learning development would just follow that process, because it was so embedded and so strong.

[00:10:42] We were wrong. We discovered that, in parallel with our software development process, or on top of it, we also have to accommodate machine learning development, which is different. It's different in the sense that in software development we have daily or weekly releases. In machine learning development, maybe you have a one-time release for your machine learning, and then a lot of time is invested in experimenting or data discovery, which could take weeks, so there are no daily or weekly releases.

[00:11:32] When it comes to the profiles involved, engineers are builders. They look at the problem and ask themselves, "How do I build this?" Data scientists, on the other hand, start the work with a question or an assumption, and they work towards confirming that assumption or answering that question. That's also something to consider in ways of working. The question here is how to bring engineers and data scientists together. In a way this is not a new problem. We have had this conversation about how to bring testers closer to engineers, or how to bring site reliability closer to engineers.

[00:12:25] My recommendation in this area is to consider multidisciplinary teams: have your data scientists, engineers, designers and product manager working together in a team. Now, we all know that this is not enough for collaboration. What we considered in our teams was defining the touchpoints needed between data scientists and engineers, so we avoid a waterfall approach in developing and shipping machine learning. By touchpoints I mean documentation such as requests for comments, demos or reviews: the kinds of activities that bring data scientists and engineers together.

People: assumptions, checklists and upskilling

[00:13:27] Now, when it comes to the people, a few learnings in this area. Initially I wanted to name this chapter "Existing systems." The logic behind it was that we noticed that our products and systems, as they are today, are not always ready to absorb machine learning models. I can't put the blame on the systems, but I can put my hope in the people that are building and maintaining those systems. And when I'm referring to people, I'm not referring only to data engineers and data scientists; I'm also referring to product managers, UX designers and UX researchers.

[00:14:28] We all know that we will have our own assumptions around user experience, the results of the model, or the development of the model. So it's crucial to engage the team in testing their assumptions and asking questions around those assumptions. My recommendation is, as always, to focus on user experience: explore and understand how users will interact with the predictions or the outcome of the model. How will users behave if they know there is AI behind the product? There is also the element of trust to be considered there.

[00:15:13] When it comes to developing and shipping machine learning, find a way to spark conversation between team members. What we're considering here is a checklist of questions that the team can consider when grooming machine learning development. Things to be considered on the checklist are, for example: what is the complexity of the model? If you compare this model with the previous model, what's the complexity? Our mistake in the past was maybe to look at the models as being the same, which was not the case. Other questions to consider for the checklist are: how can we approach machine learning in a more iterative way? Do we need all the features of the model in this iteration? What are the data quality checks to consider? And discuss with your team how to test, or how to perform a dry run before you launch the model. We're working on expanding the list, and I'm curious to see similar lists coming out from the industry as well.

[00:17:00] My last recommendation is in the people area. In one of the previous reports from Gartner, they shared that for leaders who have machine learning or AI on their agenda, the main challenge they reported is a shortage of talent related to AI or machine learning. That's why my recommendation is, when developing machine learning, to also facilitate learning among team members and to think about upskilling where this is the case.

Summary

[00:17:45] Now a short summary of what we discussed. Again, if machine learning is on your agenda, invest in data as a service, enabling and unlocking the potential of your teams. Define machine learning ops in your organisation: what is the foundation that you need in that space? Find the harmony between machine learning development and software development, and facilitate learning and growing skills within your company. I would love to continue these conversations, so please reach out to me on LinkedIn or Twitter. I'm happy to share more learnings, and thank you so much.