Knowledge Is Power. Make Machine Learning Your Superpower

Ekaterina Garbaruk Monnot, Maya Bogdanova at UXDX EMEA. Video: https://youtu.be/RnmchZt5TOw

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Why we built a machine learning training for all of R&D

[00:00:00] Maya: We're here to take you on a journey of how and, more importantly, why we created a training called Machine Learning for All R&D at Spotify. We hope at the end of this talk you'll be able to take away three main lessons we've learned, and then hopefully apply them to your organization and context. My name is Maya Bogdanova and I'm a Senior Educational Program Manager at Spotify. I'm part of a team that develops and runs internal technical trainings. I lead the education efforts in relation to machine learning engineering for our R&D community.

[00:00:32] Ekaterina: Hi, my name is Ekaterina and I'm a Director of Product at Pleo. At Pleo, we provide a spending solution for forward-thinking teams. In my previous position, I was a Senior Product Manager at Spotify, and today during this presentation I will share with you my experience from my time at Spotify.

[00:00:58] Ekaterina: Let's first set the scene. Machine learning has been gaining popularity since the start of the century, and today it would be hard to name any industry where machine learning powered products are not in use or being developed. The benefits of using machine learning for personalization, two-sided markets, forecasting, natural language processing, just to name a few, are undeniable. But so are the large costs associated with productionalizing a machine learning solution.

[00:01:35] Ekaterina: First, there are data costs: data collection, data transformation and engineering. Then there are system costs: machine learning infrastructure, integration with other systems, automated QA and alerting. And talent costs, I'm sure you all can relate to that: excellent researchers, machine learning experts developing and tuning the model, data engineers building pipelines, data scientists producing insights, and back-end engineers building scaled infrastructure. And don't forget the impact on the customers. When you introduce machine learning in your product, there are certain aspects of control that go away, so it might be leading to surprising aspects for the customer.

[00:02:19] Ekaterina: At Spotify, we have already seen the benefits of machine learning powered solutions, and we have a large appetite for improvements of our product and delivering more value to our customers. We want to democratize machine learning solutions, but we know that this has to come with a very clear need assessment, informed decisions and stakeholders that are on board.

The context at Spotify

[00:02:45] Maya: Now, in order to also understand how we went about tackling this at Spotify, you need to know a little bit more about the context in which we work at Spotify. First thing, Spotify has a very strong tech learning culture. In 2018, the tech learning team was born to support existing community efforts at Spotify. In other words, Spotifiers were creating and teaching and running trainings before the team I'm part of existed, just at a smaller scale and more ad hoc in nature. The exchange of skills between Spotifiers is the core of our educational programs, because it's at the core of our culture at Spotify. Just to give you a sense of what I mean, last year, 2020, over 220 people at Spotify volunteered to teach and facilitate more than 130 courses and onboarding programs.

[00:03:43] Maya: Second thing you need to know about us is that tech learning focuses exclusively on technical competencies, and all of our employees are part of R&D. We are a team embedded within platform mission, or our equivalent of infrastructure and operations, and we're a learning and development team, an Infra team. And like other Infra teams around us, we aim to unblock people and unleash their productivity. We just do that by providing necessary technical skills and knowledge. Enrollment in all of our programs is voluntary and self-motivated. Last year, our programs reached over 1,800 Spotifiers.

[00:04:36] Maya: And the third thing you need to know about our context is that Spotify has used machine learning solutions in its product since its early days. Recommending content is the most obvious place where machine learning solutions flourished at Spotify, and for a while the power of machine learning was concentrated around personalization. But to truly benefit from machine learning, we knew we needed to democratize it and make sure that everyone in the organization understands both the benefits and the costs associated with it.

[00:05:07] Maya: Machine learning is only powerful when used for the right problems. But how would we find the right problems if exploration and experimentation comes at a cost both for you and for your customers? And even more importantly, how could you find the right problems if cross-functional teams don't have a common understanding of machine learning? We had to empower our teams across the org to consider machine learning as any other tool in the toolbox, irrespective of the product area and expertise, and be able to explore that in a way that was informed and systematic.

The first attempt and what it taught us

[00:05:54] Maya: Let me tell you a little bit about our journey. We did what we knew worked well in the past at Spotify with a technical skill: we looked at the existing grassroots efforts to upskill people with machine learning on the smallest scale, and that was a machine learning bootcamp in existence. We started running a training aimed at engineers and data scientists who wanted to learn about machine learning at Spotify. We launched a training called Fundamentals of Machine Learning internally and we saw a huge demand. We were flooded by interest. Everyone was excited for what that future would be, all engineers holding hands and working on ML solutions.

[00:06:35] Maya: But this was far from the reality. Fifty-two people started the first course and 10 were able to complete it, and even those 10 didn't really think that the skills matched up to their needs. But why? How come so many people were hungry for machine learning skills but were not able to integrate those skills into their work?

[00:07:04] Maya: Sure, we could have spent the next year working out ways to improve this training. But instead we took the time to look deeper into the feedback and find the underlying reasons. What we saw was that by focusing on the technical members in our community, we made two mistakes. One, we left out many decision-makers for ML solutions. And two, we perpetuated the assumption that machine learning knowledge is the domain of engineers and data scientists. In other words, the missing piece was fertile soil for ML solutions to flourish.

[00:07:46] Maya: To drive change, we had to empower all decisional stakeholders, such as product managers, engineering managers, leads, researchers, designers. Anybody should be able to take part in that conversation. This may seem kind of obvious now, but when you're dealing with a subject like machine learning, it is hard to imagine a world where everyone irrespective of their background could or should be able to debate machine learning solutions.

The parallel grassroots effort

[00:08:14] Ekaterina: Parallel to these efforts in our community, there was an undercurrent that was focusing on a completely different group of people. A few years back, a grassroots training led mostly by members of the personalization mission worked to fill in the gap of knowledge for product managers. Also, roughly the same time when tech learning was evaluating the program and figuring out how to move on, there was another initiative led by a couple of Spotifiers.

[00:08:45] Ekaterina: Me and another colleague, an engineering manager, met at the coffee machine, and we started talking about machine learning. And after coffee we thought, oh, it would be so great to have more of this discussion across the community, whether that's happening near the coffee machine or within the meeting group. We saw that there is a gap in the knowledge around our peers, and we decided to do something about it.

[00:09:20] Ekaterina: Remember that we said that at Spotify there is a strong learning culture driven by the community itself. Well, this is where it all started, as part of a hack week project. We put together a workshop called Machine Learning for Product Managers. We ran the workshop for a couple of tribes, sort of departments, and then we met Maya in tech learning, and when they heard about these efforts it was a perfect opportunity for us to collaborate and take these efforts and scale them out to all disciplines and all of Spotify.

[00:09:59] Ekaterina: First, we started with a team of four people when we decided to take it out to the whole company: two product managers, one machine learning engineer and one engineering manager. And then there were two half days of the course that we named Machine Learning for All R&D. A year later, we have over 30 contributors to the course team, teaching, facilitating and improving the content, and over 210 Spotifiers have taken part in the training. Anyone is welcome: product managers, designers, user researchers, agile coaches, engineering, anyone really. And we have an audience as diverse as the community at Spotify.

The outcomes

[00:10:48] Ekaterina: Well, you might say that's all very nice, but what is the tangible outcome? To be honest, we cannot talk about all the details of the initiatives that started after the course, but we are very sure that you will soon see them in the Spotify app and you will enjoy the experience. The return on investment of learning experiences is notoriously hard to measure, but these are some outcomes that we can share with you. 99.5% of the participants in the course agree that they now have a good mental model and basic intuition about machine learning at Spotify, and the reviews of the learning experience are consistently 4.5 out of five.

[00:11:40] Ekaterina: But the numbers are only half of the story. This is what the participants of the course say. "Within one day I was already using the training in my everyday work and feeling more confident having conversations about it. It even affected my roadmap plans in a really positive way. The practical application of this course was the best part for me."

[00:12:04] Maya: Another person said, "I enjoyed the discussions with various people in different domains about machine learning solutions. I liked that the course offered a good focus on determining factors, whether machine learning is the appropriate solution for a use case or not."

[00:12:27] Ekaterina: "I started this course thinking that a bold machine learning future would mean less need for disciplines like user research, but something that stood out today, if anything, there was more need. Courses like this provide language and places for a focus."

Lesson 1: empower and educate all business stakeholders

[00:12:40] Maya: The journey of this training still continues, but we learned three fundamental lessons that we want to share with you, in the hopes that you can apply them to your context. Lesson one: empower and educate all business stakeholders.

[00:13:02] Maya: We started with a deceptively simple objective: by the end of the training, everyone should be able to identify and discuss ML solutions with their teams. But if you think about it, in order to talk about any subject you need to know enough about that subject, be able to evaluate the need of your context, and know enough about the perspective of the other people in the conversation with you. We focused the course on providing a wide range of topics and lots of discussion opportunities. Diversity of content and participants meant diversity of perspectives and experiences. This is how we built what we call at Spotify tech empathy.

[00:13:45] Maya: What does it look like in practice? There are three elements that we distilled from this. Create sessions around the needs of each type of stakeholder. This way, we made sure that all sessions provide an insight into how different disciplines approach machine learning and what problems they might encounter. Of course we have sessions like intro to machine learning and types of machine learning, but we focus on sessions like applying machine learning to your product, data for machine learning, machine learning engineering, design for machine learning, algorithmic bias, exploring all the different points of view in the ML solution development.

[00:14:30] Maya: Incorporate also, and think about, the context of your learners into the training. Get people to bring problems to apply and test their understanding, get people to brainstorm ideas. It doesn't have to be viable ideas, but it's through that exploration that they learn how to evaluate and talk about ML solutions. And the third thing is build tech empathy by creating lots of opportunities for people from different backgrounds and roles to talk about problems and solutions. In essence, what we hope you take away from this lesson is that true power lies in bringing technical knowledge to your whole community.

Lesson 2: design and develop educational programs as products

[00:15:08] Ekaterina: Lesson two: design and develop educational programs as products. I have to admit, this is my favorite lesson as a product manager. We at Spotify think of educational programs as products, with continuous integration cycles, customers to listen to, an addressable market and all else that comes with it. This mindset also means that we started with a minimum viable product, or minimum viable course, and then we moved on. We collected feedback at every step of the way to discover gaps and opportunities for improvement.

[00:15:52] Ekaterina: In fact, 10 deliveries down the line we're still iterating and improving. The new minimum viable product of a training was a six-hour solution. Today we have 10 hours' worth of content delivered over three days. By listening to the feedback from the customers we have been able to create truly learner-centric content and experience. But the number of hours doesn't matter so much. What matters is that during the iterations we were working on several aspects.

[00:16:24] Ekaterina: We focused on the structure of the course and sequencing: what sessions need to be included, how long should they be and in what order. Content of the individual sessions: what kind of depth should the content go into, or how should we keep it very friendly to any kind of listener. And lastly, strengthening the learning outcomes through practical exercises and built-in discussions. This is also the part that they really love, how we bring all the experiences to the table.

[00:17:03] Ekaterina: While we have a vision and a strategy of what the course aims to achieve, we're flexible in the details and listen carefully to the input during and after each program. You would ask how we gather all these feedbacks. There is a very practical example: after each day of the course, participants would get a form to fill. It just has a couple of questions, but it helps facilitators and instructors to adapt the content, or the depth of the content, for the following day.

Lesson 3: tap into a pool of experience as diverse as your learners

[00:17:34] Maya: Lesson three: tap into a pool of experience as diverse as your learners. Remember, we started the course with a team of four content developers who volunteered their time to create the course and teach the first cohort. To scale this to hundreds of people, we knew we had to expand the team. We started recruiting people and we cast a very large net. By tapping into the full breadth of stakeholders of ML solutions, we were able to scale the training and the content.

[00:18:05] Maya: The collective experience of the original team of four that Ekaterina was part of shaped the core content of this course, but that was just the beginning, because of the iterative nature of the development. Every time somebody new joined the course team, they had the opportunity to shape the content and build on it. So the course team teaching the cohorts now includes many more perspectives: data scientists, data engineers, designers, back-end engineers, insights managers.

[00:18:46] Maya: And what's more, they don't just teach a session or a topic. There are teams supporting each other and facilitating all content together. They participate in all discussions, in all conversations. It doesn't matter if the conversation is about data model evaluation or algorithmic bias. Each teaching team adds a layer of personal experience and practical Spotify-specific knowledge that cannot be communicated through a deck. They share a diversity of problems and opportunities. These are the stories that our learners connect with. These are the stories that stay with you months after the training.

[00:19:24] Maya: In many ways, the course team learns from each other just as much as the participants, and that's how the training becomes a conversation, not a lecture. It also indirectly communicates one very important message to our community: as far as machine learning is concerned, we're all learners here and we can all learn from each other.

[00:19:50] Ekaterina: Have we arrived at our destination? No, this is just the beginning of what we can do with the collective power of our experiences. We continue to build on the lessons that we shared with you. Lesson one, educate and empower all business stakeholders. Two, design and develop educational programs as products. Three, tap into a pool of experiences as diverse as your learners. We are able to use these lessons and apply them to other technical subjects and empower our organization. Because, although it's a cliche, knowledge is power, and we encourage you to take these lessons and provide the opportunity to your teams and companies to develop superpowers. Thank you.