Knowledge Is Power. Make Machine Learning Your Superpower

Knowledge / Inspiration

Knowledge Is Power. Make Machine Learning Your Superpower

Enabling the Team

Spotify has used Machine Learning solutions in its products since the early days. Recommending content is the most obvious place where Machine Learning solutions flourished at Spotify early on. We had seen the benefits of ML powered solutions and had a large appetite for delivering more value to our customers. But we also saw the costs of democratizing Machine Learning solutions across the organization. We knew that Machine Learning solutions require a conscious, informed decision by all stakeholders of a product.
So how did Spotify promote exploration and conscious use of Machine Learning?
We will share 3 lessons learnt a year of running a Machine Learning training for all members of RnD from product to research, to engineering and design.

Maya: We're here to take you on a journey of how and more importantly, why we created a training called Machine Learning for all R&D at Spotify. We hope at the end of this talk, you'll be able to take away three main lessons, we've learned and then hopefully apply them to your organization and context.

My name is Maya Bogdanova and I'm a Senior Educational Program Manager at Spotify. I'm part of a team that develops and runs internal technical trainings. I lead the education efforts in relation to machine learning, engineering for our R&D community.

Ekatrina: Hi, my name is Ekatrina and I'm a Director of Product at Pleo. At Pleo, we provide spending solution for forward thinking teams. In my previous position, I was a Senior Product Manager at Spotify. And today during this presentation, I will share with you my experience from my time at Spotify.

Let's first set the scene. Machine Learning has been gaining popularity since the start of the century. And today, it would be hard to name any industry where machine learning powered products are not in use or being developed. The benefits to use machine learning for personalisation, two-sided markets, forecasting, natural language processing, just to name the few are undeniable. But so are the large cost associated with productionalising machine learning solution. First, there are data costs, data collection, data transformation, and engineering. Then there are system costs, machine learning infrastructure, integration with other system, automated QA and alerting and talent costs. I'm sure you all can relate to that. Excellent researchers, machine learning experts, developing and tuning the model, data engineers, building pipelines, data scientists producing insights and back-end engineers, building scaled infrastructure. And don't forget the impact on the customers. When you introduced machine learning in your product, there are certain aspect of control that goes away. So, it might be leading to surprising aspects for the customer.

At Spotify, we have already seen the benefits of machine learning powered solutions, and we have a large appetite for improvements of our product and delivering more value to our customers. We want to democratise machine learning solutions, but we know that this has to come with a very clear need assessment, informed decisions and stakeholders that are onboard.

Maya: Now, in order to also understand how we went about tackling this at Spotify, you need to know a little bit more about the context in which we work at Spotify.

So, first thing, Spotify has a very strong tech learning culture. In 2018, the tech learning team was born to support existing community efforts as Spotify. In other words, Spotifiers were creating and teaching and running trainings when before the team I'm part of existed and just at a smaller scale and more ad-hoc in nature, their exchange of skills between Spotify is the core of our educational programs, because it's at the core of our culture at Spotify. Just to give you a sense of what I mean last year, 2020, over 220 people at Spotify volunteered to teach and facilitate more than 130 courses in onboarding programs.

Second thing you need to know about us is that tech learning focuses exclusively on technical competencies and sort of all employees for part of R&D. We are a team embedded within platform mission, or our equivalent of infrastructure and operations. And we're learning development team Info team. And like other Info teams around us, we aim to unblock people and unleash their productivity. We just do that by providing necessary technical skills and knowledge. Enrollment in all of our programs is voluntary and self-motivated. Last year, our programs reach over 1,800 Spotifiers and the third thing you need to know about our context is that Spotify has used machine learning solutions in its product since its early days recommending content is the most obvious place where machine learning solutions flourished at Spotify. And for a while, the power of machine learning was concentrated around personalisation. But to truly benefit from machine learning, we knew we needed to democratise it and make sure that everyone in the organisation understands both the benefits and the costs associated with it.

Machine learning is only powerful when used for the right problems, but how would we find the right problems if exploration, experimentation comes at the cost of both for you and for your customers? And even more importantly, how could you find the right problems, if cross-functional teams don't have the common understanding of machine learning. We had to empower our teams across the org to consider machine learning as any other tool in the toolbox, irrespective of the product area and expertise, and be able to explore that in a way that was informed and systematic.

So, let me tell you a little bit about our journey. We did what we knew worked well in the past at Spotify without a technical skill, we looked at the existing grassroots efforts to upskill people with machine learning on the smallest scale and that was a machine learning bootcamp in existence. We started running a training aimed at engineers and data scientists who wanted to learn about machine learning at Spotify. We launched a training called Fundamentals of Machine Learning internally and we saw a huge demand. We were flooded by interest. Everyone was excited for what that future will be. All engineers holding hands and working on ML solutions. But this was far from the reality. 52 people started the first course in 10 were able to complete it and even those 10 didn't really think that the skills matched up their needs, but why? How come so many people were hungry for machine learning skills, but were not able to integrate those skills into their work? Sure, we could have spent the next year working ways to improve this training. But instead, we took the time to look deeper into the feedback and find the underlying reasons of what we saw was that focusing on your technical members in our community, we made two mistakes.

One, we left out many decision-makers for ML solutions and two, we've perpetuated the assumption that the machine learning knowledge is the domain of engineers and data scientists. In other words, the missing piece was fertile soil for ML solutions to flourish. To drive change, we had to empower all decisional stakeholders such as product managers, engineering managers, leads, researchers, designers, anybody should be able to take part of that conversation. This may seem kind of obvious now, but when you're dealing with a subject like machine learning, it is hard to imagine a world where everyone irrespective of their background could or should be able to debate machine learning solutions.

**Ekatrina: **Parallel to these efforts in our community, there was an undercurrent that was focusing on a completely different group of people.

A few years back and grassroots training led mostly by members of personalisation mission work to fill in the gap of knowledge for product managers. Also, roughly the same time when tech learning was evaluating the program and figuring out how to move on, there was another initiative led by a couple of Spotifiers. Me and another colleague, an engineering manager met at the coffee machine. And we started talking about machine learning and after coffee, we thought, "Oh, it would be so great to have more of this discussion across the community", whether that's happening near the coffee machine or within the meeting group. We saw that there is a gap in the knowledge around our peers, and we decided to do something about it.

Remember that we said that at Spotify, there is a strong learning culture driven by the community itself. Well, this is where it all started as a part of a hack week project. We also put together a workshop called Machine Learning for Product Managers. We ran the workshop for a couple of tribes, sort of departments and then we met Maya in tech learning when they heard about this efforts was so perfect opportunity for us to collaborate and take this efforts and scale them out to all disciplines and all of Spotify.

First, we started with a team of four people when we decided to take it out to the whole company, two product managers, one machine learning engineer, and one engineering manager. And then there were two half days of the course that we named machine learning for all R&D. A year later, we have over 30 contributors to the course team teaching, facilitating and improving the content. And over 210 Spotifiers have taken part in the training. Anyone is welcome. Product managers, designers, user researchers, agile coaches, engineering anyone really, and we have an audience as diverse as the community at Spotify. Well, you might say that's all very nice, but what is the tangible outcome? To be honest, we cannot talk about all the details of the initiatives that started after the course, but we are very sure that you will soon see them in the Spotify app and you will enjoy the experience. The return on investment of learning experiences is notoriously hard to measure, but these are some outcomes that we can share with you: 99.5% of the participants in the course agree that they now have a good mental model and basic intuition about machine learning at Spotify.

And the reviews of the learning experience are consistent between 4.5 out of five, but the numbers are only half of the story. This is what the participants of the course say. ‘Within one day, I was already using the training and my everyday work and feeling more confident having conversations about it. Even the fact that my roadmap plans in a really positive way. The practical application of this course was the best part for me.’

**Maya: **Another person said ‘I enjoyed the discussions with various people in different domains about machine learning solutions. I liked that the course offered a good focus on deterministic factors whether machine learning is the appropriate solution for used case or not.’

Ekatrina: ‘I started this course thinking that the bold machine learning futures would mean less need for disciplines like user research, but something that stood out today, if anything, there was more need. Courses like this provide language and places for a focus.’

Maya: The journey of this training still continues, but we learned three fundamental lessons that we want to share with you in the hopes that you can apply to your context.

Lesson 1: Empower and Educate all Business Stakeholders. We started with an inclusively simple objective by the end of the training, everyone should be able to identify and discuss ML solutions with their teams. But if you think about it in order to talk about any subject, you need to know enough about that subject, be able to evaluate the need of your context and enough about the perspective of the other people in the conversation with you. So, we're focused the course on providing a wide range of topics and lots of discussion opportunities, diversity of content and participants meant diversity of perspective and experiences. This is how we built what we call at Spotify, Tech Empathy. So, what does it look like in practice? There are three elements that we kind of distilled from this. Create sessions around the needs of each type of stakeholders. This way, we made sure that all sessions provide an insight into how different disciplines approach machine learning and what problems they might encounter. Of course, we have sessions like machine learning, intro to machine learning and types of machine learning, but we focus on sessions like applying machine learning to your product, data for machine learning, machine learning engineering, design for machine learning, algorithmic bias, exploring all the different points of view in the ML solution development. Incorporate also, and think about the context of your learners into the training, get people to bring problems to apply and test their understanding, get people to brainstorm ideas. It doesn't have to be viable ideas, but it's through that exploration that they learn how to evaluate and talk about ML solutions. And third thing is build tech empathy by creating lots of opportunities for people from different backgrounds and roles to talk about problems and solutions. In essence, what we hope you take away from this lesson is that true power lies in bringing technical knowledge to your whole community.

Ekatrina: Lesson 2: Design and Develop Educational Programs as Products. I have to admit, this is my favorite lesson as a product manager. We at Spotify think of educational programs as products with continuous integration cycles, customers to listen to, addressable market and all else that comes with it. This mindset also means that when we started with a minimum viable product or minimum viable course, and then we moved on, we collected feedback at every step of the way to discover gaps and opportunities for improvement. In fact, 10 deliveries down the line we're still iterating and improving the new minimum viable product of a training was a 6 hour solution. Today, we have 10 hours’ worth of content delivered over three days. By listening to the feedback from the customers have been able to create a truly learner centric content and experience, but the numbers of hours doesn't matter so much. What matters is that during the situations we were working on several aspects. We focused on the structure of the course and sequencing, what sessions need to be included, how long should they be and in what order? Content of the individual sessions, what kind of depth should the content go in or how should we keep it very friendly to any kind of listener and lastly strengthening the learning outcomes through practical exercises and built-in discussions. This is also the part that they really love, how we bring all the experiences to the table. While we have a vision and a strategy of what the course aims to achieve. We're flexible in the details and listen carefully to the input during and after each program, you would ask how together all these feedbacks. There is a very practical example. After each day of the course, participants would get a form to fill. It just has a couple of questions, but it helps facilitators and instructors to adapt the content or the depth of the content for that following base.

Maya: Lesson 3: Tap into the Pool of Experience as Diverse as your Learners. Remember, we started the course with a team of four content developers who volunteered their time to create the course and teach the first cohort. To scale this to hundreds of people, we knew we had to expand the team. We started recruiting people and we cast a very large net. By tapping into the full breadth of stakeholders of ML solutions. We were able to scale the training in the content, the collective experience of the original team of four that Ekatrina was part of, shaped the core content of this course, but that was just the beginning because of the iterative nature of the development. Every time somebody new joined the course team, they had the opportunity to shape the content and build on it. So, the course team teaching the cohorts now includes many more perspectives, data scientists, data engineers, designers, back-end engineers, insights managers, and what's more they don't just teach a session or a topic. There are teams supporting each other and facilitating all content together. They participate in all discussions in all conversations. It doesn't matter. The conversation is about data model evaluation or algorithmic bias. Each teaching team adds a layer of personal experience and practical Spotify specific knowledge that cannot be communicated through a deck. They share a diversity of problems and opportunities. These are the stories that our learners connect with. These are the stories that stay with you months after the training. In many ways, the course team learns from each other just as much as the participants. And that's how the training becomes a conversation, not a lecture. It also indirectly communicates one very important message to our community. As far as machine learning is concerned, we're all learners here and we can all learn from each other.

Ekatrina: Have we arrived at our destination? No, this is just the beginning of what we can do with the collective power of our experiences. We continue to build on the lessons that we shared with you. Lesson 1, educate and empower all business stakeholders. 2, design and develop educational programs as products. 3, tap into the pool of experiences, as diverse as your learners.

We are able to use these lessons and apply them to other technical subjects and empower our organization. Because although it's a cliché, knowledge is power, and we encourage you to take these lessons and provide the opportunity to your teams and companies to develop superpowers. Thank you.