Effective Context Switching for Platform Success: Balancing Internal and External Users

Srividhya Chandrasekaran at UXDX Community: UI Efficiency & Balancing User Contexts for Platform Success. Video: https://youtu.be/BCs87uZp8pw

Readable transcript: edited from the recording's captions for readability (fillers and false starts removed, punctuation and section headings added). Wording is the speaker's own. Timestamps are positions in the video. Names marked [?] could not be verified against the audio.

Introduction and background

[00:00:00] Srividhya: I joined Spotify in 2022, and prior to that I was a PM at a startup called Alert Innovation, in the micro-fulfillment space. Alert was eventually acquired by Walmart. We worked with bots that would deliver online grocery orders to Walmart customers. Interesting stuff. In this talk I'm going to focus on sharing insights about effective context switching for platform success, and balancing the needs of both internal and external users.

[00:00:33] A little bit about me. I was born and raised in South India. I have an undergraduate degree in computer science engineering. I moved to the US for career opportunities, and earned an Executive MBA from the Quantic School of Business and Technology. I have decades of experience in test automation and quality engineering, but I organically shifted to product management, and I have found my niche in platform product management.

[00:01:01] I'm currently leading product for Soundcheck, which is a pioneering Backstage commercial plugin. Soundcheck focuses on enabling engineers to effectively track development and operational standards, and Soundcheck plays a crucial role in measuring the tech health of software components, providing invaluable guidance to shape behavior and enhance DevOps practices within organizations. Beyond my role, I'm deeply committed to leveraging technology for positive societal change. I volunteer to empower women in STEM, and in addressing climate change.

What Backstage is and why Spotify built it

[00:01:39] Well, what's Backstage? It might be surprising to some that Spotify, a music app, launching a top-tier developer portal called Backstage was a natural progression. Spotify has a tech-centric approach. We prioritize engineers, fostering a culture of agility and autonomy, and Backstage reflects this commitment, empowering developers to innovate rapidly and at scale. Backstage is used for building internal developer portals. It was created by Spotify, donated to the CNCF, and adopted by thousands of companies.

[00:02:13] Let's go back to the year 2016. Spotify was in hypergrowth mode, expanding from a music streaming app to a highly personalized audio experience. But onboarding new engineers was becoming hyper confusing. Teams were focused on speed to market, but they were slowing down because of internal fragmentation. While our headcount kept going up, our individual developer effectiveness metrics kept going down. We measured the number of days new engineers took to get to their 10th pull request, and it was skyrocketing to 60 days. We knew we needed to solve this, and Backstage was introduced, and this number dropped to just 20 days.

[00:02:53] Every development shop roughly wants the same things: speed without compromising safety, scale without compromising quality, and the ability to tame their increasingly chaotic software ecosystem. Spotify is no different. So Backstage was built to tame the chaos, and that changed everything, aligning a distributed, autonomous culture and bringing together hundreds of teams, thousands of engineers and tens of thousands of software components. So you can imagine the scale. Backstage at Spotify now enables better collaboration, unlocks collective potential, and empowers teams to do what they do best.

[00:03:32] I'll quickly run through the features of Backstage. It's important that we understand this as I dive deep into the core of the presentation. There are three main jobs to be done. You can create new software in seconds. You can manage all the software in one centralized location. And you can explore the entire ecosystem, enabling collaboration across your organization. Instead of being frustrated by infrastructure, your developers can get back control over it. It frees them to focus on what they want to focus on, which is building features, increasing business impact, etc.

Why Soundcheck was built and its impact

[00:04:09] I love retelling the story to help users understand the problem we're trying to solve using Soundcheck, so bear with me before I dive into the core focus of this session. This section is important so that I can share examples of what challenges we faced and how we solved them, so you can learn from those insights.

[00:04:29] Pre-Soundcheck, at Spotify there were around 30,000-plus software components in Backstage, and that number was growing fast. Manual methods to track and maintain these components for quality, reliability or compliance did not scale. This complex spreadsheet that you are seeing here worked for a while, but it had its limitations. Autonomous teams were making technical decisions without much guidance or standardization, which in turn led to fragmentation. This led our internal teams to build Soundcheck. Humans are not that effective when they need to keep track of too many things, right? So Soundcheck does the tracking for you instead. Soundcheck is designed to ensure that software components maintain high quality and align with the company's best practices.

[00:05:15] This is a quick timeline chart. It's inspired by Backstage's timeline chart. If you see here, System Z was born. That's the beginning of Backstage. And this is a quick timeline of Soundcheck. System Z became Backstage at some point, and in 2020 Soundcheck was launched internally for Spotify users. In 2021 Spotify donated Backstage to the CNCF, and in December 2022 we launched Soundcheck externally as a commercial plugin, along with other plugins in the bundle. We envision a future where users can effortlessly access and enhance software tech health, and that's where we aim to go.

[00:06:01] A quick TL;DR on the impact of Soundcheck. The first thing is a productivity boost: the more you pass, the more you deploy. As squads increase their track pass rate, deployments increase as well. The next one is efficiency gains: squads experience about a 20% decrease in master build failures and a 25% decrease in high-urgency incidents when maintaining a track pass rate of 85%. And the next one is a virtuous cycle: when high-urgency incidents do occur, squads release fixes more quickly. Squads average about three high-urgency incidents a month, which translates to about five hours saved monthly per squad.

[00:06:43] All right, it was important to walk you through the product I lead and the problem it's solving, in order to understand the insights and the takeaways I'll be sharing from my learnings. So let's dive right in.

Internal and external user needs were different

[00:06:55] In December 2022, like I mentioned, we launched Soundcheck externally, and this is what we learned from externalizing Soundcheck. Internal and external user needs were different. These users had conflicting requirements. For instance, Spotify's internal users were more advanced on the platform engineering maturity curve. They had been using Soundcheck for approximately two years, and required neither documentation nor guidance to comprehend its features. However, this contrasted with the onboarding experience of our external users, who were first-time users of the product.

[00:07:34] And resource allocation is important. Resources such as time, budget and manpower are limited, and prioritizing one group's needs over the other can lead to dissatisfaction and inefficiency. So meeting both internal and external needs was important to stay aligned. We realized that we needed to develop a strategy that helped with this goal. Ensuring that prioritization aligns with the organization's overall strategy and goals is extremely crucial. Any misalignment here can result in a product or service that fails to meet key objectives or market needs. And we all know that market conditions, user expectations and organizational needs are constantly evolving, and we need to adapt continuously and balance these needs as well.

[00:08:23] Sorry. So we learned that a strong product strategy and vision for both internal and external user needs was essential.

Engaging both user types: interviews, prototype reviews and beta testing

[00:08:32] All PMs are going to relate to this slide, because I'm sure we have all been in a situation where planning is complete. We have prioritized three internal user needs and probably three external user needs, and we are pretty happy that OKRs are defined and we are done with the planning phase. What we look forward to next is working with the team to implement what we've planned for. And then suddenly new priorities and user needs surface. This is something that we have faced at Spotify as well while developing Soundcheck, and I want to talk a little bit about our approach and what we did to solve for such challenges.

[00:09:15] The first thing we did was engage with both user types. We did user interviews and design prototype reviews. We conducted comprehensive user interviews with both internal and external users, to gather detailed insights into their needs, preferences and pain points. The next thing we did was organize design prototype review sessions with both user groups, to ensure that the product design met the diverse requirements and expectations of all our users, and we used the feedback from these sessions to iterate and refine the product design.

[00:09:48] The next thing we did was conduct beta testing sessions and actively seek feedback. Those sessions were run with both internal and external users, who interacted with the product in a real-world environment, and this encouraged beta testers to provide candid feedback about their experience, including any issues they encountered and suggestions for improvement. We then used this feedback to identify and address any potential problems before the full product launch.

Intense prioritization with ICE scores

[00:10:21] The next thing I want to talk about is how we implemented intense prioritization. We used a model called ICE scores for feature prioritization. ICE stands for impact, confidence and effort. It's a scoring model for evaluating potential features. When we talk about impact, we're talking about assessing the potential positive impact of a feature on the user experience and the overall product success. Confidence is about understanding the level of confidence in the estimated impact and the feasibility of the feature. And the next one is effort, determining the amount of effort, resources and time required to implement the feature.

[00:11:07] So we used ICE scores for feature prioritization, and then prioritized features that had high impact and high confidence scores but required reasonable effort. This ensured that the most valuable and feasible features were developed first, irrespective of the specific user needs that we were focusing on.

A unified product strategy: learning from internal users at scale

[00:11:25] What was also very important for us was to establish a unified product strategy to serve both user needs. We learned from internal users at scale. At Spotify scale we have internal users that we can learn from, and this helped us establish a product strategy that leveraged the insights and experiences of internal users before releasing the product to external users. We used the internal user base to also conduct extensive testing, we gathered feedback again, and we made the necessary adjustments to the product.

[00:12:02] This approach allows for refining and optimizing the product based on real-world usage data from internal users, and it also ensures a smoother and more successful external release. The feedback from the features that we tested internally and then released externally has been that they have provided a much larger impact. By implementing this strategy, we have effectively balanced the needs of both internal and external users, we've ensured that the product is aligned with the overall strategic vision, and we've delivered a high-quality user experience.

Takeaways

[00:12:43] Here are some suggested takeaways from this session. The first thing I want to talk about is building the platform as a product. This is specifically important because a user-centered focus, actively understanding and addressing the unique needs of both internal and external users, is extremely crucial for product success. We also want to gather ongoing user feedback through interviews, prototype evaluations and beta testing. This helps refine the product on an iterative basis.

[00:13:16] The next takeaway I would suggest is to establish effective feedback mechanisms. Actively seek and incorporate feedback from diverse user groups. Doing so helps identify potential issues early in the cycle. Scheduling structured feedback sessions and incorporating the feedback can help meet user expectations before a full release.

[00:13:42] As I mentioned, utilize the ICE model, which stands for impact, confidence and effort, for feature prioritization. This ensures that the most valuable and feasible features are developed first. This method also helps balance the effort and resources with the potential benefits of new features.

[00:14:00] Engage in an iterative development process. This is also extremely important, because you learn from internal users at scale, and this allows for extensive testing and optimization before external release. Iterative refinement based on real-world usage data helps in delivering a more polished and user-friendly product.

[00:14:21] The next one is something that's really close to my heart, which is to ensure team alignment with the strategic vision. While a strong product strategy that considers both internal and external user needs is essential, it is important to involve the team at every step in order to validate your hypotheses, because products are not built by individuals, they are built by teams. So clearly explain the long-term vision and strategy of the product to your team, and explain how it aligns with the broader company goals. Highlight the customer needs and use cases that the product strategy aims to address. Define measurable, specific OKRs that align with the strategic vision, and discuss with the team how these OKRs will guide the team's priorities and efforts.

[00:15:07] Explain the rationale behind the prioritization of features and initiatives, because we want teams to understand why we have prioritized a certain feature over another, rather than expecting them to go in blind and just develop based on the prioritization. It should not be top-down. It should be something that is discussed with the team in a collaborative manner. Also ensure that the team understands why certain tasks are prioritized, as I mentioned, and also how they can contribute to the strategic mission. Pitching sessions, where teams can contribute ideas that will achieve the strategic vision, are also something that can be used and leveraged.

[00:15:55] Also, encourage open and transparent communication channels with the team. Regularly update the team on progress towards strategic goals, and if the strategy changes for any reason, make sure that the team is updated. Always identify opportunities for collaboration, not just with the team that you're working with, but with other teams across the organization as well. That way you can learn how different products are built and how different user needs are evolving. Facilitate these collaborations to ensure alignment, and make sure that there's a shared understanding across teams. It's also important to establish a culture of open feedback and constructive criticism, and also be open to receiving feedback from the team and stakeholders.

[00:16:41] The last point I wanted to talk about is scalability and adaptability. Strategies that work well for internal users can provide valuable insights and scalable solutions for external users. So be flexible in adapting the product based on user feedback. This ensures continuous improvement and long-term product success.

Q&A

[00:17:04] Srividhya: All right, everyone, let's switch gears and play a quick round of stump the presenter. If you have any questions, or if there's something that you're curious about, here's the chance to ask.

[00:17:13] Host: Going through a few things: the big things that came out here in terms of teams were the alignment of internal and external priorities. That was a new concept, I believe, for a lot of people. Not necessarily a new concept, but it's not often that people are thinking in that way. In fact, even at UXDX, when we hear talks on prioritization, one of the biggest challenges is how you actually align all parties and all stakeholders. So some questions here around that are about beta testers. How did you source beta testers? If you can break it down into both internal beta testers and external beta testers.

[00:18:02] Srividhya: Sure, that's a great question. Internally it was more of a volunteering opportunity, where I would just go into these Slack channels and request beta testers, users that were already using Soundcheck. And I have seen that they are really interactive in a user Slack channel within the company itself. So once I requested, and explained the reason why I needed beta testers, there were a lot of volunteers who came up and said, "Hey, I'm interested, and I want to share feedback." So internally, I would say, it was easier than sourcing external adopters.

[00:18:44] Externally, we constantly work with adopters on a quarterly basis. We also talk to them on a weekly basis, and there's a customer success team that constantly interacts with the adopters. Since I have direct access to customers, I'm able to request sessions where I walk through a prototype design or something like that and request their feedback on it. This has been especially helpful in identifying what it is about a design that the users like, and what it is that they don't like, ahead of actually beginning the implementation. That saved a lot of time, I would say, in that case.

[00:19:30] Host: And what are your specific methods for gathering and prioritizing the feedback from the user groups?

[00:19:37] Srividhya: Yes, as I mentioned, the ICE model. We evaluate the impact of what that particular feature is going to do for our users, and it's very important to understand the data here, and the metrics. If a feature is going to help solve a problem for a user, and it's going to be low-hanging fruit, which is low effort for the team to get out, that's something that I would prioritize. Other than that, if it's a business-critical feature, and adopters or customers are not able to proceed further without a specific feature, that's also something that I would prioritize. And in general, apart from using an ICE scoring model, there are also case-by-case situations where we look at features and we understand the data about some of the features being really useful over others. That's how we use metrics: we use the data to take the decisions for us.

[00:20:37] Host: There are a few comments on ICE. Somebody's mentioned that it's great, they're going to even use it on a daily basis for their team, in terms of their own personal thinking. But specifically on the confidence part of that, there are questions around data. Is it your gut feel? Can you dive more into how you, at Spotify or yourself, define confidence in something?

[00:21:05] Srividhya: Sure. I think confidence is definitely part gut feel, but there's also data to support it. And I leverage the team here to help me out, so it's not just my score that's going to decide what the priority is. I also work with the team to understand what their thought processes are, and what their confidence is on a specific feature. From a PM perspective, for example, a feature might sound easy to implement, but it's the engineers who are going to understand or know how difficult the implementation is. So it's important to bring in all the voices in the room and talk to the team to understand the specific confidence. Effort is something engineers can prioritize easily, but confidence is something where we want both... on my team, my engineering manager, myself and the entire team vote and decide on the overall confidence rating, or the score, for a particular feature.

[00:22:05] Host: And have there ever been disagreements?

[00:22:09] Srividhya: Definitely. And it's always a constructive conversation that we have. If there are disagreements, we pick that up, we talk through it, and then there's always a perspective that someone can share which helps us lean towards one direction or the other.

[00:22:25] Host: Yeah, and it's a great leading question on the team. What is too many people on the team to be involved? How do you make that decision? Because you said you need to involve everybody, but how much is too much? Define everybody.

[00:22:42] Srividhya: At Spotify things are easy for us, because we have the squad model. We are small, focused teams. In my team we have one EM, one PM, and probably five to six engineers, so it's a small team. We don't do this in a synchronous manner. If we have ICE scoring that we need to do for a bunch of features, it's an async spreadsheet that all of us spend time on. Instead of sitting together and talking through the scores of each of these, we go and add our numbers there, the team adds theirs, and I go and review those numbers. Then we have a synchronous conversation, where we talk through the numbers and see which ones need to be discussed. So from that perspective, small, focused teams are really helpful for taking faster decisions, and that's something that I've seen in my experience as well.

[00:23:35] Host: Great, and great to hear and get an understanding of your squad approach, which is fantastic. What tools do you use that help you on this journey, or in your work on a day-to-day basis, maybe around internal users and teams?

[00:23:56] Srividhya: You mean for prioritization? Is the question focused on prioritization?

[00:24:01] Host: I'm not sure, but I would gauge that it's anything from down to Slack. Do you use things like user testing tools, and what tools do you use that better help you, even from a data perspective? Is there anything that you use on that front?

[00:24:17] Srividhya: Slack is definitely something that we use. For beta testing, it's more of a model where there's a folder and a spreadsheet. I don't know why there's an echo, sorry about that. We create a folder, and then all the beta testers go and add their inputs to it, and then we go and look at it. So it's more on a needs basis. There are some tools that we decide on: maybe a spreadsheet, maybe it's just a document where they go and add their thoughts. We're not too heavy on the tooling, because focusing too much on the tools might make us focus less on the actual data and the content. So we keep it really light when it comes to tooling.

[00:25:06] Host: Yeah, fantastic. And how much do you forward plan? How far in advance are you ahead of the game, in terms of your future thinking on your projects and what you're going to do next? A lot of people obviously do it... you talked about OKRs, and quarterly, maybe every two weeks. What does that look like for you as a team?

[00:25:29] Srividhya: At least a quarter. I'm already thinking about what features or what enhancements we need to do on the product side for the next quarter, so that's a minimum, I would say. And then there's also a six-month period that we think through, and we decide that the second half of the six-month period is probably not going to be high fidelity. It'll be low-fidelity planning. But the first half is something where we'll probably be really clear on what we want to do for the next quarter.