Behind Every AI Feature is an A/B Test
Talk
Behind Every AI Feature is an A/B Test
14 May, 9:55 am - 10:30 am (America/New_York) - Stage 2
14 May, 13:55 - 14:30 (UTC)
Continuous Design
UXDX USA 2025
Launching a product feature powered by a model requires one essential step: A/B testingIn this session, we’ll detail exactly what you need on the experimentation side to make sure your model is performing better, ideally, then the control. The challenge in experimentation isn’t running the tests themselves, but rather the nuances and details key to designing a well-structured experiment, enabling you to gain the right insights to evolve your product and inform your larger AI strategy. Discussion points include:
- Understanding model behavior offline before starting the online evaluation phase
- Verifying experiments before launch through canaries or spot-check tooling
- Basic design principles that are fundamental to launching an A/B test on the product
- Simple strategies we implemented for Spotify’s Home page to increase the quality of experiments launched on the product
Leemay Nassery, Engineering Leader,Spotify
