TL;DR We summarize the findings in our recent paper, Schultzberg, Ankargren, and Frånberg (2024), where we explain how Spotify’s decision-making [...] The post Risk-Aware Product Decisions in A/B Tests with Multiple Metrics appeared first on Spotify Engineering ..read more

For the past decade, Spotify has used approximate nearest-neighbor search technology to power our personalization, recommendation, and search systems. These technologies allow engineers and researchers to build systems that recommend similar items (like similar tracks, artists, or albums) without needing to run slow and expensive machine learning algorithms in real time. Spotify led the pack [...] The post Introducing Voyager: Spotify’s New Nearest-Neighbor Search Library appeared first on Spotify Engineering ..read more

When we want to determine the causal effect of a product or business change at Spotify, A/B testing is the gold standard. However, in some cases, it’s not possible to run A/B tests. For example, when the intervention is an exogenous shock we can’t control, such as the COVID pandemic. Or when using experimental control [...] The post How to Accurately Test Significance with Difference in Difference Models appeared first on Spotify Engineering ..read more

As companies mature, it’s easy to believe that the core experience and most user needs have been resolved, and all that’s left to work toward are the marginal benefits, the cherries on top. Cherries on top might add delight and panache, but they rarely cause fundamental shifts in performance and success. And as a business, [...] The post Experimentation at Spotify: Three Lessons for Maximizing Impact in Innovation appeared first on Spotify Engineering ..read more

In Part 1 of this series, we introduced the within-unit peeking problem that we call the “peeking problem 2.0”. We showed that moving from single to multiple observations per unit in analyses of experiments introduces new challenges and pitfalls with regards to sequential testing. We discussed the importance of being clear about the distinctions between [...] The post Bringing Sequential Testing to Experiments with Longitudinal Data (Part 2): Sequential Testing appeared first on Spotify Engineering ..read more

Spotify’s approach to challenges in sequential testing with longitudinal data  At Spotify, we’re constantly improving our data infrastructure, which means we can get feedback on experiments earlier and earlier. To allow for early feedback in a risk-managed manner, we use sequential tests to monitor regressions in the experiments. However, when moving toward smaller and smaller [...] The post Bringing Sequential Testing to Experiments with Longitudinal Data (Part 1): The Peeking Problem 2.0 appeared first on Spotify Engineering ..read more

Introduction In the fast-paced world of streaming, personalization plays a vital role in enhancing user experiences. At Spotify, our Home serves as a personalized surface where users retrieve familiar content and discover new content tailored to their preferences. We are constantly trying to optimize the Home experience to provide more value to our users; however, [...] The post Experimenting at Scale, the Spotify Home Way appeared first on Spotify Engineering ..read more

TL;DR Understanding algorithmic impact is critical to building a platform that serves hundreds of millions of listeners and creators every day. Our approach includes a combination of centralized and distributed efforts, which drives adoption of best practices across the entire organization — from researchers and data scientists to the engineer pushing the code. How we [...] The post Lessons Learned from Algorithmic Impact Assessments in Practice appeared first on Spotify Engineering ..read more

We did a couple of new things in Wrapped this year, and one of these is a thing called Your Listening Personality. You got one. There are 16 of them, in total: It’s not a lottery or a horoscope, though. As the four-letter codes at the bottom of each one hint, each Listening Personality is [...] The post <strong>What’s a “Listening Personality”?</strong> appeared first on Spotify Engineering ..read more

At Spotify, we’ve tried to uphold the tradition of running one major holiday marketing campaign at the end of each year. The campaigns typically run on various platforms such as the Spotify Homepage and the Premium page.  More recently, we wanted to run more campaigns throughout the year, perhaps during different holidays like Halloween or [...] The post What it Took to Scale Spotify’s Marketing Efforts appeared first on Spotify Engineering ..read more