How the voice and algorithm works

“What’s wrong? I’m X. From this moment on, I’ll be your very own AI DJ on Spotify. Let’s go.” —Greetings from Spotify’s new AI DJ

Debuting on February 22nd, the feature combines years of Spotify’s machine learning and user listening data (e.g. songs you played repeatedly in 2018, new genres you’ve been into lately) and Sonantic’s generative AI. supported by A voice AI company that Spotify acquired for his $95 million last summer.

Between AI DJ’s release and the beginning of March, we saw quite a few repeat customers. “On days when a user is tuning in, a fan spends 25% of his listening time he DJing,” he said in a Spotify blog post. The post goes on to say that more than half of those who listened for the first time used the DJ again the next day.

Spotify has been leaning towards personalization technology for years, but in the past six months since its acquisition of Sonantik, a piece of the generative AI puzzle has become possible, which it says has made this feature now working. said Zaid Sultan, Spotify’s vice president of personalization.

“Since then, we’ve had that special piece of magic.”

Spotify’s team of AI DJs consisted of more than 10 people, Sultan said, though Spotify declined to reveal an exact number. Writer’s room of editors, data curators and screenwriters. Sonantic’s own voice AI team.

The AI ​​DJ voice itself is modeled after Xavier “X” Jernigan, a full-time Spotify employee and head of cultural partnerships at the company. Def Jam and his Jernigan, who has music industry experience at Sony Music and Sean “Diddy” Combs’ personal assistant as his assistant, is the voice of the feature after appearing as his podcast host on Spotify. Selected as a model.

” [machine learning] The models we use are from the Sonantic acquisition,” Sultan says, adding, “What they focus on is the quality of the output, especially the emotion. The great thing about this process is… is being able to work closely with X in the studio so that the input of is mapped to emotions in very high quality, so you know when X tends to breathe. [and] What does he sound like when he is excited? ”

Jernigan carried a notebook with him all day long, jotting down phrases he often used to greet people and refer to music. song. Based on this kind of information and Jernigan’s inflections, we trained a model to develop an AI-generated version of his voice.

Sultan says the result the company didn’t expect: Sonantic’s voice AI is working that too good.

“Once I tried [doing] User research on DJ’s early prototypes shows that the voice is so real that if you don’t know what it is, you wouldn’t think it was AI. A person who has recorded many things—[this is] It’s a radio, or a station…a product I’ve already seen,” Sultan said.

To communicate the level of personalization, Sultan and his team had to try to find a way to “onboard” users. The team conducted user research to determine the length of his intro snippets, language, and more.

“Unless you understand that this is AI just for you, you cannot understand how personalized it is. [Spotify] It has been wrapped for 2 years. You don’t realize that it can give you the artists you love.