Spotify’s new ‘DJ’ feature is the first step into the streamer’s AI-powered future

Image credit: Spotify

Spotify has bigger plans for the tech behind its new AI DJ feature after seeing positive consumer reaction to the new feature. AI DJ, which launched just before the company’s Stream On event in LA last week, combines personalized music selections with audio commentary provided by his AI-generated voice that sounds lifelike. ration. But under the hood, this feature leverages the latest AI technology, large language models, and generated speech. All of this builds on Spotify’s existing investments in personalization and machine learning.

Spotify believes these new tools don’t necessarily have to be limited to one feature. As such, we are currently experimenting with other applications of this technology.

While the highlight of Spotify’s Stream On event was a revamp of its mobile app focused on TikTok-like discovery feeds of music, podcasts and audiobooks, AI DJs will become an integral part of the streaming service’s new experience. I was. Introduced to Spotify’s Premium subscribers in the US and Canada in late February, the DJ is designed to get to know you so you can play whatever you want to hear at the touch of a button.

A revamp of the app now features DJs at the top of the screen below the music sub-feed for subscribers, serving both as a way to stream their favorite music and as a way to encourage free users to upgrade.

To create the commentary that accompanies the music DJs stream, Spotify says it tapped into the knowledge base and insights of its in-house music experts. You can expand the description to fit the user. And unlike his ChatGPT, which tries to craft an answer by extracting information found on the wide web, Spotify’s more limited database of musical knowledge makes the DJ’s commentary relevant and accurate.

The actual music selections selected by DJs are based on our existing understanding of user tastes and interests and reflect what was previously programmed into personalized playlists such as Discover Weekly.

AI DJ’s voice, meanwhile, was created using technology Spotify acquired from Sonatic last year, and is based on the voice of Xavier “X” Jernigan, Spotify’s head of cultural partnerships. Surprisingly, the voices sound incredibly realistic and not at all robotic. “I can hear myself all day long,” he joked.)

“The reason it sounds so good is actually the purpose of the team we acquired, Sonatic Technology. Sultan explained in a conversation with TechCrunch after Stream On ended. “When you listen to AI DJ, you know where the poses are for breathing. You hear different intonations. You hear the excitement of certain genres,” he says.

Of course, natural-sounding AI voices are nothing new. Years ago, Google stunned the world by creating its own human-sounding AI. However, its implementation within Duplex has led to criticism. As the AI ​​dialed into the business on behalf of the end user, it was initially not made clear that it was not a real person. There shouldn’t be similar concerns about Spotify’s feature, which is even called “AI DJ”.

To make Spotify’s AI voices sound natural, Jernigan went into the studio and worked with voice technology experts to create high-quality voice recordings. There, he was instructed to read different lines using different emotions, which were fed to his AI model. Spotify wouldn’t disclose how long or in detail this process would take, noting that the technology is evolving and calling it a “secret sauce.”

“From a high-quality input with many different permutations, [Jernigan] After that, you don’t have to say anything. Now it’s purely AI-generated voice,” he says of the generated voice, Sultan. Still, Jernigan occasionally pops into his room at Spotify’s writers to provide feedback on how he read a line, ensuring continued input.

Image credit: Spotify screenshot

But while AI DJ is built using a combination of Sonantic and OpenAI technologies, Spotify is also investing in internal research to better understand modern AI and large language models.

“We have a research team working on state-of-the-art language models,” Sultan told TechCrunch. In fact, there are hundreds of people working on personalization and machine learning. For AI DJ, Sultan said the team is using his OpenAI model. “But generally there are large research teams understanding all the possibilities across large language models, generated speech and personalization. . “We want to be known for our AI expertise.”

However, Spotify may or may not use its own in-house AI technology to power future developments. You may decide that it makes more sense to work with a partner, as we do now with OpenAI. But it’s too early to say.

“We publish papers all the time,” says Sultan. “We invest in the latest technology. As you can imagine, in this industry he LLM is that kind of technology. So we develop our expertise.”

This underlying technology allows Spotify to expand into other areas including AI, LLM and generative AI technology. The company has yet to reveal what these areas will look like for consumer products. (I hear that chatbots like ChatGPT are one option being experimented with, but this is one of many other experiments, so nothing has been decided as to launch).

“We haven’t announced any exact plans for when we’ll be expanding into new markets, new languages, etc. But it’s the platform, the technology. We can do that, and we want to share more as it evolves.” I’m thinking,’ says Sultan.

Early consumer feedback on AI is promising, Spotify says

The company didn’t want to develop a full suite of AI products because it didn’t know how consumers would react to DJing. Do people want AI DJs? Will they be involved in features? None were clear. After all, Spotify’s voice assistant (“Hey Spotify”) was discontinued due to lack of adoption.

But there were early signs that the DJ feature could work well.Before launch, Spotify tested the product internally with employees, and usage and re-engagement metrics were “very good.” It was good for

Sultan said the general adoption so far is consistent with what Spotify has seen internally. This means we may spin up future products using the same foundation.

“People are spending hours on this product every day. It helps them make choices, discoveries, tells them what music they should listen to next, and explains why. And you’ll find that it’s an emotional thing,” Sultan said.

Additionally, Spotify found that on the days people listened to music, they listened to DJs 25% of the time, and more than half of those who listened for the first time used the feature again the next day. clarified. However, this feature is not yet 100% rolled out in the US and Canada, so these metrics are in early stages. However, the company believes they are promising.

“I think this is one great step in building a really valuable product-user relationship,” says Sultan. But he warns that the challenge going forward will be “finding the right application and building it right.”

“In this case, we said this is an AI DJ for music. But it’s definitely fun to dream about what else you can do and how fast you can do it, ”he adds.