PRESSED / PLAY : autoencoder music recommendations¶

---

There's this iconic YouTube Video that mixes the sounds of Frank Oceans's Blonde with the collective visual art of Hayao Miyazaki and Katsuhiro Otomo's Akira. The result is a stunning collage of emotionsal sensation that really bridges the gap between sight and sound. There's one particular section that is simply sublime: a little 5 second snippet of a glider flying off into a pink and neon blue sunset sky while Pink + White plays in the background; a moment of perfect synergy between sight and sound. It's a particularly well curated example of something I feel that we all experience on a regular basis: that sensation of the song in your ears being perfectly matched to the vision hitting your eyes, be it a vibrant sunset or the neon colors of downtown at night, the musical and the visual enhancing each other in perfect ouroboros.

PRESSED / PLAY is an exploration of this cross-modal connection, leveraging the power of deep learning and the vast musical landscape thar Spotify offers. At the heart of this project is the challenge: given an image, can we deliver a playlist or a set of song selections to match the vibe of the image? Such a system isn't just tech innovation, but also a novel way for us to discover music, turning everyday visual experiences into personalized musical ones. As we delve deeper, we'll uncover the steps and considerations that went into bringing this idea to fruition. From data collection to the intricacies of model selection and finally integrating with Spotify's recommendation engine, this journey is as much about the technicalities as it is about the art of merging two sensory worlds.

---

the problem statement - translating art¶

---

"how can we capture the essence of an image and translate it into a musical experience?"

this problem is multifaceted:

1. cross-modal translation: At its core, this is a cross-modal problem where the challenge lies in bridging two vastly different types of data – visual images and musical genres or attributes.

2. subjectivity of interpretation: The interpretation of images is highly subjective. The same image might evoke feelings of joy for one person and nostalgia for another. The music that resonates with these emotions could differ drastically.

3. integration with existing systems: Once we've predicted a mood, how do we translate that into a tangible outcome for the user? In our case, this means creating a playlist of songs that aligns with the predicted mood and vibe.

4. scalability and generalization: The solution needs to be scalable, catering to a wide array of images and diverse user tastes. It shouldn't be limited by the dataset's initial constraints.

my aim is a system that can accept any image as input and, by understanding its content and emotion, produce a curated playlist that musically resonates with the image; being not only a technical challenge but also an artistic one, opening up new avenues for personalized content discovery and a richer multimedia experience.

---

the stack¶

---

---

data - collection and creation¶

---

before we could even get started determining what model and architecture on which to base our system, we face 2 main underlying issues: how exactly do we build our training data, and how do we use that to generate a prediction?

the basic inference blueprint¶

Luckily, the Spotify API has our solution, coming with an inbuilt endpoint, accessible via spotipy through sp.recommendations, which accepts as input, up to 5 seeding genres, and a list of target attributes with which to generate the predictions. The attributes which I've chose to work with are:

which are all relatively self-explanatory, aside from valence, which is essentially just a measure of how "happy" a song is, in simplest terms.

let's see this process in action! continuing to use Pink + White as our example, lets use the sp.recommendations endpoint to find some songs that are similar

and thats it! the recommendation algortihm by itself is easy enough to use, but the problem is how we retrieve the target audio features from an image

the data problem¶

We have a problem. Unlike other kinds of sandbox data science projects, there are no publicly available out-of-the-box datasets we could use to drive our model. We need an extensively labelled collection of images with audio features derived from songs, but that's not something that currenty exists on the internet. So let's make one.

imported are the functions that I built to accomplish, with names that are exactly as they say on the tin

• fetch_covers gets all the playlist ids and cover images of a particular user; which we by default set to 'spotify'
• fetch_track_ids gets all the track ids in a particular playlist, which we use to get all the trackids for the dataset
• fetch_tracklist_data: gets all the audio features we want from a track, applied over all the tracks we've obtained

As seen, the returned objects are a list of URLs linking to cover images and a list (of lists) of audio features. We have 2 main possible approaches from here: we could take the average of the audio features of each list, giving us a one to (many) relationship from picture to audio features, or we could train the same image to each song individually.

The second approach would allow us to let the model fit to a more diverse set of observations, which would allow it to generalize better, but we would then be training the model to associate the same image with a number of different audio features. To solve this problem, we can apply a set of random transformations to each image, and thus essentially ensure that every song is associated with a unique image, introducing diversity to the dataset and allowing us to bootstrap outselves out of what otherwise would have been a 200 item training dataset.

And here, I opt to use dask, since stitching together the data is an extremely computationaly intensive operation and dask helps to speed it up by an order of magnitude.

And there it is! We've turned a 200 item dataset into over 14000 items. Now that we have our data, we can now work on the model that will bridge the gap between images and the audio features.

---

the model - intution and process¶

---

VAE with attention: the intution¶

Variational Autoencoders (VAEs) and attention mechanisms are separately very poewrful in various applications. In our context, this combination architecture is especially robust for cross-model inference: let's explore why

1. data compression: At the core of VAEs is the capability to compress complex data into a concise latent representation. This compression helps in capturing the essence of data like images or songs without much loss of information.
2. generative tendencies: VAEs not only encode data but can also generate new, similar data. This generative ability aids tasks like recommendation.
3. attention's flexibility: Attention mechanisms allow models to focus variably on different parts of the input. This dynamic focus means that the model can weigh parts of an image or song differently based on their relevance.
4. interpretability: One of the allures of attention mechanisms is the interpretability it brings. It offers insights into what regions of input data the model deems important.
5. cross-modal translation: Combining VAEs and attention provides a robust framework for translating information across different modalities, like images to music genres.

layer-by-layer deep dive¶

1. encoder¶

The encoder's job in a VAE is to take input data and produce a latent representation. This latent space captures the essential features of the data.

Given input $ x $, the encoder outputs two vectors: mean $ \mu $ and variance $ \sigma^2 $. These vectors define a Gaussian distribution from which we can sample latent variables.

$$ \mu, \log \sigma^2 = \text{Encoder}(x) $$

2. sampling¶

Using the mean and variance from the encoder, the VAE samples a point in the latent space. This sampling introduces the stochastic element of the VAE.

We sample $ z $ from the Gaussian distribution:

$$ z = \mu + \sigma \odot \epsilon $$

where $ \epsilon $ is a random normal variable, and $ \odot $ denotes element-wise multiplication.

3. decoder¶

The decoder takes the sampled latent variable $ z $ and attempts to reconstruct the original input: $ \hat{x} $, the reconstructed input from the decoder.

$$ \hat{x} = \text{Decoder}(z) $$

4. attention mechanism¶

Embedded within the VAE architecture, the attention mechanism allows the model to focus on specific parts of the input when constructing the latent representation. Given an input sequence $ x_1, x_2, ... x_n $, the attention scores for each element are computed as:

$$ \alpha_i = \frac{\exp(\text{score}(x_i, z))}{\sum_{j=1}^{n} \exp(\text{score}(x_j, z))} $$

The final context vector, which is a weighted sum of the input elements, is:

$$ c = \sum_{i=1}^{n} \alpha_i x_i $$

5. loss function¶

The VAE has a unique loss function comprising two parts: reconstruction loss and KL divergence.

• reconstruction loss: measures the difference between the original input and its reconstruction.

$$ \mathcal{L}_{recon} = \| x - \hat{x} \|^2 $$

• kl divergence: measures the difference between the encoded distribution and a standard normal distribution. it acts as a regularizer.

$$ \mathcal{L}_{KL} = -0.5 \sum_{i=1}^{n} (1 + \log \sigma^2 - \mu^2 - \sigma^2) $$

The total loss is the sum of these two:

$$ \mathcal{L} = \mathcal{L}_{recon} + \mathcal{L}_{KL} $$

in practice¶

implementing the layers and mechanisms mentioned in PyTorch

Now to train this model, I didn't do it locally on my own machine, since my resulting dataset was pretty big (76000) rows and the training would've been much more efficient on a GPU, which I don't have. Sadge. To get around this, I uploaded the code to and ran the training job on AWS Sagemaker, which allowed me to run the training in the cloud on GPU, also allowing me to integrate S3 into how I served the model in the end.

here's the script for how I engineered that:

Once the training finishes successfully, the model is deployed as such:

---

genre-lization - the last puzzle pieces¶

---

So we've collected our data, trained a model, and are now ready to make our predictions right? We have one last step. Remember how we needed to supply the target features (which we can predict with our model now, wow!) and also up to 5 genres?

Getting all of a user's genres ia relatively trivial. I just take all of their top artists, and aggregate the genres of those artists:

You might immediately notice some things here that stand out. What are genre mappings? Why would a genre need to be mapped?

A caveat of using Spotify's sp.recommendations framework is Spotify only allows seeding for a very small and narrow subet of genres:

But the genres for our artists aren't limited to that. In fact, Every Noise at Once has amassed a staggering collection of every genre that has been encountered in the Spotify ecosystem. Herein laid a new problem: how do I map out all of these over 5000 genres to their closest seedable genre?

I tried a number of methods, including word similarity metrics, Word2Vec / FastText similarity comparison, but in the end, none of them contain the outside contextual knowledge to properly perform a mapping like:

As impressive as FastText is, unfortunately it can't do translation on the fly.

So I used ChatGPT. And so the genremappings.json file was created, allowing us to properly best map our user's top genres to genres that are seedable by Spotify.

---

putting it all together¶

---

At the end of it all, here's the final code that's run in my Django app when a user uploads an image:

an image is uploaded, an inference is made, a playlist is created, and we have a vibe. nice!

---

user feedback¶

---

a crucial part of any subjective art technology is user feedback. to do so, I implemented a cute little "like" / "dislike" feedback system on the site that gives either a feedback value of -1 or 1, which we can integrate into our training by switching up the loss function:

original Loss:

$$ \text{loss} = \text{reconstruction loss} + \text{kl divergence} $$

adjusted Loss with RAML:

$$ \text{loss} = \text{reconstruction loss} - \lambda \times \text{reward} + \text{kl divergence} $$

here, $\lambda$ is a hyperparameter that controls the influence of the reward in the loss. The reward acts as a regularizer, encouraging the VAE to generate samples that are more likely to be liked by users.

in code, it looks something like

every time a user submits feedback, its uploaded to a DynamoDB database, where whenever I want to retrain the model, the data is downloaded and sticthed into a parquet, which is then used to retrain the model with the feedback data

and the training is done as such:

---

final remarks¶

---

And that's it! all the steps and intuitions behind this explained. Pretty cool, right?

And if you want to get hands on with my toy, you can visit it here at pressedplay.rdszhao.com. You just need a Spotify account. If you're interested, email me at rdszhao@gmail.com to try it out. Thanks for reading, and cheers! All the code can be found here