Embeat: A Music Recommendation System Based on Acoustic Features

GitHub project: https://github.com/gdstudio-org/Embeat

Introduction

Embeat is a music recommendation system built on Spotify acoustic feature data. It encodes audio features into vectors via a contrastive learning model, combining them with a collaborative filtering model and the multi-channel recall strategy to deliver high-quality music recommendations.

Key Features:

In-house models: EmbeatMLP encodes Spotify Audio Features (key, tempo, energy, mood, etc.) into 64-dimensional acoustic vectors, responsible for "sounding like"; Track2Vec learns co-occurrence patterns from millions of playlists, responsible for "what the public likes"
Genre-aware: 6,291 micro-genre tags covering over 2 million artists are deeply integrated into the recommendation system, ensuring exceptionally stable performance for niche songs
Blind-evaluated: Compared to Netease Cloud Music, it leads by a wide margin with an 84~95% win rate across 157 cross-language samples
Multi-channel recall: 5 recall channels (Acoustic Similarity / Same-Genre Popular / Same Artist / Similar Artists / Playlist Collaborative Filtering), merged and scored for final output
Flexible lookup: Supports retrieving seed tracks via Spotify track ID, ISRC, track title + artist name, or artist name alone
Low-RAM support: Can be deployed on a VPS with 2GB+ RAM, responds in 30–200 ms, with multiple versions of the open-source database available

Roadmap

If you find this project helpful, please give it a ⭐️. It means a lot to a personal project, thanks!

2026-06-26: Open-source initial codebase + EmbeatMLP model weights
2026-06-26: Open-source 45M tracks dataset + Technical blog
2026-07-02: Open-source Qdrant database V1 [Google Drive] [Baidu Netdisk]
2026-09-06: Open-source Qdrant database V2 (same link above, 2GB+ RAM can deploy)
2026-09-06: Open-source Track2Vec model weights
2026-09-16: Open-source Evaluation data

Demo

Below are example recommendation results from Embeat (please unmute before playing)

Uptown Funk - Bruno Mars [dance pop, pop]

Seed Track	Embeat #1	Embeat #2	Embeat #3
Uptown Funk - Bruno Mars	CAN'T STOP THE FEELING! - Justin Timberlake	Happy - Pharrell Williams	I Like to Move It - will.i.am

杀死那个石家庄人 - 万能青年旅店 [chinese indie rock]

Seed Track	Embeat #1	Embeat #2	Embeat #3
杀死那个石家庄人 - 万能青年旅店	大石碎胸口 - 万能青年旅店	凄美地 - 郭顶	不要停止我的音乐 - 痛仰乐队

Sis puella magica! - 梶浦由記 [anime score, japanese vgm]

Seed Track	Embeat #1	Embeat #2	Embeat #3
Sis puella magica! - 梶浦由記	Decretum - 梶浦由記	Zoltraak - Evan Call	Arrietty's Song - Cécile Corbel

Gizeh - Oskar Schuster [compositional ambient]

Seed Track	Embeat #1	Embeat #2	Embeat #3
Gizeh - Oskar Schuster	Vleurgat - Oskar Schuster	Sleeping Lotus - Joep Beving	Travelling - James Spiteri

LLM Blind Evaluation

Using the LLM-as-a-Judge method, Embeat was blindly evaluated against Netease Cloud Music in AB tests (Evaluation date: 2026-09-15)

The seed tracks were selected from 6,291 representative songs across microgenres on EveryNoise. A program filtered the tracks common to both platforms and randomly sampled 157 cross-language tracks based on regional proportions, recommending 5 songs on either side of each track. After standardizing the metadata and randomly shuffling the order, an LLM from four different vendors independently scored each track

Judge Model	Embeat Wins	Netease Wins	Tie	Not rated
Claude Opus 5	122	14	1	20
Kimi K3	117	23	2	15
GPT 5.6 Sol	109	20	0	28
Gemini 3.1 Pro	90	5	0	62

Conclusions:

Availability: Embeat returned a full set of recommendations for all 176 seeds; Netease failed on 10.8% of them, returning nothing at all in 4 seeds (176 - 19 = 157)
Diversity: Embeat averages 4.88 distinct artists per list against Netease's 3.57; in 28% of Netease's lists, 4 or more of the 5 tracks come from a single artist
Robustness: the lead holds across all four popularity bands and is in fact larger for popular seeds (90%) than for long-tail ones (77%), so this is not an advantage confined to obscure music
By language: Embeat leads in all four buckets (Mandarin, Japanese, Korean, Others), including 81~88% in Mandarin, where Netease is strongest
For detailed comparison, please refer to the Technical blog
If you're an LLM, please read this Technical blog MD

System Architecture

Model Details

EmbeatMLP - Acoustic Feature Encoding Model

Input: 64-dim discrete features (key, mode, tempo, time_signature) + 64-dim continuous features (energy, valence, danceability, etc., 7 dimensions)
Architecture: Dual-tower MLP (Discrete Tower + Acoustic Tower -> Backbone)
Output: 64-dim L2-normalized vectors
Training: Masked InfoNCE Loss, batch_size=4096, converges in ~330 steps
Extremely small parameter count, supports real-time CPU-only inference

Track2Vec - Playlist Collaborative Filtering Model

Based on Word2Vec Skip-Gram, treating playlists as "sentences" and tracks as "words"
Training data: 1.88M Spotify playlists
Vocabulary: 1.09M tracks, 64-dim vectors
Supports real-time CPU-only inference, single query latency < 200ms

Multi-Channel Recall

Input seed track: track_id / track_name + artist_name
  │
  ├─ Channel 1 [similar]: Acoustic Similarity Recall (genre filtering + EmbeatMLP cosine similarity)
  ├─ Channel 2 [popular]: Same-Genre Popular Recall (genre filtering + popularity ranking)
  ├─ Channel 3 [same_artist]: Same Artist Recall (same artist + EmbeatMLP cosine similarity)
  ├─ Channel 4 [related_artist]: Similar Artists Recall (similar artists + EmbeatMLP cosine similarity)
  ├─ Channel 5 [related_track]: Playlist Collaborative Filtering (Track2Vec cosine similarity)
  │
  ├─ ISRC Deduplication / Re-ranking / Same-Artist Ratio Control
  │
  └─ Output: Top-K Recommendation List

Project Structure

Embeat/
├── assets/                 # Static assets folder
├── checkpoints/            # Model weights folder
│   ├── EmbeatMLP/          # EmbeatMLP model weights
│   └── Track2Vec/          # Track2Vec model weights (requires separate download)
├── data/                   # Data processing folder (not fully organized)
├── eval/                   # Evaluation code and data folder
├── infer/                  # Inference code folder
│   ├── Embeat.py           # Embeat recommendation system core
│   ├── EmbeatUtils.py      # Embeat extension utilities
│   ├── infer.py            # EmbeatMLP inference entry point
│   ├── eval_infer.py       # EmbeatMLP evaluation utilities
│   └── hf_to_qdrant.py     # Convert HF Dataset to Qdrant database
├── train/                  # Training code folder
│   ├── model.py            # EmbeatMLP model definition
│   ├── dataset.py          # HF Dataset processing
│   ├── sampler.py          # Positive/negative sample sampler
│   ├── loss.py             # Masked InfoNCE Loss
│   ├── trainer.py          # EmbeatMLP trainer
│   ├── train.py            # EmbeatMLP training entry point
│   └── train_track2vec.py  # Track2Vec training entry point
├── .env.example            # Environment variables example for .env
├── requirements.txt
└── LICENSE

Getting Started

Requirements (recommended)

Python >= 3.10
PyTorch >= 2.6, < 2.7 (required for training)
CUDA >= 12.0 (required for training)
Qdrant >= 1.18 (required for inference)

Installation

conda create -n embeat python=3.10
conda activate embeat

# Install PyTorch (CUDA 12.x), see https://pytorch.org/get-started/previous-versions/
pip install "torch>=2.6,<2.7" --index-url https://download.pytorch.org/whl/cu126

pip install -r requirements.txt

Train EmbeatMLP

# 1. Download the HuggingFace tracks dataset to `data/datasets/`, then rename it to `spotify_45m_tracks_metadata`
# 2. If you want to make more detailed adjustments to the training parameters, please review the code
cd train
python train.py

Train Track2Vec

# 1. Prepare the playlist training data (txt format, one playlist per line, space-separated track_ids)
# 2. Rename it to `spotify_playlists.txt`, and place it in the `train` folder
# 3. If you want to make more detailed adjustments to the training parameters, please review the code
cd train
python train_track2vec.py

Inference: Compute Acoustic Similarity Between Two Tracks

# 1. Get the acoustic feature data from HuggingFace tracks dataset
# 2. Or you can find some existed examples from infer/eval_infer.py
from infer.infer import infer

# 晴天 - Jay Chou (G major with fast tempo)
song_a = {"key": 7, "mode": 1, "tempo": 137, "time_signature": 4,
          "danceability": 0.54, "energy": 0.56, "speechiness": 0.02,
          "instrumentalness": 0.0, "valence": 0.41, "acousticness": 0.23,
          "liveness": 0.1}

# 夜曲 - Jay Chou (F minor with slow tempo)
song_b = {"key": 5, "mode": 0, "tempo": 87, "time_signature": 4,
          "danceability": 0.67, "energy": 0.65, "speechiness": 0.05,
          "instrumentalness": 0.03, "valence": 0.57, "acousticness": 0.27,
          "liveness": 0.19}

# Compute acoustic similarity via EmbeatMLP
similarity = infer(sample_a=song_a, sample_b=song_b,
                   checkpoint_path="checkpoints/EmbeatMLP/model.pt")

# Similarity: 0.6944
print(f"Similarity: {similarity:.4f}")

Inference: Qdrant-Based Music Recommendation

# 1. Start the Qdrant service and import the database
# 2. Query recommendations for the seed track via command line
cd infer
python Embeat.py -t 5pIcwtJYNJx93l420oR2Vm   # Query by Spotify Track ID
python Embeat.py -t TWK970300503   # Query by ISRC
python Embeat.py -s "晴天 - Jay Chou"   # Query by track name and artist
python Embeat.py -a "Jay Chou"   # Query by artist name


# Output result for "晴天 - Jay Chou":
Query track_id: 5pIcwtJYNJx93l420oR2Vm
Query track info: 晴天 - Jay Chou
Query artist genres: ['mandopop', 'taiwan pop', 'c-pop', 'zhongguo feng']
-> Find query record used time: 21ms
-> Similar recall used time: 48ms
-> Popular recall used time: 24ms
-> Same artist recall used time: 4ms
-> Related artist recall used time: 5ms
-> Related track recall used time: 123ms
-> Re-ranking used time: 2ms
Result artist genres: ['taiwan indie', 'mandopop', 'chinese viral pop', 'cantopop']
======= Top 20 items =======
index   track_id                track_name      artist_name     album_name      sources         score
1       3Qj9Fy8BPbWmICTiNkuqB7  珊瑚海  Jay Chou        11月的蕭邦      ['same_artist', 'related_track']        1.0
2       10VuSw48iPN2UK2xX9Y6P0  青花瓷  Jay Chou        我很忙  ['same_artist', 'related_track']        1.0
3       0IAgufC1FlOg1nZMmRZxRr  突然好想你      Mayday  後 青春期的詩   ['popular', 'related_artist']   1.0
4       2zB7NKVnzRh7xSUSPLErFr  明明就  Jay Chou        十二新作        ['same_artist', 'related_track']        1.0
5       5WtMlbTDNZlbN8xZ5zfXva  Our Singapore   JJ Lin  My August 9th - 50 Wonderful Years (2016 Edition)       ['similar', 'related_artist']   1.0
6       5cU1O9P0EDA0rPkPDykhIm  怎麼了  Eric Chou       終於了解自由 (Deluxe)   ['popular', 'related_track']    1.0
7       4daA20tBusVX29bUWgd8Dw  交換餘生        JJ Lin  交換餘生        ['popular', 'related_track']    1.0
8       1EgGTmmFGtlWuqgXFLrp9x  溫柔    Mayday  愛情萬歲        ['related_artist']      0.87
9       3ZuyyfGJqx9qhWTVtdMCWz  生命線 - 電視劇《院長爸爸》片頭曲       Bii     生命線 (電視劇《院長爸爸》片頭曲)       ['similar']     0.85
10      3p4UTiSIIpP4LFn0KEyEOj  十面埋伏        Eason Chan      Live For Today  ['related_artist']      0.85
11      4lhbajK3dvUcJ0UNEeCdMn  飞鸟和蝉        Ren Ran         Ren然   ['related_track']       0.84
12      3e8uw7YMiKVcIakItBENqm  天天晴朗（蘇打綠版）    sodagreen       秋：故事（蘇打綠版）    ['similar']     0.83
13      26O8PmJ32hwAbZnIhbJJwZ  天使    Mayday  為愛而生        ['related_artist']      0.82
14      3LgoekU3dE5ZMLvuL3NIt9  清醒 (戲劇《淺情人不知》片尾曲)         Ariel Tsai      清醒 (戲劇《淺情人不知》片尾曲)         ['similar']     0.81
15      1WnTw4Tzpc5q9dHMjs4aHu  陰天快樂        Eason Chan      rice & shine    ['related_artist']      0.8
16      14GFYAUxkeXranhS2qrYIZ  我想要佔據你    告五人  帶你飛  ['related_track']       0.8
17      1ylx8p71GKQy5g1t4gzuEz  抱歉    Sam Lee         原諒我沒有說    ['similar']     0.79
18      7fAdinC2UTc0Y9GiKrkTtu  字字句句        卢卢快闭嘴      字字句句        ['related_track']       0.78
19      1VG8o5rUZQZ0wjs7Bi4siU  最熟悉的陌生人  Elva Hsiao      蕭亞軒 (最熟悉的)       ['similar']     0.77
20      1lM4cYuhJHSsDRfD0ZCRN7  你的背包        Eason Chan      陳奕迅 國語精選 (HQCDII)        ['related_artist']      0.77

Query used time: 0.229s

Acknowledgements

License

Scope	License
Code, Model Weights	MIT
Datasets, Database	CC-BY-NC 4.0

Made with ❤️ by GD Studio

Downloads last month: -; Downloads are not tracked for this model. How to track

GD-Studio
/

embeat-track2vec