Browser-based AI DAW
VibeCheck is what happens when you ask "Can a browser analyze music like a human?" and then spend 6 months proving the answer is "surprisingly yes".
🎵 The Challenge
Build a browser-based Digital Audio Workstation component that can:
1️⃣ Separate a full song into individual stems (vocals, drums, bass, other)
2️⃣ Analyze the emotional mood (happy, sad, energetic, chill)
3️⃣ Do all of this client-side without uploading files to a server
🧠 How It Works (The AI Magic)
- Audio loading: Web Audio API reads MP3/WAV files into audio buffers (decoded PCM samples)
- Stem separation: TensorFlow.js runs a pre-trained U-Net model (Spleeter architecture) directly in the browser
- Input: Spectrogram (time-frequency representation)
- Output: 4 separate waveforms (vocals, drums, bass, other)
- Processing time: ~30 seconds for a 3-minute song on a decent GPU
- Mood analysis: Extract audio features (tempo, pitch variance, spectral centroid) → Feed into classifier model
- High tempo + major key = "Energetic/Happy"
- Slow tempo + minor key = "Sad/Melancholic"
- Trained on 10k labeled songs from Free Music Archive
🎨 User Interface Highlights
- Waveform visualization: Canvas API renders multi-track timeline (like Ableton Live but in a browser)
- Playback controls: Play/pause, solo/mute stems, adjust volume per track
- Mood heatmap: Color-coded segments show emotional intensity over time (inspired by Spotify's audio analysis)
- Export stems: Download separated tracks as individual WAV files
⚡ Technical Deep Dive
| Challenge | Solution |
| Model size (150MB) | Quantized INT8 weights → 40MB, only 5% accuracy loss |
| GPU memory limits | Process 5-second chunks sequentially instead of full song at once |
| Latency on CPU | WebGL backend (2x faster than WASM on most devices) |
// Real code snippet: Running inference in the browser
const model = await tf.loadGraphModel('model/stem_separator.json');
const spectrogram = audioBufferToSpectrogram(audioBuffer);
const [vocals, drums, bass, other] = await model.predict(spectrogram);
🎯 Real-World Use Cases
- Karaoke creators: Remove vocals to generate backing tracks
- Music students: Isolate instrument parts to learn by ear
- DJs/remixers: Extract clean stems for mashups
- Podcast editors: Separate background music from speech
🚀 Performance benchmark: Processes a 4-minute song in 45 seconds on M1 MacBook (WebGPU acceleration). That's faster than some desktop DAWs!
Tech stack: TensorFlow.js, Web Audio API, Canvas, React, TypeScript, Vite (bundle size optimized with tree-shaking)
- StackJavaScript, Deep Learning, Web Audio API
- Sourcehttps://github.com/Shorya-agarwal/VibeCheck
