I turned Nintendo Switch Joy-Con controllers into musical instruments using HD Rumble
Hi everyone!
I'm 13 years old and I've been working on an open-source project called Joy-Con MIDI Player.
The goal is to turn Nintendo Switch Joy-Con controllers into real musical instruments by using their HD Rumble motors to reproduce musical notes as vibrations.
The project is written in Rust and currently includes two applications:
🎼 joycon-midi-player.exe
* Play MIDI files
* Select which tracks to use
* Manually assign each track to any connected Joy-Con
* Support multiple Joy-Con controllers simultaneously
🎹 joycon-live-midi.exe
* Connect a MIDI keyboard or a virtual MIDI port (such as loopMIDI)
* Perform in real time
* Instantly convert MIDI notes into HD Rumble vibrations
How it works:
* MIDI notes are converted into frequencies
* Frequencies are encoded into HD Rumble packets
* Packets are sent to Joy-Con controllers through BetterJoy
* The motors reproduce musical pitches as vibrations
Current limitations:
* One Joy-Con can only play one note at a time (no true polyphony on a single controller)
* Some notes, especially in live piano mode, can produce loud mechanical rattling noises depending on the frequency
* For the best experience, it is recommended to place the Joy-Con on a soft surface (such as a mouse pad) or hold it firmly in your hand
* Some very low bass notes are barely perceptible and may be difficult to feel
* Very high frequencies may feel similar to each other
* A small amount of Bluetooth latency may occur depending on your adapter and drivers
* In most cases, the latency is very small and does not significantly affect the experience
Planned improvements:
* Better polyphony handling
* Smarter note prioritization
* Cross-platform support
* Direct Bluetooth communication without BetterJoy
GitHub:
https://github.com/AkatsudoYTB/joycon-midi-player
Download:
https://github.com/AkatsudoYTB/joycon-midi-player/releases
I hope this project can inspire other young developers. If a 13-year-old can build something like this, anyone can learn to code amazing things.
Feedback, ideas, and suggestions are very welcome!