HARP

What is HARP?

HARP lets musicians use different AI models through one application, standalone or alongside their DAW. It loads each model’s controls for tasks such as separating vocals, transcribing audio to MIDI, or generating musical variations. Models run locally or in the cloud, and results return to HARP for playback, saving, or sending back to the DAW.

I joined the HARP team in July 2026, working under the supervision of Prof. Zhiyao Duan at the University of Rochester. My focus has been on deploying and validating music and audio AI models within the HARP framework.

Deployment & Validation

I deployed 29 music and audio AI models to HARP, using PyHARP to adapt the original research code and set up each model's inputs, outputs, and controls. I then hosted the models on Hugging Face Spaces through Gradio.

I also helped improve HARP's validation process. Some failures were caused by the tests themselves, rather than the models, such as audio inputs that were too short or a lack of support for text outputs. I submitted test configurations for six failing models and a fix for text-output validation. My custom validators, including those for MERIT and MuQ-MuLan, were merged into HARP.

While testing DDSP-Guitar, I noticed it was receiving piano MIDI instead of guitar MIDI. I raised the issue and suggested making the MIDI instrument and channel configurable, which the HARP team lead later implemented and merged.

My contributions on GitHub ↗

What I Learned

Working on HARP gave me hands-on experience with a wide range of music AI models, including source separation, MIDI generation, and speech synthesis. I really enjoyed exploring the original code and figuring out how to get each model running. What surprised me was that even when a model deployed successfully, it didn’t always work as expected. Sometimes the issue was with the validation tests rather than the model itself. For example, I found that DDSP-Guitar was being tested with a piano instrument setting instead of guitar. Working through issues like this helped me understand how these models work and what it takes to make research code usable.

I’m really grateful to Frank Cwitkowitz and Prof. Zhiyao Duan for their guidance and the opportunity to contribute to HARP. It was also my first time contributing to an open-source project, which made it especially meaningful to me.