GSoC 2021 (Part III): The Summary

The Google Summer of Code season just ended. My project was about implementing generative music models in Julia, especially the Music Transformer. There were a lot of things implemented during this period and it was a fantastic learning experience. Lets take a look at what was done.

List of Major Contributions during the Season

Packages

Pull Requests

A Short Description of the Contributions

MIDI.jl

Link to the PR: https://github.com/JuliaMusic/MIDI.jl/pull/142.

This was the first contribution made as part of GSoC 2021. It added type support for the events in the MIDI spec and functions to encode and decode these events to and from their binary representations. I learnt a great deal while working on this PR as it was also my first foray into metaprogramming in Julia.

NoteSequences.jl

A NoteSequence is an abstract representation of a musical sequence. The package NoteSequences.jl has utility functions for creating NoteSequences from MIDI files, manipulating them, converting them to various representations like the Melody or Performance representation, and also to model inputs (one-hot vectors). The package also has functions for exporting NoteSequences to midi or audio.

PerformanceRNN.jl

A Julia-based implementation of the PerformanceRNN. These were the pre-trained models added.

  • PerformanceRNN - An LSTM that does not encode music note velocities but can generate music with expressive timing.

  • PerformanceRNN with dynamics - An LSTM that is aware of note velocities and can generate music with both expressive timing and dynamics.

MusicTransformer.jl

A Julia-based implementation of the MusicTransformer. These were the pre-trained models added.

  • UnconditionalMusicTransformer - A piano performance language model trained on over 10,000 hours of performances obtained from the transcriptions of piano recordings on YouTube.

  • MelodyConditionedMusicTransformer - A melody-conditioned piano performance language model. This model can be conditioned with a monophonic melody input to generate a polyphonic performance (it generates an accompaniment for the given melody).

The work also involved implementing the MultiheadRelativeAttention layer and TransformerRelative blocks for adding support for the relative-positional attention mechanism in Transformer networks.

MusicModels.jl

MusicModels.jl is planned to be the main repository for music datasets and generative music models in Julia. Both the MusicTransformer and PerformanceRNN models were moved over to MusicModels.jl, and additionally a few music datasets were also added - MAESTRO (raw), MAESTRO LM (processed), the Lakh MIDI Dataset (raw).

Feel free to take a look at the previous blogs for a more detailed description of the packages, why they were made and how to use them to generate music!

Conclusion

The benefits of this project is that it sets the stage for and kick starts further music related projects and research in the Julia language. My GSoC'21 journey may have ended, but it is not the end of my journey with JuliaMusic (and Julia). There were a lot of opportunities to contribute to the ecosystem before, and now that there are more tools in the arsenal of JuliaMusic, there are even more ideas to continue working on.