“LnH: The Band” is a hobby project of Amir (@amirsaffari) and Aida (@aidamash) currently residing in a tiny black computer box in their apartment.
Why music as a generative/creative bot?
We have been working on generative algorithms as fun projects for a while. Although there are many areas one can explore, such as text or images; we think music and sound are the most challenging of them. Our eyes and brains overlook the visual noise a lot easier. They compensate for what is missing or wrong. However, our ears are not as forgiving about music or sound. We generally notice the slightest of errors in a musical piece. As a result, achieving a good quality is generally much harder.
Also it is fun and you get immediate feedback to changes you make!
How does it work?
It’s basically an Artificial Intelligence based Twitter bot: it parses the tweets that are sent to @lnh_ai following the format explained here. It then finds what is called a seed track using NLP and the title you provide. LnH has deep learning based models for each genre and uses these models and a small sample from the beginning of the seed track to compose a song.
What are these models?
Each model is what we call a “Deep Dynamic Graphical Model”. It uses deep recurrent neural networks with a specific architecture that allows them to model complex musical notes. A model is for an entire band, meaning they are capable of modelling multiple polyphonic instruments and various aspects of musical notes such as pitch and duration. So in a nutshell a single model is responsible for composing songs for an entire genre.
How do you train them?
The training data for these models are all Midi files gathered automatically from the Internet. We have not cleaned up these files (as there’re many of them), so there’re many songs that are either incomplete or in wrong genre.
How do you compose?
Once a model is trained, you can recursively generate musical notes by predicting the most likely output distribution based upon a recent history. Then you can sample a new note from this distribution. We remove the initial seed segment (and a bit more) from the beginning before finalising the composition.
How much music theory LnH uses?
Not much actually. The models themselves are very generic and you can use them for other sequence modelling tasks with structured outputs (later on that). We haven’t done any modifications to the original Midi data other than converting them to numerical values so that we can feed them to these models.
How fast is the training and composition?
We use about 200 songs from a genre to train a model. On a small PC with a NVIDIA 1070 GPU, it takes about 6-12 hours to train it using a Keras+Tensorflow backend. Composition is quicker, on average composing a song with about 4000 musical notes takes about 5 min.
What is the output format?
The composition leads to a Midi file which then gets rendered to audio using Timidity++ and sound-fonts. This gets further compressed to an OGG format.
How is it different than X?
Generally speaking these generative models for music roughly follow similar approaches and use similar concepts. However, not many of these models are capable of modelling multiple polyphonic instruments (i.e. polyphonic multi-voice) as well as structured nature of musical notes. Also our approach is totally data driven as we don’t use music theory as a prior for feature extraction or during composition.
This is an exciting and fast moving field and there are new developments almost everyday (and there are also established companies dedicated to this field such as JukeDeck), so it is certain that someone has done/or soon will do better than what we have here.
Where do you host LnH?
It is currently hosted on our PC at home.
I tweeted at LnH, but did not hear anything back?
Since it is hosted at home, depending on our broadband connectivity, we might have outages. Also from time to time we upgrade the models, so the bots will be nonoperational during that time. Additionally, if the bot couldn’t parse the tweet into a coherent instruction, it will not attempt any composition.
Can I download and use a song composed by LnH?
By all means! The songs themselves are distributed under Creative Commons Attribution 4.0 International (CC BY 4.0). We upload each song to clyp.it and if you add “.mp3” or “.ogg” to the end the link, you will have access to the audio files. If you like a song, let’s know on Twitter. We would love to know your thoughts.