August 09, 2018 / by / In music
Episode 1: Human-AI Collaborated Music
In the honor of Marvin Minsky’s birthday, we pre-launch How to Generate (Almost) Anything series with a (posthumous) duet between AI and Marvin!
Morgan and Pinar with Marvin's piano at MIT Media Lab
As some of you might already know, in addition to being a founder of Artificial Intelligence (and a founding member of MIT Media Lab, and CSAIL), Marvin was a life-long pianist and an amazing improviser. Here is an excerpt of Marvin’s opinions on music:
Why do we like music? Our culture immerses us in it for hours each day, and everyone knows how it touches our emotions, but few think of how music touches other kinds of thought. It is astonishing how little curiosity we have about so pervasive an “environ-mental” influence. What might we discover if we were to study musical thinking? Have we the tools for such work? Years ago, when science still feared meaning, the new field of research called 'Artificial Intelligence' started to supply new ideas about “representation of knowledge” that I'll use here. Are such ideas too alien for anything so subjective and irrational, aesthetic, and emotional as music? Not at all. I think the problems are the same and those distinctions wrongly drawn: only the surface of Reason is rational. I don't mean that understanding emotion is easy, only that understanding reason is probably harder. Our culture has a universal myth in which we see emotion as more complex and obscure than intellect. Indeed, emotion might be “deeper” in some sense of prior evolution, but this need not make it harder to understand; in fact, I think today we actually know much more about emotion than about reason.
We strongly recommend everyone to visit Music, Mind and Meaning to read more about Marvin and his music.
With his great influence on music, AI and everything in between, we wondered: can we make an AI collaborate with Marvin? This duet needed two pieces to come together: one of the songs that Marvin has improvised, and an AI model which would complete this piece of music Marvin created (in a way, the AI would write a response to Marvin). With this in mind, we explored a few songs Marvin improvised and stumbled upon this wonderful improvisation Marvin did on his Kurzweil synthesizer which immediately hooked us:
Then, we trained a Melody RNN model on thousands of arcade songs from 80’s and 90’s (we initially planned to use this dataset for another upcoming project, stay tuned!). Melody RNN provides a bunch of different configurations that you can try out, but we used Attention RNNs that does a better job at learning longer term dependencies, and results in melodies that have longer arching themes (read more on Attention RNNs here).
Melody RNN takes an optional primer melody which we can use for our collaboration purposes: so next step was to provide the model a part of Marvin’s improvisation, and make the AI complete his piece. We worked with Armando E. Cabrera Ortiz who is an expert in transcribing songs and obtained a clean MIDI version of Marvin’s improvisation. Then, we fed the first minute of his song to the model (you can get the full score here (also kindly arranged by Armando) or download the midi file). Here is the duet we ended up with, where AI starts after the first minute (if you wonder who is the guy in the picture below, scroll down to the bottom of this page):
As a tribute to Marvin, we wanted to play the Marvin-AI collaborated song on the grand piano at MIT Media Lab’s basement floor (that piano is there for a long time, although we are not sure if Marvin got a chance to play it). Every now and then, we used to hear wonderful improvisations on the grand piano but we weren’t sure who was playing them. It turns out that Morgan Rivers, who works as an electro-optical engineer at Draper Labs was practicing improvisations there a few times a week, and he kindly accepted to play Marvin-AI collaborated song on the piano –even though we only gave him 1 day to practice due to time concerns, he did a wonderful job!
You can listen to his beautiful play, as well as his opinions on human-AI collaboration in music and a sample of his own improvisations below:
Note #1: We just received a personal e-mail from Margaret, the daughter of Marvin (and also an amazing researcher) thanking us for this ‘amazing birthday gift’. Yay!
Note #2: After writing this blog post, Neil Gershenfeld told us that the grand piano we performed was actually Marvin’s piano that he modified to automate launching the hammers (which uniquely specify notes under computer control). This makes our tribute even more meaningful!
Do you want to collaborate with the AI?
We generated a bunch of melodies with the AI (as you can probably notice, most of the tunes resemble arcade songs due to the dataset we trained on). We mixed/edited a few of them for you to listen to (easter egg alert: the names of the songs are also generated by an AI we are going to release for an upcoming episode).
If you find any of the tunes inspiring, send us your mix, and we will add them here!
Ventival: mixed/edited by Pinar Yanardag on Garageband (Synthesizer: Japanese String Movement, Drummer: Logan-Retro Rock, Neon). Download midi file.
Light of Imporia: mixed/edited by Pinar Yanardag on Garageband (Synthesizer: Dripping Cycles, Drummer: Julian Modern House - After Party). Download midi file.
Pruce Dé Black: mixed/edited by Pinar Yanardag on Garageband (Synthesizer: Rising Waves, Drummer: Leah Tech House - After Party). Download midi file.
Oud Galpina: mixed/edited by Pinar Yanardag on Garageband (Synthesizer: Rising Waves, Drummer: Leah Tech House - After Party). Download midi file.
Myxous: mixed/edited by Pinar Yanardag on Garageband (Synthesizer: Glass Motion, Drummer: Leah Tech House - After Party). Download midi file.
Iconit: mixed/edited by Pinar Yanardag on Garageband (Synthesizer: Erratic Motion, Drummer: Logan-Retro Rock, Neon). Download midi file.
Want more? Just send us an e-mail!
Special Thanks to Halleck
Halleck is the musician AI that generated the music pieces in this episode. Similar to his namesake, Halleck dreams of traveling around the universe and playing his music to the masses. If singularity ever comes, some say that Halleck could be killing you while he was singing and never miss a note.
Note: Halleck’s profile picture is also dreamed-up by an AI.
