Presented By
O’Reilly + Intel AI
Put AI to Work
April 15-18, 2019
New York, NY

Make Music Composing Easier for Amateurs: A Hybrid Machine Learning Approach

Andrew Caosun (Horace Mann School)
4:55pm5:35pm Thursday, April 18, 2019
Interacting with AI
Location: Regent Parlor
Secondary topics:  AI case studies, Models and Methods

Who is this presentation for?

Computer-aided music making, entrepreneurs

Level

Intermediate

Prerequisite knowledge

some knowledge on music theory

What you'll learn

Our goal is to make music entertainment and education accessible to music untrained people. More specifically, we aim to propose a model that takes users original musical ideas in an intuitive way, automatically modifies the original musical scores and output musically appropriate melodies. To achieve this, we propose a model that units Hidden Markov Model (HMM) with Recurrent Neural Network (RNN) algorithm that assist music composing for musically untrained users. The HMM allows us to take users’ original music score as input and generate melody that conforms music theory/regulation and reflects user specified preference. We further incorporate Convolution Neural Networks (CNN) to preserve the music semantics such as rhythm, octave, and chord and Long-Short Term Memory (LSTM) networks to capture the dependency in the context of a whole measure. Thus, the hybrid model leverages the power of deep learning to build a holistic model that learns both local and global correlations of data. Trained on large existing digital library of different genres of music, the hybrid deep learning algorithm is further extended to allow users to specify magnitude of revision, duration of music segment to be revised, choice of music genres, popularity of songs, and co-creation of songs in social settings. These extensions enhance user understanding of music knowledge, enrich their experience of self-music learning, and enable the social aspects of music making.

Description

Creating your own musical pieces is one of the most wonderful ways of enjoying music. However, many lack the basic musical skills to do so. In this paper, we seek to explore how machine learning algorithms can enable musically untrained users to create their own music.

To achieve this, we propose a Neural Hidden Markov Model (NHMM), which is a hybrid of a Hidden Markov Process and convolution Neural Network algorithm with LSTM. This model takes users original musical ideas in an easy intuitive way, automatically modifies the input and generates musically appropriate melodies as output. We further extend the model to allow users to specify magnitude of revision, duration of music segment to be revised, choice of music genres, popularity of songs, and co-creation of songs in social settings. These extensions enhance user understanding of music knowledge, enrich their experience of self-music learning, and enable social aspects of music making. The model is trained using Columbia’s publicly available Million Songs Dataset. We also conduct experiments on melody generation.

We also design a mobile application with intuitive, interactive, and graphical user interface which is suitable to the elderly and young children. Different from most existing literature focusing on computer music composing itself, our research and application aim at using computers to aid human composition and enriching music education of musically untrained people.

Keywords: music technology, computer-aided music composing, machine learning (ML), Hidden Markov Model (HMM), Recurrent Neural Network (RNN), LSTM, Convolutional Neural Network, human-computer interaction

Photo of Andrew Caosun

Andrew Caosun

Horace Mann School

I am senior at Horace Mann School. I’ve been actively involved with Concerts in Motion since middle school, spending Sunday afternoons singing with seniors in nursing homes. I’ve also participated in seasonal events at the Turtle Bay Music School where we raised a music education fund for children from disadvantaged families. The friendships I’ve developed during these events have helped me to understand just how much music can mean to someone. Instead of just listening to someone singing once a week, everyone should be able to create their own music. Thus, I had the idea of combining my love for singing and recent technological advancements to help others compose their own pieces. I developed this research under the guidance of Professor David Gu.

We are among the first to apply a hybrid of HMM and convolutional neural network with LSTM to the music composing. The hybrid approach is further extended to allow users to specify magnitude of revision, duration of music segment to be revised, choice of genres, popularity of songs, and co-creation of songs in social settings. The mobile user interface we designed are intuitive, interactive, and flexible, suitable to the elderly and young children.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)