Put AI to Work
April 15-18, 2019
New York, NY

Make music composing easier for amateurs: A hybrid machine learning approach

Baohong Sun (Horace Mann School)
4:55pm5:35pm Thursday, April 18, 2019
Interacting with AI
Location: Regent Parlor
Secondary topics:  AI case studies, Models and Methods

Who is this presentation for?

  • Entrepreneurs and those interested in computer-aided music making



Prerequisite knowledge

  • Familiarity with music theory (useful but not required)

What you'll learn

  • Explore a framework that unifies hidden Markov models and deep learn algorithms (RNN) with modeling components that consider long-term memory and semantics of music


Creating your own musical pieces is one of the most wonderful ways of enjoying music, but many lack the basic musical skills to do so. Andrew Caosun discusses a neural hidden Markov model (NHMM)—a hybrid of a hidden Markov process and a convolutional neural network algorithm with LSTM. This model takes users’ original musical ideas, automatically modifies the input, and generates musically appropriate melodies as output.

The model is extended to allow users to specify magnitude of revision, duration of music segment to be revised, choice of music genres, popularity of songs, and cocreation of songs in social settings. These extensions enhance user understanding of music knowledge, enrich their experience of self-music learning, and enable social aspects of music making. The model is trained using Columbia’s publicly available Million Songs Dataset.

Andrew also explains how he and his team designed a mobile application with an intuitive, interactive, and graphical user interface that’s suitable for the elderly and young children. Unlike most existing literature focusing on computer music composing itself, their research and application aim to use computers to aid human composition and enrich music education for those without musical training.

Photo of Baohong Sun

Baohong Sun

Horace Mann School

Andrew Caosun is a senior at Horace Mann School. He’s been actively involved with Concerts in Motion since middle school, spending Sunday afternoons singing with seniors in nursing homes, and has also participated in seasonal events at the Turtle Bay Music School, raising a music education fund for children from disadvantaged families. The friendships he developed during these events helped him understand just how much music can mean to someone, giving him the idea to combine his love for singing and recent technological advancements to help others compose their own pieces. His research, under the guidance of David Gu, applies a hybrid HMM and convolutional neural network with LSTM to compose music.