MPATE-GE 2623: Music Information Retrieval

Instructor: Assist. Prof. Magdalena Fuentes, PhD
Email: mfuentes@nyu.edu
Fall 2023, 3 credits
Class meetings: 12:00 PM – 1:40 PM W
Classroom: 35 West 4th Street, EDUC building, room 610
Office hours: https://bit.ly/mf_office_hours

Course description

This course gives a comprehensive overview of research in Music Information Retrieval (MIR). MIR is a multidisciplinary field which uses knowledge from areas as diverse as signal processing, machine learning, cognition and music theory. The course will explore how this knowledge is applied to different problems within MIR, with an emphasis on common research methodologies.

Course objectives

The goal of this course is to provide students with:

Broad knowledge of common topics in MIR, including their definitions, challenges, and evaluation metrics.
Awareness of the current state-of-the art and remaining challenges for each topic.
The ability to critically analyze the behavior of MIR systems.
Practical skills for working with open source MIR resources in Python.

Student Learning Outcomes

Explain and describe common tasks in MIR, including definitions, challenges and evaluation metrics.
Code MIR pipelines using open source Python libraries.
Assess and analyze the behavior of MIR systems.
Explain current trends in MIR research.

Recommended Skills

Basic knowledge of Python programming and Digital Signal Theory are required.

Required Course Materials

A computer with Internet access.

Course Requirements

Class participation

This course will be highly participatory and interactive. We will have conversational lectures and group readings. For the class readings, we follow the role-playing paper reading format of Jacobson and Raffel. Each student will participate by choosing one role in the semester.

Homework assignments

There will be 4 homework assignments. Each assignment will consist of Python code snippets and written components. These assignments are individual. You will have two weeks to complete each assignment. Your lowest score will be dropped.

Final Project

The final project can be done in groups of 2-3, and will have a code and presentation component. A project can cover the assessment of an existing MIR system in new data, or a new system proposed by the students. All projects should include a baseline system for comparison.

Your grade for the class will be determined by the following break-down:

40% assignments (individual)
40% final project (groups)
20% class participation (individual)

Class Environment

Please do your part by seeking to promote the success of others, and by treating each other in ways that respect and celebrate the diversity of talent. Policies on preparation, accessibility (Moses Center, 212-998-4980), mental health (NYU Wellness Exchange, 212-443-9999), preferred names and pronouns, class expenses, feedback, academic integrity, and inclusion match those of my other courses.

Resources at NYU

NYU Brightspace · Bobst Library · NYU Writing Center · IT Service Desk

Weekly Schedule

Week 0: Supplemental materials

Python/Numpy tutorial and GitHub starter course

Week 1, 09/06: Course introduction & Music representations

Overview of MIR field, course structure, GitHub classroom
Sheet music, piano-roll, MIDI, audio
ASSIGN: paper readings for the semester

Week 2, 09/13: Time-Frequency Representations

STFT, log-frequency spectrogram, constant-Q transform
Recommended reading: McFee’s Digital Signals Theory, Chapters 5 and 9

Week 3, 09/20: Machine Learning and MIR

Supervised learning, features, train/validation/test, classifiers, metrics, data augmentation
ASSIGN: Homework 1

Week 4, 09/27: Sound Classification

Instrument identification, MFCCs, confusion matrices, precision/recall/F-measure
Pitch shifting, time stretching, resampling, background noise, dynamic range compression

Week 5, 10/04: Beats and Downbeats

Beat and downbeat tracking, onsets and novelty curves, metrics
Recommended reading: FMP 6.1.1, 6.3 and ISMIR tutorial
DUE: Homework 1 · ASSIGN: Homework 2

Week 6, 10/11: Music Similarity

Embeddings, fingerprinting, version identification, music recommendation
Recommended reading: FMP 3.2, 7.2, 7.3

Week 7, 10/18: Chords [Class ONLINE]

Chord estimation, chroma, metrics
DUE: Homework 2 · ASSIGN: Homework 3

Week 8, 10/25: Open Week

Week 9, 11/01: Pitch

Pitch tracking, melody estimation, salience representations
DUE: Homework 3 · ASSIGN: Homework 4

Week 10, 11/08: Transcription

Transcription, non-negative matrix factorization, metrics
ASSIGN: project topic, data and baseline

Week 11, 11/15: Source Separation

Harmonic-percussive and instrument separation, stems, metrics
DUE: project topic, data and baseline · DUE: Homework 4 · ASSIGN: project code and data

Week 12, 11/22: NO CLASS (Thanksgiving)

Week 13, 11/29: Multimodal MIR

Overview and applications, embedding space, contrastive loss

Week 14, 12/06: Music Generation

Stable diffusion models, variational models
DUE: project code and data · ASSIGN: project presentation

Week 15, 12/13: Final Presentations

DUE: project presentation