MPATE-GE 2039: Deep Learning for Media

Instructor: Magdalena Fuentes
Email: mfuentes@nyu.edu
Spring 2023, 3 credits
Class meetings: 2:00 PM – 4:30 PM W
Classroom: 35 West 4th Street, EDUC building, room 610
Office hours: https://bit.ly/mf_office_hours

Course Description

Deep learning, a sub-field of machine learning and artificial intelligence, has promoted breakthroughs in managing and creating media content, and continues to shape the future of the multimedia landscape. This course provides a hands-on, project-oriented introduction to deep learning, for the classification, retrieval, and creation of media content, with emphasis in audio-visual content. Students will create and work with existing deep learning models using Python libraries, and think critically about the application of these models for media. This course will provide students with an understanding of how these tools work, and how to use them in the context of their work, so they can participate in the ongoing revolution between machine learning and media content.

Course Objectives

The goal of this course is to provide students with:

Student Learning Outcomes

By the end of the course, students will be able to:

Required Skills

Basic knowledge of Python is required. Basic knowledge of linear algebra is recommended.

Required Course Materials

A computer with Internet access.

Course Requirements

Class participation

This course will be highly participatory and interactive. We will have conversational lectures, live-coding demonstrations and group readings. It is expected and essential that students participate actively in class. For the class readings, we are going to follow the role-playing paper reading format of Jacobson and Raffel. In this format, students read the same paper, but each student takes one specific role for that week. We will work with three different roles, to be determined at the beginning of the course. Each student will participate by choosing one role in the semester. Papers will be assigned to students during the first week of classes. See a short guide on how to read a paper.

Homework assignments

There will be 3 homework assignments. Each assignment will consist of Python code snippets and written components to be completed by you. These assignments are individual. You will have two weeks to complete each assignment. For the homework assignments, your lowest score will be dropped. If you can’t finish one assignment for any reason, you should make the time to cover the material, but no need to panic about late assignments.

Final Project

The final project can be done in groups of 2-3, and will have a code and presentation component. Projects will be presented during the last class. A project can cover the assessment of an existing deep learning system in new data, or a new system proposed by the students. The project will have four assignments through the semester as follows:

Topic, group definition and data report. Each group will propose a project’s topic and composition and will submit a short report (max. 2 pages) describing the scope of the work and data resources they will use. The project’s final scope and composition will be determined jointly with the instructor. Examples of feasible projects will be provided by the instructor in advance to guide the students in the definition of their project.

Documented code, and data. Students will submit their working (and well documented!) code along with the data they investigated. The code should run out-of-the-box, and should have instructions on how to run it if needed.

Final Presentation. All students from each group should participate in a final presentation. Presentations should last between 5-10 minutes, and will be followed by a short Q&A from the class.

Your grade for the class will be determined by the following break-down:

Classes will be in person weekly unless indicated by the instructor, and materials will be available on NYU Brightspace.

Recommended Readings

Main textbook:

Other recommended readings:

Class Environment

Please do your part by seeking to promote the success of others, and by treating each other in ways that respect and celebrate the diversity of talent.

Resources at NYU

Access your course materials: NYU Brightspace
Databases, journal articles, and more: Bobst Library
Assistance with strengthening your writing: NYU Writing Center
Obtain 24/7 technology assistance: IT Service Desk

Weekly Schedule

Week 0: Supplemental materials
Week 1, 01/25: Introduction to Machine Learning
  • Course structure and schedule
  • Why deep learning? Why now?
  • Frame a machine learning problem, collect a dataset, understand your data, choose a measure of success
  • Recommended reading: DLwP Chapter 1, Chapter 6
  • Practice: GitHub basics, submitting homework
Week 2, 02/01: The building blocks of neural networks
  • Data representations: scalars, vectors, matrices, tensors, sequential data
  • Data operations
  • Practice: train a first neural network for handwritten digit classification; adapt the network for audio instrument classification
  • Recommended reading: DLwP Chapter 2.1-2.3
Week 3, 02/08: The engine of neural networks
  • Gradient-based optimization: derivative, derivative of tensors, stochastic gradient descent, backpropagation
  • Practice: gradient descent
  • Recommended reading: DLwP Chapter 2.4
Week 4, 02/15: Setting up a deep learning project
  • Keras and Tensorflow, layers and models, loss function, metrics, inference
  • Recommended reading: DLwP Chapter 3
  • ASSIGN: paper readings for the semester
Week 5, 02/22: Classification, model generalization and evaluation
  • Underfitting and overfitting, improving model fit, model biases and ethical implications, interpretability
  • Practical example: a biased classifier, an ethically questionable classifier, a classifier that doesn’t generalize
  • Recommended reading: DLwP Chapters 4, 5
  • ASSIGN: homework 1 · Role-playing paper reading 1
Week 6, 03/01: Convolutional Neural Networks
  • An introduction to Computer Vision, convolutional networks, max-pooling
  • Practical example: image classification, training from scratch
  • Recommended reading: DLwP Chapter 8
Week 7, 03/08: Transfer learning and Data augmentation
  • Image data augmentation, feature extraction with a pre-trained model
  • Recommended reading: DLwP Chapter 8
  • DUE: homework 1 · ASSIGN: homework 2 · Role-playing paper reading 2
Week 8, 03/15: Spring Break
Week 9, 03/22: Audio understanding
  • An introduction to Machine Listening, audio manipulation, spectrograms, audio data augmentation
  • Recommended reading: Computational Analysis of Sound Scenes and Events, Chapter 4
  • DUE: homework 2 · ASSIGN: homework 3
Week 10, 03/29: Audio classification
  • Temporal convolutional networks for audio
  • Training from scratch and feature extraction with a pre-trained model
  • Role-playing paper reading 3
Week 11, 04/05: Cross-modal retrieval
  • Overview and applications, embedding space, contrastive loss
  • Practical example: audio-visual cross modal retrieval
  • DUE: homework 3 · ASSIGN: project 1 · Role-playing paper reading 4
Week 12, 04/12: Neural style transfer
  • Overview of style transfer in image and audio; style loss and content loss
  • Recommended reading: DLwP Chapter 12.3
  • DUE: project 1 · ASSIGN: project 2 · Role-playing paper reading 5
Week 13, 04/19: Image generation
  • GANs for image generation, challenges and tricks, diffusion techniques overview
  • Recommended reading: DLwP Chapter 12.5
  • Role-playing paper reading 6
Week 14, 04/26: Audio generation
  • Variational Autoencoders for audio generation, sampling from audio latent spaces
  • Recommended reading: DLwP Chapter 12.4
  • DUE: project 2 · ASSIGN: project 3 · Role-playing paper reading 7
Week 15, 05/03: Final project presentations
  • DUE: project 3