Course Description

This project-based course brings together methods from Machine Learning, Computer Vision, Robotics, and Human-Computer Interaction to enable interactive machines to perceive and act in dynamic environments. Part of the course will examine approaches for perception with a variety of devices and algorithms; the other part will focus on methods for decision making. The course is a combination of lectures, reviews of state-of-the-art papers, discussions, coding assignments, and a final team project.

Instructor: Marynel Vázquez (marynel.vazquez at
T.F.: Sherry Qiu (sherry.qiu at

Class Hours: Mondays & Wednesdays, 1:00pm - 2:15pm
Class Location: AKW 200

Office Hours:
- Mondays, 2:30pm - 3:30pm (Marynel, AKW 402)
- Fridays, 11:30am - 1:30pm (Sherry, AKW 410)

Canvas Link:

Learning Objectives

At the end of this course, students will have gained an understanding of:

  • the challenges involved in building (autonomous) interactive systems, such as robots, virtual embodied agents, interactive displays, wearables, etc.;
  • the limitations and advantages of various sensing techniques; and
  • the differences between well-established frameworks for sequential decision making, such as reinforcement learning and imitation learning.

The assignments will teach students about the Robot Operating System (ROS), and provide practical experience with a robotic platform and an embedded AI computing device (Nvidia Jetson TX2). Students will also be able to demonstrate their ability to work in a team and to communicate scientific content to a peer audience.


A basic understanding of probability, calculus, and algorithms is expected for this course, as well as proficiency in Python and high-level familiarity with C++. Students who do not fit this profile may be allowed to enroll with the permission of the instructor.


The following topics will be covered in the course:

  • Perception
    • Sensor Design Choices for Interactive Systems
    • Projective Geometry for Computer Vision
    • Recursive State Estimation
    • Neural Networks for Making Inferences from Sensor Data
  • Decision Making
    • Markov Decision Processes
    • Imitation Learning
    • (Inverse) Reinforcement Learning

See the Schedule for more details.


The course grade will be based on:

  • Student Presentations (15%). Students will get the opportunity to present multiple times. Presentations will be graded based on clarity, how well students answer questions from the rest of the class, and how well they relate papers to other course material.

  • Coding Assignments (50%). There will be 5 coding assignments. Students will have 3 late days for the assignments, to be used as they need throughout the semester.

  • Final Project (30%). Students will work on a final project creating an interactive robotic system.
    ★ View the student’s projects here.

  • Participation (5%). Being engaged and asking questions will be rewarded.


The class sessions will be divided into lectures and student presentations. Recent technical papers will be used as the main text reference, along with a few chapters of the books:


This course is made possible through the generous support of Yale’s Provost Office, IT at Yale, Google Cloud, and Nvidia.