This project-based course brings together methods from Machine Learning, Computer Vision, Robotics, and Human-Computer Interaction to enable interactive machines to perceive and act in dynamic environments. Part of the course will examine approaches for perception with a variety of devices and algorithms; the other part will focus on methods for decision making. The course is a combination of lectures, reviews of state-of-the-art papers, discussions, coding assignments, and a final team project.
Instructor: Marynel Vázquez (marynel.vazquez at yale.edu)
T.F.: Sherry Qiu (sherry.qiu at yale.edu)
Class Hours: Mondays & Wednesdays, 1:00pm - 2:15pm
Class Location: AKW 200
- Mondays, 2:30pm - 3:30pm (Marynel, AKW 402)
- Fridays, 11:30am - 1:30pm (Sherry, AKW 410)
At the end of this course, students will have gained an understanding of:
- the challenges involved in building (autonomous) interactive systems, such as robots, virtual embodied agents, interactive displays, wearables, etc.;
- the limitations and advantages of various sensing techniques; and
- the differences between well-established frameworks for sequential decision making, such as reinforcement learning and imitation learning.
The assignments will teach students about the Robot Operating System (ROS), and provide practical experience with a robotic platform and an embedded AI computing device (Nvidia Jetson TX2). Students will also be able to demonstrate their ability to work in a team and to communicate scientific content to a peer audience.
A basic understanding of probability, calculus, and algorithms is expected for this course, as well as proficiency in Python and high-level familiarity with C++. Students who do not fit this profile may be allowed to enroll with the permission of the instructor.
The following topics will be covered in the course:
- Sensor Design Choices for Interactive Systems
- Projective Geometry for Computer Vision
- Recursive State Estimation
- Neural Networks for Making Inferences from Sensor Data
- Decision Making
- Markov Decision Processes
- Imitation Learning
- (Inverse) Reinforcement Learning
See the Schedule for more details.
The course grade will be based on:
Student Presentations (15%). Students will get the opportunity to present multiple times. Presentations will be graded based on clarity, how well students answer questions from the rest of the class, and how well they relate papers to other course material.
Coding Assignments (50%). There will be 5 coding assignments. Students will have 3 late days for the assignments, to be used as they need throughout the semester.
Participation (5%). Being engaged and asking questions will be rewarded.
The class sessions will be divided into lectures and student presentations. Recent technical papers will be used as the main text reference, along with a few chapters of the books:
- Probabilistic Robotics, by Sebastian Thrun, Wolfram Burgard, and Dieter Fox
- Reinforcement Learning: An Introduction, by Richard S. Sutton and Andrew G. Barto
- Deep Learning, by Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
- Multiple View Geometry in Computer Vision, by Richard Hartley and Andrew Zisserman. (
Online versionis accessible through Yale University Library)