News

Course Info

Syllabus

Description

This is an introductory graduate course in machine learning. This course will focus on both supervised and un-supervised learning methods, covering both theory and practice. The course is geared towards students who wish to develop a working knowledge of the recent advances in machine learning, and how these are applied in various domains.

Machine learning deals with how to design computer programs that learn from “experience.” Residing at the intersection of computer science and statistics, machine learning aims to extract useful information from data (often referred to as the training data) and leverages this information to create computer models capable of carrying out useful, non-trivial tasks, such as designing cars that can drive on their own, filters for blocking junk email, diagnostics tools for disease discovery, etc. By many accounts machine learning is the “greatest export” of computer science (and statistics) to other disciplines.

The course will cover the following topics:

• regression;
• classification;
• clustering;
• dimensionality reduction;
• mixture-models; and
• neural networks and deep learning.

Prerequisites

The course assumes that students are comfortable with statistics, basic linear algebra, and programming.

Reading material

We live in exciting times. Copious amount of information about machine learning is available on the internet. Check out the awesome machine learning on Github for list of machine learning courses and free, open source books.

Lectures

Important: Each lecture will include a programming activity. Please bring your laptops in to the lectures. Also ensure that your laptop has Python, Numpy, Scipy, Matplotlib, and Sklearn installed. The easiest way to achieve this it to download the Anaconda Python distribution.

Week 1

Tony Joseph will lead the first lecture. I am away at a conference in Berlin.

• Machine learning (ML) introduction
• ML basics
• K-nearest neighbours

Week 2

• Clustering
• Principle component analysis

Exercise

The goal is to use Kmeans or Meanshift to cluster the "make circles" dataset into two clusters. [Code]

Week 3

• Linear regression
• Optimization

Class notes

• Linear regression
• Optimization
• 1D linear regression
• Linear regression derviation

Exercise

Experiments with linear regression. [Code] [Data]

Week 4

• Probabilistic view of regression
• Logistic regression
• Softmax

Class notes

• Notes

Week 5

• Layered view of logistic regression and softmax

Class notes

• Notes

Week 6

• Neural networks

Week 7

• Tensorflow (Convolutional network for Cifar-10 image classification)

Week 8

• Recurrent Neural Networks and Long Short Term Memory Networks

Week 9

• Bayesian reasoning

Class notes

• Notes

Week 10

• Gaussian processes

Week 11

We will continue our discussion of Gaussian processes.

Week 12

• Course project presentations

Papers

Each student needs to select a relevant machine learning paper and give a 20 minutes presentation, outlining the contributions, strengths and weaknesses of that paper. To get the process moving I have started to put together a list of papers. Each of you is asked to select "one" paper that catches your interest. I will use FIFO to resolve ties.

Feel free to suggest another relevant machine learning paper

Reading papers

Presentation

Each paper presentation is give or take 20 minutes long, followed by a discussion. It is expected that all of you would've read the paper before coming to the lecture. The presentation should focus on the "key contribution" of the paper and how the topics covered in the paper fit into the larger machine learning landscape. Pay close attention to how paper is written, how ideas are presented, how methods are developed, how arguments are structured and how results are used to bolster the key idea of the paper.

Presentation schedule and papers are available via course Slack.

Project

The students can work on projects individually or in pairs. The project can be an interesting topic that the student comes up with himself/herself or with the help of the instructor. The grade will depend on the ideas, how well you present them in the report, how well you position your work in the related literature, how thorough are your experiments and how thoughful are your conclusions. (Taken from Raquel Urtasun CSCI 2515 project description.)

Administration

1. In class, Wed., Nov. 29
2. 10 min total time, 7 min presentation and demo, 3 min QA and discussion
3. To ensure timely proceedings, please upload project presentations via Blackboard by Tue., Nov. 28 midnight in pdf format.
4. Project report due on Fri., Dec. 8, 11:59 pm

Assignments

• Assignment 1 Due back Oct 11, 11:59 pm
• Assignment 2 Due back Nov 1, 11:59 pm
• Assignment 3 Due back Nov 27, 11:59 pm

Resources

I recommend reading Part 1 of “Deep Learning” by I. Goodfellow, Y. Bengio and A. Courville to brush up on linear algebra and statistics. The book is available at here

We will be using Python for the programming part of this course. For Python, I recommend the Anaconda distribution, which comes pre-loaded for nearly all the packages that we will be using in this course. Of course you are welcome to use any variant/distribution of Python that suits you.

Here you’ll find a number of tutorials showcasing Python use in machine learning. I strongly recommend that you become comfortable with the following four Python packages/environment:

• numpy;
• scipy;
• matplotlib;
• jupyter notebook; and
• TensorFlow.

Books

• "Deep Learning" by I. Goodfellow, Y. Bengio and A. Courville
• "Pattern Recognition and Machine Learning" by C. Bishop
• "Machine Learning: A Probabilistic Perspective" by K. Murphy
• "Generalized Linear Models" by P. McCullagh and J.A. Nelder
• "The Elements of Statistical Learning: Data Mining, Inference, and Prediction" by T. Hastie, R. Tibshirani and J. Friedman
• "Machine Learning with TensorFlow" by N. Shukla (This book is particular useful for those interested in applied machine learning.)

Code

Code examples used in this course are available on Github (https://github.com/uoit-ml/mcsc-ml).