Seminar on Theoretical Machine Learning

Learning Controllable Representations

Richard Zemel
University of Toronto; Member, School of Mathematics
April 2, 2020
As deep learning systems become more prevalent in real-world applications it is essential to allow users to exert more control over the system. Exerting some structure over the learned representations enables users to manipulate, interpret, and even obfuscate the representations, and may also improve out-of-distribution generalization. In this talk I will discuss recent work that makes some steps towards these goals, aiming to represent the input in a factorized form, with dimensions of the latent space partitioned into task-dependent and task-independent components.

Some Recent Insights on Transfer Learning

Samory Kpotufe
Columbia University; Member, School of Mathematics
March 31, 2020
A common situation in Machine Learning is one where training data is not fully representative of a target population due to bias in the sampling mechanism or high costs in sampling the target population; in such situations, we aim to ’transfer’ relevant information from the training data (a.k.a. source data) to the target application. How much information is in the source data? How much target data should we collect if any? These are all practical questions that depend crucially on 'how far' the source domain is from the target.

Margins, perceptrons, and deep networks

Matus Telgarsky
University of Illinois
March 26, 2020
This talk surveys the role of margins in the analysis of deep networks. As a concrete highlight, it sketches a perceptron-based analysis establishing that shallow ReLU networks can achieve small test error even when they are quite narrow, sometimes even logarithmic in the sample size and inverse target error. The analysis and bounds depend on a certain nonlinear margin quantity due to Nitanda and Suzuki, and can lead to tight upper and lower sample complexity bounds.

Joint work with Ziwei Ji.

Your Brain on Energy-Based Models: Applying and Scaling EBMs to Problems of Interest to the Machine Learning Community Today

Will Grathwohl
University of Toronto
March 10, 2020
In this talk, I will discuss my two recent works on Energy-Based Models. In the first work, I discuss how we can reinterpret standard classification architectures as class conditional energy-based models and train them using recently proposed methods for large-scale EBM training. We find that adding EBM training in this way provides many benefits while negligibly affecting discriminative performance, contrary to other hybrid generative/discriminative modeling approaches.

Learning from Multiple Biased Sources

Clayton Scott
University of Michigan
February 25, 2020
When high-quality labeled training data are unavailable, an alternative is to learn from training sources that are biased in some way. This talk will cover my group’s recent work on three problems where a learner has access to multiple biased sources. First, we consider the problem of classification given multiple training data sets corrupted by label noise, and describe a weighted empirical risk minimization strategy where the weights are optimized according to the degree of corruption of each source.