Teaching - Sampling: From MCMC to Generative Modeling

Please fill the following questionnaire:

Course

Dates: 26th, 27th morning; 2nd, 16th, 23rd, 30th (afternoon). At least 2 practical sessions (2nd - Langevin Monte Carlo - and 30th). Bring your computers.

Slides are in construction and are susceptible to change by the time I teach the course, especially Part III is in construction. Part I~30 slides, Part II~80 slides, Part III~30 slides. Part II is long because we will spend time on some important notions such as gradient flows, score, or continuity equations which are crucial for NF and DM for instance.

  • Part I Motivation, Divergences between probability distributions, Reminders on basics from MCMC
  • Part II (Sampling as Optimization of the KL): gradient flows in finite and infinite dimensions, continuity equation, Langevin Monte Carlo
    + stochastic Langevin dynamics based on these notes.
  • Part III (Generative Modeling): Normalizing Flows (NF), Diffusion Models (DM) - if time allows briefly GANs and VAEs.

Additional Resources:

(Practical) Resources.

We won't have time to go through everything of course, but here are several practical Python notebooks that we may go through together or by yourself; or can even serve as a basis for a project if implemented in combination with novel ideas or datasets.
  • Langevin Monte carlo/Metropolis Hasting/Hamiltonian Monte Carlo
    • Stochastic Langevin with linear model on simulated data link (solution after the practical session) here
    • Bayesian Logistic Regression on simulated data, link
    • Metropolis Hasting, link
    • MALA vs HMC, link
  • Normalizing Flows
    • Normalizing Flow in 2D on Mixture of Gaussians, link - check ReadMe to run it on Google Colab.
    • To go further on Normalizing Flows (on images), link
  • Deep Generative Models
    • GANs and VAEs, Deep Learning Indaba 2019, link
    • Denoising Diffusion, Deep Learning Indaba 2012, link

Project.

The course project will give the students a chance to explore MCMC and/or generative modeling in greater detail. Course projects will be done in groups of up to 3 students and can fall into one or more of the following categories:
  • Application of MCMC or deep generative models on a novel task/dataset.
  • Algorithmic improvements into the evaluation, learning and/or inference of deep generative models.
  • Theoretical analysis of any aspect of existing deep generative models.
The report must include theoretical, methodological and experimental considerations. You will be evaluated on these three aspects. Please indicate the contribution of each member of the team in the report. We do not expect the students to reproduce some of the high-dimensional experiments presented in some of these papers as we are aware of the compute limitations.

The deadline for submission is 26th of April. You will be asked to present your result during an oral defence (eg with slides and/or notebook). Please send your
* pdf
* colab link with experiments/code
to anna.korba@ensae.fr.

Some examples:
  • Implement in Python Langevin Monte Carlo with birth and death dynamics as in this paper or that paper. No need to fully undertand the paper, but at least the pseudocode and reproduce the experiments. Can you identify setting where birth-death dynamics improve over standard LMC?
  • Study the results of Bayesian logistic regression on real data (see here for an example of what can be done) on a classification dataset (see UCI repository, or some examples as the ones in Section 5 of this paper).
  • Investigate the performance of Bayesian Neural Networks (see a pytorch implementation here) on different datasets (eg Fashion Mnist, or other classification datasets as the ones in Section 5 of this paper).
  • Investigate the data mollification effect as in this paper to compare different generative models on a set of experiments.