Deep Learning (CAS machine intelligence)

This course in deep learning focuses on practical aspects of deep learning.

For the hands-on part we provide a docker container (details and installation instruction).

Other resources

We took inspiration (and sometimes slides / figures) from the following resources.

Deep Learning Book (DL-Book) http://www.deeplearningbook.org/. This is a quite comprehensive book which goes far beyond the scope of this course.
Convolutional Neural Networks for Visual Recognition http://cs231n.stanford.edu/, has additional material and youtube videos of the lectures. While the focus is on computer vision, it also treats other topics such as optimization, backpropagation and RNNs. Lecture notes can be found at http://cs231n.github.io/.
More TensorFlow examples can be found at dl_tutorial
Another applied course in DL: TensorFlow and Deep Learning without a PhD

Syllabus

The course is split in 8 sessions, each 4 hours long.

Tensorchiefs are Oliver Dürr, Beate Sick and Elvis Murina.

Day	Topic and slides	Additional Material	Exercises and homework
1	Deep learning basics slides Overview of deep learning Computational graphs, feeding and fetching Loss function (crossentropy) Gradient descent and generalizations Example: linear regression	Nature review article DL-book chapter 6 TensorFlow Intro	01 Matrix Multiplication Exercises \| Notebook \| Solution 02 Linreg (with_slider_solution) Exercises \| Notebook Solution LinearRegression 03 Homework: Frozen Graph (Artstyle Transfer) Exercises \| Solution Exercise Homework: Mandelbrot Notebook
2	Multinomial Logistic Regression slides Logistic regression Multinomial Logistic Regression	DL-book chapter 6	TF-playground 04 Logistic Regression (Challenger) Exercises \| Solution 05 Multinomial Logistic Regression Exercises Solution
3	Going Deeper / Tricks of the trade slides Fully connected network Backpropagation and Gradient Flow ReLU Regularization: Early stopping L2 (Weight Decay) Dropout	Backpropagation DL-book chapter 7 (for regularization)	06 Fully Connected Network MNIST Exercises Solution 07 Fully Connected network (Tricks of the Trade) Exercises Solution
4	Convolutional Neural Networks I slides Batch-Normalization Why going beyond fully connected NN? What is convolution? Building a CNN Simlarities between a CNN and the brain	Convolution arithmetic demo CNN and activation maps	finger-exercise ReLu Exercise\| Solution 08 CNN Exercises Solution CNN1 with centered MNIST\| Solution CNN1 without centering\| Solution CNN2 MNIST 09 and 10 Homework: 8 Faces Data overview\| Exercise \| FC Solution\| CNN Solution
5	Convolutional Neural Networks II slides Typical CNN architectures Transfer learning: Use pretrained nets for fine-tuning or feature generator Feature/activation maps in detail Understand CNN features	live CNN in browser Understanding CNNs through visualization cs231 on visualization CNNs Use keras to visualize which image features activate a feature map	11 finger-exercise CNN Exercise\| Solution 12 transfer learning Exercises \| Solution 13a optimized image Exercise\| Solution\| Notebook 13b adversarial example Exercises \| Solution
6	Recurent Neural Networks slides Recurrent Neural Networks Vanishing Gradient Problem LSTMs	Karpathy (May 2015) The unreasonable effectiveness of Recurrent Neural Networks (blog post) Colah (August 2015) Understanding LSTM Networks (blog post) DL-book chapter 10	14 Simple RNN: Exercises \| Notebook \| Solution Solution part a 15 Char RNN Exercise \| Notebook
7	Un- and Semi-supervised Learning I slides How to do unsupervised feature learning or representation learning? Autoencoder and denoising AE Use unsupervised learned features for pattern recognition Use unsupervised learned features for classifcation with few labeled instances	Bengio's Review on representation learning	16 FC and CNN Autoencoders: Exercises \| Solution 17 Using unsupervised learned featurs for classification: Exercises \| Notebook \| Solution
8	Un- and Semi-supervised Learning II slides Classical and denoising AE Classical and denoising variational AE Ladder network for un- and semi-supervised learning Spotlight talks Poster presentations	DL-book chapter 14 The ladder network presented by its inverntor H. Valpola	18 Using a Variational autoencoder to generate digit examples: Exercises \| Notebook \| Solution

dl_course

Deep Learning (CAS machine intelligence)

Other resources

Syllabus