MathematicalEngineeringDeepLearning

Material for The Mathematical Engineering of Deep Learning. See the actual book content on deeplearningmath.org or (when it is out) purchase the book from CRC press.

This repository contains general supporting material for the book.

Below is a detailed list of the source code used for creating figures and tables in the book. We use Julia, Python, or R and the code is sometimes in stand alone files, sometimes in Jupyter notebooks, sometimes as R Markdown, and sometimes in Google Colab. Many of our static illustrations were created using TikZ by Ajay Hemanth and Vishnu Prasath with the source of their illustrations also available so you can adapt it for purposes.

Chapter 1

Figure	Topic	Source Code
1.1	Fast.ai example	Python Google Colab
1.3	Architectures	TikZ(a), TikZ(b), TikZ(c), TikZ(d), TikZ(e), TikZ(f)
1.4	Neurons	TikZ(b), TikZ(d)
1.5	Data on earth	Julia

Chapter 2

Figure	Topic	Source Code
2.1	Supervised Learning	TikZ
2.2	Unsupervised Learning	TikZ
2.3	Simple regression	R
2.4	Breast Cancer ROC curves	R
2.5	Least Squares	TikZ
2.6	Loss functions	Julia
Table 2.1	Linear MNIST classification	Julia
2.7	Gradient Descent Learning Rate	Python
2.8	Loss Landscape	R
2.9	Generalization and Training	TikZ or Julia
2.10	Polynomial fit	R
2.11	K-fold cross validation	TikZ
2.12	K-means clustering	R
2.13	K-means image segmentation	R
2.14	Breast Cancer PCA	R
2.15	SVD Compression	Julia

Chapter 3 (Figures under construction)

Figure	Topic	Source Code
3.1	Logistic regression model curves	R(a) R(b)
3.2	Linear decision boundary for logistic regression	R
3.3	Components of an artificial neuron	TikZ
3.4	Loss landscape of MSE vs. CE on logistic regression	Python
3.5	Evolution of gradient descent learning in logistic regression	R
3.6	Shallow multi-output neural network with softmax	TikZ
3.7	Multinomial regression for classification	R
Table 3.1	Different approaches for creating an MNIST digit classifier.	Julia
3.8	Feature engineering in simple logistic regression	R
3.9	Non-linear classification decision boundaries with feature engineering in logistic regression	R
3.10	Non-linear classification decision boundaries with feature engineering in multinomial regression	R
3.11	Single hidden layer autoencoder	TikZ
3.12	Autoencoder projections of MNIST including using PCA	R TikZ
3.13	Manifolds and autoencoders	R TikZ
3.14	MNIST using autoencoders	R
3.15	Denoising autoencoder	TikZ
3.16	Interpolations with autoencoders	Julia

Chapter 4 (Figures)

Figure	Topic	Source Code
4.1	Convexity and local/global extrema	Python
4.2	Gradient descent with a time dependent learning rate	Python
4.3	Stochastic gradient descent	Python
4.4	Early stopping in deep learning	Julia
4.5	Non-convex loss landscapes	Python
4.6	Momentum enhancing gradient descent	Python
4.7	The computational graph for automatic differentiation	TikZ
4.8	Line search concepts	Python
4.9	The zig-zagging property of line search (zoom in)	Python
4.10	Newton's method in one dimension	Python

Chapter 5 (Figures under construction)

Figure	Topic	Source Code
5.1	Fully Connected Feedforward Neural Networks	TikZ(a), TikZ(b)
5.2	Arbitrary function approximation with neural nets	Julia
5.3	Binary classification with increasing depth	R
5.4	A continuous multiplication gate with 4 hidden units	TikZ
5.5	Several common scalar activation functions	Julia
5.6	Flow of information in general back propagation	TikZ
5.7	Simple neural network hypothetical example	TikZ
5.8	Flow of information in standard neural network back propagation	TikZ
5.9	Computational graph for batch normalization	TikZ
5.10	The effect of dropout	TikZ

Chapter 6 (Figures under construction)

Figure	Topic	Source Code
6.1	TBD

Chapter 7 (Figures under construction)

Figure	Topic	Source Code
7.1	TBD

Chapter 8 (Figures under construction)

Figure	Topic	Source Code
8.1	TBD

Chapter 9 (Figures under construction)

Figure	Topic	Source Code
9.1	TBD

Chapter 10 (Figures under construction)

Figure	Topic	Source Code
10.1	TBD