MIT 6.390 Intro to Machine Learning: 6 Neural Networks (Repost)

  6.390 - Intro to Machine Learning

Reposted from Chapter 6: Neural Networks of the MIT 6.390 Intro to Machine Learning course notes, © Massachusetts Institute of Technology and the 6.390 (formerly 6.036) course staff, licensed under CC BY-NC-SA 4.0. Changes made on 2026-11-20: converted from the course website to Markdown; side notes and boxed notes shown as quotes; links to other chapters point to the original site. This post is shared under the same license.

You’ve probably been hearing a lot about “neural networks.” Now that we have several useful machine-learning concepts (hypothesis classes, classification, regression, gradient descent, regularization, etc.), we are well equipped to understand neural networks in detail.

Read more

MIT 6.390 Intro to Machine Learning: 7 Convolutional Neural Networks (Repost)

  6.390 - Intro to Machine Learning

Reposted from Chapter 7: Convolutional Neural Networks of the MIT 6.390 Intro to Machine Learning course notes, © Massachusetts Institute of Technology and the 6.390 (formerly 6.036) course staff, licensed under CC BY-NC-SA 4.0. Changes made on 2026-11-20: converted from the course website to Markdown; side notes and boxed notes shown as quotes; links to other chapters point to the original site; a figure from MathWorks left out, since the course license doesn’t cover it. This post is shared under the same license.

So far, we have studied what are called fully connected neural networks, in which all of the units in one layer are connected to all of the units in the next layer. This is a good arrangement when we don’t know anything about the kind of mapping from inputs to outputs that we will be asking the network to approximate. However, if we do know something about our problem, it is better to build that knowledge into the structure of the neural network. Doing so can save computation time and significantly reduce the amount of training data required to arrive at a solution that generalizes robustly.

Read more

MIT 6.390 Intro to Machine Learning: 8 Representation Learning (Repost)

  6.390 - Intro to Machine Learning

Reposted from Chapter 8: Representation Learning of the MIT 6.390 Intro to Machine Learning course notes, © Massachusetts Institute of Technology and the 6.390 (formerly 6.036) course staff, licensed under CC BY-NC-SA 4.0. Changes made on 2026-11-20: converted from the course website to Markdown; side notes and boxed notes shown as quotes; links to other chapters point to the original site. This post is shared under the same license.

In previous chapters, we have largely focused on supervised learning, using training samples that have both features/inputs and corresponding labels/outputs, to learn models that predict labels for new data. A recurring theme, however, has been the importance of how data is represented. In Chapter 5, we saw that choosing the right feature transformation can make the difference between a model that works and one that doesn’t. In Chapter 7, we saw that convolutional neural networks learn hierarchical features automatically from data. This chapter steps back to ask a broader question: can we learn good representations of data directly, and why would we want to?

Read more

MIT 6.390 Intro to Machine Learning: 9 Transformers (Repost)

  6.390 - Intro to Machine Learning

Reposted from Chapter 9: Transformers of the MIT 6.390 Intro to Machine Learning course notes, © Massachusetts Institute of Technology and the 6.390 (formerly 6.036) course staff, licensed under CC BY-NC-SA 4.0. Changes made on 2026-11-20: converted from the course website to Markdown; side notes and boxed notes shown as quotes; links to other chapters point to the original site. This post is shared under the same license.

Transformers are a very recent family of architectures that were originally introduced in the field of natural language processing (NLP) in 2017, as an approach to process and understand human language. Since then, they have revolutionized not only NLP but also other domains such as image processing and multi-modal generative AI. Their scalability and parallelizability have made them the backbone of large-scale foundation models, such as GPT, BERT, and Vision Transformers (ViT), powering many state-of-the-art applications.

Read more

MIT 6.390 Intro to Machine Learning: Appendix B Optimizing Neural Networks (Repost)

  6.390 - Intro to Machine Learning

Reposted from Appendix B: Optimizing Neural Networks of the MIT 6.390 Intro to Machine Learning course notes, © Massachusetts Institute of Technology and the 6.390 (formerly 6.036) course staff, licensed under CC BY-NC-SA 4.0. Changes made on 2026-11-20: converted from the course website to Markdown; side notes and boxed notes shown as quotes; links to other chapters point to the original site. This post is shared under the same license.

B.1 Momentum and adaptive learning rates

B.1.1 Running averages

We’ll start by looking at the notion of a running average. It’s a computational strategy for estimating a possibly weighted average of a sequence of data. Let our data sequence be \(c_1, c_2, \ldots\); then we define a sequence of running average values, \(C_0, C_1, C_2, \ldots\) using the equations

Read more
Deep Neural Networks from Scratch