Foundation of neural language modelin

instead of using n-gram, in order to predict the next word (based on probabilities), they convert each sentence into a semantic vector --> show similarity --> understand language patterns

Introduced Residual Networks (ResNet) & solved the "vanishing gradient" problem

nstead of learning H(x), learn F(x) = H(x) - x, then add x back (diff bt layers) ---> easier

New paradigm - wanted to explore

For huge prompts, they used Python to store the input in a variable, then wrote code to process it in smaller chunks and recursively called the model on each chunk to solve the query. (A --> call ---> B ---> call and so on)

Transformer architecture could beat standard CNNs at image recognition by splitting images into "patches" (tokens) ---> ViT

chop images into patches, treat them like words, and throw a standard Transformer at it. 1) At small compute budgets, ResNets and hybrids slightly edge out ViT. 2)As compute increases, ViT overtakes them all

Training stability

normalize the train data in mini batch --> cal mean and var of each batch ---> xi hat = (xi - mean) / sqrt(var**2 + ϵ) --> yi = \gamma \hat{xi} + \beta

Introduced CLIP and the backbone for modern zero-shot image classification and multimodal understanding.

ViTs are limited by fixed categories and expensive labels. Solution : Train on 400 million image-text pairs from the internet with a simple contrastive objective – make matching pairs close, non-matching pairs far ---> CLIP : A Zero-shot model that matches supervised ResNet-50 on ImageNet & way more robust

Adaptive learning rates

θ = θ − (momentum 1 /sqrt(momentum 2)) which mom 1 = avg & mom 2 = var ​

Simplifies self-supervised contrastive learning, showing how heavy data augmentation combined with a contrastive loss can learn powerful visual representations without any labels.

Different from other models (supervised learning on ImageNet) : data augmentation + non-linear head at the end of network + loss function

Learned embeddings — foundation of all modern NLP

vector("King")−vector("Man")+vector("Woman")≈vector("Queen")

Shows standard CNNs (like ResNet) learn to classify images based on local textures rather than global shapes.

ImageNet-trained CNNs are texture machines != humans --> Train on Stylized-ImageNet where textures are uninformative ( bias towards shapes) ---> more robust, better at detection, and match human behavior

Where the attention mechanism was born

1) English sentence → Encoder → ONE vector → Decoder → French sentence 2) The alignment model : score = v * tanh(W * decoder_state + U * encoder_state)

RandAugment is a highly practical, state-of-the-art technique for automating data augmentation strategies without needing to manually tune endless augmentation parameters.

AutoAugment (RL) and friends required expensive separate search on proxy tasks ---> Solution : Reduce the entire augmentation policy to 2 parameters – N (# of transforms) and M (global magnitude). Randomly select transforms, apply with same magnitude.

it fundamentally shifted the paradigm of computer vision from focusing on algorithms to focusing on data scale and quality

Data + Compute + Deep Networks defeated handcrafted computer vision features.

Generative modeling before diffusion took over

min​ G max D ​V(D,G) D tries to maximize its ability to detect fake samples. G tries to minimize D’s ability to detect them. V(D,G)=Ex∼pdata​​[logD(x)]+Ez∼pz​​[log(1−D(G(z)))] If image is real → D should say 1 If image is fake → D should say 0 G wants D to be wrong

Pioneered the shift from manually designed data augmentation to automated, data-driven approaches in computer vision (RL).

AutoAugment frames augmentation design as a search problem ---> The Search Space: A Policy of Sub-Policies : A policy consists of 5 sub-policies. For each image in a mini-batch, one sub-policy is chosen at random and applied. Each sub-policy has 2 operations (applied sequentially) ---> They use a RNN controller that predicts augmentation policies. The controller has 30 softmax predictions. 1. Train a "child model" (small neural network) on the target dataset using this augmentation policy 2. Evaluate the child model on a validation set to get accuracy 3. Use that accuracy as a reward signal to update the controller via Proximal Policy Optimization (PPO) Insane part : these policies transfer to other datasets and architectures!

THE paper. Start of modern AI.

Attention(Q,K,V) = softmax(QK^T / √d_k) V PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

Bridged the performance gap between self-supervised learning (SSL) and supervised learning in computer vision + InfoNCE Loss

1.Original image ↓ Two augmented versions ↓ 2.image → neural network → vector (representation) ↓ 3.Image ↓ Encoder (ResNet) ↓ Projection head (MLP) ↓ Representation used for training ↓ Contrastive Loss

Bidirectional pretraining revolution

The sun rises in the [MASK]. ↓ 1.Masked LM: predict randomly masked words using FULL context 2.Next Sentence Prediction: predict if two sentences are consecutive

Scale works + "too dangerous to release"

huge data + huge model → learn many tasks automatically

Triplet Loss

A triplet contains: 1. Anchor face 2. Positive face (same person) 3. Negative face (different person) Goal : distance(anchor, positive) < distance(anchor, negative)

Few-shot learning and emergent abilities + GPT-3

Big language model + examples in prompt = new task solved

The contrastive loss + Siamese Networks

Instead of classifying data directly, learn a space where similarity becomes distance.

The empirical case for bigger = better

If we keep scaling models and data, performance keeps improving smoothly. If you have fixed compute, you should not: make the model extremely huge or use extremely huge data Instead: You should balance them.

Variational Autoencoders (VAEs)

A generative model: pθ(z) * pθ(x | z) (prior + decoder) ↓ Problem is that we don't have true posterior pθ(x | z) ↓ Approximate the marginal likelihood pθ(x) ---> qϕ(z∣x) ↓ Use the reparameterization trick to get low-variance gradients ↓ Optimize the variational lower bound (ELBO) with SGD

Balance model size with data — changed how every lab trains

Training tokens ≈ 20 × number of parameters

intro to diffusion Models

Real Image ↓ Eventually → pure random noise ↓ x0 → x1 → x2 → ... → xT ↓ xT → xT-1 → xT-2 → ... → x0 ↓ pθ​(xt−1​∣xt​) ↓ Given a noisy sample, predict the less noisy version.

connected diffusion models with denoising score matching

We define a Markov chain: Forward Process ↓ q(xt​∣xt−1​) = N(xt​;1−βt​ ​xt−1​,βt​I) ↓ Add a little Gaussian noise xT​∼N(0,I) ↓ Reverse process (learned) : we want : pθ​(xt−1​∣xt​) --> ϵθ​(xt​,t) ↓ Loss=E[∣∣ϵ−ϵθ​(xt​,t)∣∣2] ↓ Given a noisy sample, predict the less noisy version."

Scale without proportional compute cost

In transformer layer, instead of FeedForward(x), we can use: ↓ Expert_i(x) (only one selected) ↓ chosen by : router(x) → probabilities over experts ↓ Then : pick argmax → choose one expert

How diffusion became practical — behind the images you generate

1.Train an autoencoder to compress images into a small, meaningful latent space — do this once. ↓ 2.Train a diffusion model (U-Net with cross-attention) to denoise in that latent space, conditioned on text or other signals. ↓ 3. At inference: sample noise in latent space, run the U-Net iteratively to denoise it, then decode back to pixels.

introduced a highly efficient encoder-decoder architecture with skip connections, enabling accurate image segmentation—particularly in medical imaging

Contracting path (encoder) ↓ bottleneck Expanding path (decoder) ↑

Why ChatGPT behaves like ChatGPT

maxE[rϕ​(x,y)] using PPO↓ Objective : Reward−β⋅KL(policy∣∣basemodel) They don’t let the model drift too far.

Foundation of reasoning models (o1 / DeepSeek-R1)

transforming inference into a structured latent-variable generation process ↓ p(y∣x)=z∑​p(y∣z,x)p(z∣x)

Fine-tune billion-parameter models on one GPU

Φ←Φ0​+ΔΦ (ΔW = fine tuned mat) ↓ ΔW=BA ↓ W=W0​+BA --> h=W0​x+BAx ---> h = w0x+Bz ↓ LoRA learns: “Which r-dimensional subspace of the input-output mapping should be modified?” ↓ FIrst : A∼N(0,2) and 𝐵=0 ---> ΔW=0 ↓ ΔWx←rα​BAx --> rank(ΔW)≪d

Why you can have 100K+ context windows today

Standard attention: Compute → write → read → compute → write → read ↓ FlashAttention: Load once → compute everything → write once ↓ To do this, they calculate softmax in another way : normal softmax : softmax(xi​)=∑j​exj​exi​​ ↓ Decomposable Softmax : m(x)=max(x),ℓ(x)=i∑​exi​−m(x) ↓ Then for two blocks 𝑥(1),𝑥(2) m=max(m1​,m2​) & ℓ=em1​−mℓ1​+em2​−mℓ2​ ↓ It is not straightforward, ask chatgpt fot proof

Simpler alternative to RLHF — most open-source models use this now

Kicked off the open-source LLM movement

How most production LLM apps work today

Multimodal LLMs — simple and open

What if transformers aren't the final form?

Cheap frontier models + reasoning via RL

Foundation model for vision — capstone for your vision interest

Chinchilla law

LIME

2) min g𝛜G L(f,g,𝜋x0)+𝛀(g) 3) 𝜋x'(z') = exp(-D(x',z)**2 / 𝝈**2) 4) L(f,g,𝜋x0) = 𝛴 𝜋x'(z') (fx(z0) g(z0))**2

SHAP

ϕi(f,x)=∑z′⊆x′ ∣z′∣!(M−∣z′∣−1)! / M! [fx(z′)−fx(z′∖i)] for each subset z' of features : 1) Calculate the weight |z'|!(M - |z'| - 1)!/M! 2) Calculate the marginal contribution [f_x(z') - f_x(z' \ i)] 3)Sum it all up Gives the fair contribution of feature i to the prediction