LLM From Scratch

Linear algebra, calculus, probability, and information theory for deep learning

Perceptrons to deep networks, backpropagation, activation functions, optimization

RNNs, LSTMs, GRUs, seq2seq, and the attention revolution

Self-attention, multi-head attention, positional encoding, encoder-decoder

BPE, WordPiece, Unigram, SentencePiece, multilingual tokenization

Autoregressive models, masked LMs, training objectives, perplexity

Distributed training, data/tensor/pipeline parallelism, ZeRO, mixed precision

Kaplan/Chinchilla laws, compute-optimal training, data scaling, inference scaling

MDPs, value functions, Bellman equations, policy gradient, REINFORCE

TRPO, PPO, clipping, GAE, GRPO, reward shaping

Reward modeling, the RLHF pipeline, KL penalties, InstructGPT, process rewards

DPO derivation, GRPO, Constitutional AI, RLAIF, KTO, IPO

GPT-1 through GPT-4, architectural evolution, scaling milestones

LLaMA 1-3, Mistral, Qwen, DeepSeek, the open-source ecosystem

GPT-4, Claude, Gemini, o-series reasoning models, DeepSeek V3/R1

Flash Attention, GQA, MQA, KV cache, PagedAttention, linear attention

INT8/INT4, GPTQ, AWQ, GGUF, pruning, distillation, speculative decoding

Sparse MoE, routing algorithms, Mixtral, Switch Transformer, load balancing

Full fine-tuning, LoRA, QLoRA, adapters, prompt tuning, PEFT methods

Vision-language models, CLIP, LLaVA, audio, video, Sora

CoT prompting, reasoning models (o1, R1), test-time compute, verification

ReAct, RAG, multi-agent systems, MCP, computer use, memory systems

Mechanistic interpretability, probing, alignment, red-teaming, governance

Catastrophic forgetting, EWC, progressive networks, replay methods

Knowledge editing, model merging, online learning, temporal adaptation

Web crawling, deduplication, PII scrubbing, quality filtering, decontamination

KV cache, PagedAttention, speculative decoding, vLLM, production deployment

MMLU, Chatbot Arena, LLM-as-judge, safety evaluation, leaderboards

Few-shot, chain-of-thought, structured prompting, system prompts, best practices

Self-Instruct, distillation, RLAIF, self-play, model collapse, data augmentation