Topics

optimization

5 posts

Feb 26, 2026·208 min read·foundation

LLM Inference: From Black Box to Production

A ground-up explanation of LLM inference, from black box to production optimizations. Covers tokenization, embeddings, attention, KV cache, memory bottlenecks, batching, PagedAttention, and quantization, using TinyLlama 1.1B as the running example.

Oct 27, 2025·112 min read·intermediate

Backpropagation Part 2: Patterns, Architectures, and Training

Every gradient rule, from convolutions to attention, follows one pattern: the vector-Jacobian product. See past the memorized formulas to the unifying abstraction, understand how residuals and normalization tame deep networks, and learn why modern architectures are really just careful gradient engineering.

Oct 24, 2025·138 min read·foundation

Backpropagation Part 1: From Graphs to a Working MLP

Backprop computes a million gradients for the price of two forward passes. From computational graphs to adjoints, from chain rule to a working neural network, this is the algorithm that made deep learning possible; and is demystified here step by step.

Oct 8, 2025·131 min read·foundation

Multi-Layer Perceptrons: How Neural Networks Bend Space to See

Why does a 3-layer network solve problems a 1000-neuron single layer cannot? Understanding forward propagation, the exponential efficiency of depth, and how simple operations compose into hierarchical reasoning.

Sep 30, 2025·110 min read·foundation

Gradient Descent: Theory, Mathematics, and Implementation

Why does walking downhill in parameter space solve everything from linear regression to GPT? A rigorous treatment of gradient descent: convergence theory, variants and the challenges of real-world optimization.