Writing

Mar 24, 2026·102 min read·foundation

Floating Point: Designing a Number System from 32 Bits

Start with a blank 32-bit register and one question: how do you encode real numbers in binary? Each piece of IEEE 754 float32, the sign bit, biased exponent, implicit leading 1, subnormals, arrives as the only reasonable fix to a concrete problem. Then connect representation to arithmetic: the alignment step that erases small terms during addition, rounding that breaks associativity, catastrophic cancellation, and why 0.1 + 0.2 returns 0.30000000000000004.

Feb 26, 2026·208 min read·foundation

LLM Inference: From Black Box to Production

A ground-up explanation of LLM inference, from black box to production optimizations. Covers tokenization, embeddings, attention, KV cache, memory bottlenecks, batching, PagedAttention, and quantization, using TinyLlama 1.1B as the running example.

Oct 30, 2025·115 min read·advanced

Backpropagation Part 3: Systems, Stability, Interpretability, Frontiers

Theory assumes infinite precision; hardware delivers float16. Bridge the gap between mathematical backprop and production systems. In this post, we cover a lot of "practical" ground from PyTorch's tape to mixed precision training, from numerical disasters to systematic testing, from gradient monitoring to interpretability. What breaks, why, and how to fix it.

Oct 27, 2025·142 min read·intermediate

Backpropagation Part 2: Patterns, Architectures, and Training

Every gradient rule, from convolutions to attention, follows one pattern: the vector-Jacobian product. See past the memorized formulas to the unifying abstraction, understand how residuals and normalization tame deep networks, and learn why modern architectures are really just careful gradient engineering.

Oct 24, 2025·172 min read·foundation

Backpropagation Part 1: From Graphs to a Working MLP

Backprop computes a million gradients for the price of two forward passes. From computational graphs to adjoints, from chain rule to a working neural network, this is the algorithm that made deep learning possible; and is demystified here step by step.

Oct 8, 2025·131 min read·foundation

Multi-Layer Perceptrons: How Neural Networks Bend Space to See

Why does a 3-layer network solve problems a 1000-neuron single layer cannot? Understanding forward propagation, the exponential efficiency of depth, and how simple operations compose into hierarchical reasoning.

Sep 30, 2025·110 min read·foundation

Gradient Descent: Theory, Mathematics, and Implementation

Why does walking downhill in parameter space solve everything from linear regression to GPT? A rigorous treatment of gradient descent: convergence theory, variants and the challenges of real-world optimization.

Sep 21, 2025·55 min read·foundation

The Perceptron: A Deep Dive

Understanding the fundamental building block of neural networks through intuition, mathematics, and implementation.

Sep 8, 2025·14 min read·foundation

Model Context Protocol (MCP), Part 1 — Why was this inevitable?

A narrative of the conditions, failed alternatives, and timing that made MCP the standard way to connect LLMs to real tools and data.

EarlierOct 2025Backpropagation Part 1: From Graphs to a Working MLPOct 2025Multi-Layer Perceptrons: How Neural Networks Bend Space to SeeSep 2025Gradient Descent: Theory, Mathematics, and ImplementationSep 2025The Perceptron: A Deep DiveSep 2025Model Context Protocol (MCP), Part 1 — Why was this inevitable?

Tools

Interactive Visualization

Neural Network Playground

Build, train, and visualize multi-layer perceptrons in real time. Experiment with architectures, datasets, and hyperparameters to develop intuition for how neural networks learn.

Interactive Visualization

Batch-Aware NVMe Schedule

A small visible-window KV-cache recall trace for seeing how FIFO leaves foreign work ahead of a request's tail, and how a request-aware dispatch heuristic changes that.

Interactive Visualization

Gridworld: Dynamic Prog

A 10×10 deterministic gridworld for watching Bellman backups and greedy policy improvement solve a known MDP. Pick a route template or click any cell to choose the destination, then scrub value iteration one sweep at a time.

Interactive Calculator

Vocab Size Calculator

Pick a preset or dial in V, d, L, T, B, and dtype to see the real tax of vocabulary on a transformer: embedding parameters, untied LM head, logits FLOPs per token, and KV-cache bytes.

Interactive Visualization

Tokenizer Playground

Pick from twenty-nine production tokenizers — tiktoken, Llama, Gemma, Mistral, Qwen, DeepSeek, Cohere, Phi, Yi — paste any text, and watch every model carve it into pieces. Click a token to walk its BPE merge tree.

Interactive Visualization

Tokenizers, Side by Side

Pin two to four tokenizers against the same input and watch where they carve it differently. Highlighted pills mark the spans only some encoders share; the bar at the top prints counts plus a spread so the multilingual penalty becomes a number.

Interactive Visualization

Train your own BPE

Paste a paragraph, dial in a vocabulary size, and watch Karpathy's minBPE mint one merge token at a time. Each learned token gets compared byte-for-byte against GPT-4's cl100k_base so you can see where your forty-byte trainer agrees with the production tokenizer — and where it goes its own way.