New architecture papers arrive every week, and most claims are made on hardware and at scales you will never reproduce. This course teaches you to understand the main ideas beyond the plain transformer, implement small versions of them yourself, and measure them fairly on an ordinary CPU, so you can decide what is worth using. You start by building your own bench and lab environment: CPU PyTorch, Hugging Face transformers and a pinned set of dependencies. Then you build a tiny decoder-only transformer from scratch, train it, and record its speed, memory and quality in a results table. One benchmark harness is reused by every architecture that follows, so every row in the table is comparable.
Recursion is covered in all three of its meanings. You build recursive networks over trees and a Tree-LSTM, then weight-tied looped transformers, and look at Universal Transformers, Mixture-of-Recursions, HRM and the Tiny Recursive Model. The Recursive Language Model section builds an RLM loop by hand: the long prompt lives as a variable in a Python REPL, and the root model calls sub-models on pieces of it. You run it against a local model served by Ollama, read the full trace, try the official library, and see where a small root model fails.
The middle of the course works through the main efficiency ideas. Sparse, sliding-window and linear attention are implemented and timed; FlashAttention, MQA, GQA and latent attention are explained with honest notes on what needs a GPU. You implement a selective state space layer, train a tiny Mamba-style model, and compare pretrained Mamba and Pythia on the same data. RWKV, RetNet, Gated DeltaNet and today’s hybrid stacks are explained, and an associative recall task shows why hybrids keep an attention layer.
