Inside a 3,184-Parameter Language Model
A one-layer transformer you can take apart in the browser. Attention, the MLP, a logit lens, sampling and ablations, all computed live from weights trained for the purpose, and what those weights turned out to contain.