Transformer Architecture

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

Open a fresh terminal in a new folder, e.g. mkdir lab-5 && cd lab-5.
Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
Install the dependencies noted on each starter snippet (the small pip install … line above the code).
Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · A minimal decoder-only transformer block

save as: 05_01_lab.py

python

import torch, torch.nn as nn

class Block(nn.Module):
    def __init__(self, d=128, h=4, mlp=4):
        super().__init__()
        self.ln1, self.ln2 = nn.LayerNorm(d), nn.LayerNorm(d)
        self.attn = nn.MultiheadAttention(d, h, batch_first=True)
        self.mlp = nn.Sequential(nn.Linear(d, mlp*d), nn.GELU(), nn.Linear(mlp*d, d))
    def forward(self, x, mask):
        a, _ = self.attn(self.ln1(x), self.ln1(x), self.ln1(x), attn_mask=mask, need_weights=False)
        x = x + a
        return x + self.mlp(self.ln2(x))

T = 16
mask = torch.triu(torch.ones(T, T), diagonal=1).bool()
print(Block()(torch.randn(2, T, 128), mask).shape)

Run with: python 05_01_lab.py

02 · Rotary position embeddings (RoPE) in 20 lines

save as: 05_02_lab.py

python

import torch

def rope(x, base=10000):
    B, T, D = x.shape
    half = D // 2
    freqs = 1.0 / (base ** (torch.arange(0, half).float() / half))
    t = torch.arange(T).float()
    angles = torch.outer(t, freqs)            # (T, half)
    cos, sin = angles.cos(), angles.sin()
    x1, x2 = x[..., :half], x[..., half:]
    return torch.cat([x1*cos - x2*sin, x1*sin + x2*cos], dim=-1)

print(rope(torch.randn(1, 8, 16)).shape)

Run with: python 05_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

1
Minimal GPT (≤200 lines)
Do — Implement a decoder-only transformer (embedding + N blocks + LM head) and overfit it on a single sentence until loss < 0.05.
Expected artifact — Training log showing loss going from ~10 → <0.05, then a sample that reproduces the sentence verbatim.
Run free:
2
Compare PE schemes
Do — Swap learned positional embeddings, sinusoidal, and RoPE in your minimal GPT. Train each for the same steps.
Expected artifact — A table of final losses; RoPE should generalise best to longer sequences than seen in training.
Run free:
3
KV-cache sanity
Do — Add a KV cache to your model's generate(). Verify that token-by-token output matches a no-cache full forward.
Expected artifact — Generated text identical between cached and uncached modes; cached version ≥3× faster for 100 tokens.
Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.

NIST National Initiative for Cybersecurity Education (NICE)
US national framework for cybersecurity work roles and competencies.
Financial Action Task Force (FATF)
Global AML/CFT standards referenced by financial-crime curricula.
FBI Internet Crime Complaint Center — Annual Reports
Primary loss and typology data used in fraud-investigation training.
US Federal Trade Commission — Consumer Sentinel Data Book
Official complaint statistics for teaching fraud trend analysis.