Foundations of LLMs

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

Open a fresh terminal in a new folder, e.g. mkdir lab-1 && cd lab-1.
Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
Install the dependencies noted on each starter snippet (the small pip install … line above the code).
Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · Hello, LLM — generate text with a small open model

pip install torch transformers accelerate

save as: 01_01_lab.py

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.float16, device_map="auto"
)

prompt = "Explain scaling laws in one paragraph."
inputs = tok(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=160, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

Run with: python 01_01_lab.py

02 · Count parameters and estimate FLOPs

save as: 01_02_lab.py

python

def count_params(m):
    return sum(p.numel() for p in m.parameters())

n = count_params(model)
print(f"params: {n/1e6:.1f}M")
# Chinchilla rule of thumb: optimal tokens ≈ 20 × params
print(f"chinchilla-optimal tokens ≈ {20*n/1e9:.1f}B")

Run with: python 01_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

1
Param-count audit
Do — Load three open models of different sizes (e.g. Qwen2.5-0.5B, Llama-3.2-1B, Mistral-7B). Print parameters in millions and the Chinchilla-optimal token budget for each.
Expected artifact — A 3-row table: model, params (M), optimal tokens ≈ 20 × params. The 7B row should report ~140B optimal tokens.
Hint: sum(p.numel() for p in model.parameters())
Run free:
2
Hallucination probe
Do — Ask a small model (≤1.5B) five questions whose answer is a specific year (e.g. 'Year RoPE was introduced?'). Record correct vs hallucinated answers.
Expected artifact — A markdown table with columns question / model answer / ground truth / correct?, and an accuracy score at the bottom.
Run free:
3
Scaling-law plot
Do — Train a tiny transformer (≤1M params) for 1000 steps on shakespeare.txt at three model sizes. Plot final loss vs params on a log-log axis.
Expected artifact — scaling.png showing a downward-sloping line — bigger model, lower loss.
Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.

NIST National Initiative for Cybersecurity Education (NICE)
US national framework for cybersecurity work roles and competencies.
Financial Action Task Force (FATF)
Global AML/CFT standards referenced by financial-crime curricula.
FBI Internet Crime Complaint Center — Annual Reports
Primary loss and typology data used in fraud-investigation training.
US Federal Trade Commission — Consumer Sentinel Data Book
Official complaint statistics for teaching fraud trend analysis.