Skip to main content
GACS will never ask for your seed phrase, private keys, or payment. free and ad-free.
GACS.app — Global Anti-Crime & Safety
Sign up / Log in
Back to lessonLab · Module 8 / 12
Lab

RLHF & Alignment

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

  1. Open a fresh terminal in a new folder, e.g. mkdir lab-8 && cd lab-8.
  2. Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
  3. Install the dependencies noted on each starter snippet (the small pip install … line above the code).
  4. Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · DPO — Direct Preference Optimization
pip install trl
save as: 08_01_lab.py
python
from trl import DPOTrainer, DPOConfig
from datasets import load_dataset

ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:5%]")
trainer = DPOTrainer(
    model=model, ref_model=None,             # ref_model=None → use base + adapter
    args=DPOConfig(
        output_dir="dpo-out",
        beta=0.1, learning_rate=5e-7,
        per_device_train_batch_size=2,
        num_train_epochs=1, max_length=2048,
    ),
    train_dataset=ds, tokenizer=tok,
)
trainer.train()
Run with: python 08_01_lab.py
02 · Reward-model scoring loop (for PPO)
save as: 08_02_lab.py
python
def rm_score(prompt, response, rm, tok):
    x = tok(prompt + response, return_tensors="pt").to(rm.device)
    with torch.no_grad():
        return rm(**x).logits[0, 0].item()

print(rm_score("Q: 2+2?\n", "A: 4", rm, tok))
Run with: python 08_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

  1. 1
    DPO on UltraFeedback

    Do — Run DPO on 5k preference pairs starting from a LoRA-SFT checkpoint. Train for 1 epoch.

    Expected artifact — trainer logs showing reward margin increasing; win-rate vs base model >55% on 50 held-out prompts.

    Run free:
  2. 2
    Pairwise judge eval

    Do — Use gpt-5-mini as a judge to compare base vs DPO model on 30 prompts.

    Expected artifact — judge_results.json with per-prompt verdicts and an aggregate win-rate.

    Run free:
  3. 3
    Reward hacking probe

    Do — Construct 5 prompts where your DPO model now gives sycophantic or evasive answers. Document them.

    Expected artifact — reward_hacks.md listing prompts, model outputs, and what the model is gaming.

    Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.