Skip to main content
GACS will never ask for your seed phrase, private keys, or payment. free and ad-free.
GACS.app — Global Anti-Crime & Safety
Sign up / Log in
Back to lessonLab · Module 9 / 12
Lab

Evaluation & Benchmarking

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

  1. Open a fresh terminal in a new folder, e.g. mkdir lab-9 && cd lab-9.
  2. Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
  3. Install the dependencies noted on each starter snippet (the small pip install … line above the code).
  4. Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · Run MMLU + HumanEval with lm-eval-harness
save as: 09_01_lab.sh
bash
pip install lm-eval
lm_eval --model hf \
        --model_args pretrained=Qwen/Qwen2.5-1.5B \
        --tasks mmlu,hellaswag,gsm8k \
        --batch_size auto \
        --output_path results/
Run with: bash 09_01_lab.sh
02 · LLM-as-a-judge with a rubric
save as: 09_02_lab.py
python
JUDGE = """You are a strict grader. Score the answer 1-5 on:
correctness, completeness, style. Return JSON: {"correctness":int,"completeness":int,"style":int}.
Question: {q}
Answer: {a}"""

def judge(q, a, client):
    msg = client.chat.completions.create(
        model="gpt-5-mini", response_format={"type":"json_object"},
        messages=[{"role":"user","content": JUDGE.format(q=q, a=a)}])
    return msg.choices[0].message.content
Run with: python 09_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

  1. 1
    Run lm-eval-harness

    Do — Evaluate one open model on mmlu, hellaswag, and gsm8k. Save results/.

    Expected artifact — A results.json with three task scores and a README.md summarising them.

    Run free:
  2. 2
    Custom eval set

    Do — Hand-write 25 domain-specific test cases (input, expected substring). Build a pass/fail harness.

    Expected artifact — evals/run.py prints 'PASS X/25' and writes evals/report.html.

    Run free:
  3. 3
    Judge-model calibration

    Do — Have a human grade 20 outputs, then have an LLM judge the same. Compute agreement.

    Expected artifact — Cohen's κ ≥ 0.6 between human and LLM judge.

    Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.