Skip to main content
GACS will never ask for your seed phrase, private keys, or payment. free and ad-free.
GACS.app — Global Anti-Crime & Safety
Sign up / Log in
Back to lessonLab · Module 8 / 12
Lab

Evaluation & Benchmarking

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

  1. Open a fresh terminal in a new folder, e.g. mkdir lab-8 && cd lab-8.
  2. Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
  3. Install the dependencies noted on each starter snippet (the small pip install … line above the code).
  4. Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · Tiny agent eval harness
save as: 08_01_lab.py
python
CASES = [
    {"q":"Capital of France?", "must_include":["Paris"]},
    {"q":"What is 12*11?",    "must_include":["132"]},
]

def eval_agent(agent):
    results = []
    for c in CASES:
        out = agent(c["q"])
        ok = all(s.lower() in out.lower() for s in c["must_include"])
        results.append({"q":c["q"], "pass":ok, "out":out[:80]})
    acc = sum(r["pass"] for r in results) / len(results)
    return acc, results

print(eval_agent(lambda q: react(q)))
Run with: python 08_01_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

  1. 1
    Golden test set

    Do — Author 25 (question, must_include) pairs covering happy paths and edge cases.

    Expected artifact — tests/golden.jsonl with 25 entries.

    Run free:
  2. 2
    Eval harness

    Do — Run your agent against the golden set and print accuracy plus failures.

    Expected artifact — Console: 'PASS X/25'; failures saved to evals/failures.jsonl.

    Run free:
  3. 3
    Regression gate

    Do — Add a CI script that fails if accuracy drops more than 5 points vs the last commit.

    Expected artifact — .github/workflows/eval.yml that runs the harness and exits non-zero on regression.

    Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.