Skip to main content
GACS will never ask for your seed phrase, private keys, or payment. free and ad-free.
GACS.app — Global Anti-Crime & Safety
Sign up / Log in
Back to lessonLab · Module 11 / 12
Lab

Scaling & Optimization

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

  1. Open a fresh terminal in a new folder, e.g. mkdir lab-11 && cd lab-11.
  2. Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
  3. Install the dependencies noted on each starter snippet (the small pip install … line above the code).
  4. Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · 4-bit quantization with bitsandbytes
save as: 11_01_lab.py
python
from transformers import BitsAndBytesConfig, AutoModelForCausalLM
import torch

bnb = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B", quantization_config=bnb, device_map="auto"
)
Run with: python 11_01_lab.py
02 · Speculative decoding with a draft model
save as: 11_02_lab.py
python
from transformers import AutoModelForCausalLM, AutoTokenizer
target = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.1-8B-Instruct", device_map="auto")
draft  = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3.2-1B-Instruct", device_map="auto")
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B-Instruct")
ids = tok("Explain speculative decoding:", return_tensors="pt").to(target.device)
out = target.generate(**ids, max_new_tokens=200, assistant_model=draft)
print(tok.decode(out[0], skip_special_tokens=True))
Run with: python 11_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

  1. 1
    Quantize and compare

    Do — Load the same model in fp16, int8, and nf4. Measure VRAM, latency, and MMLU score for each.

    Expected artifact — quant.csv with rows fp16/int8/nf4 and columns vram_gb, ms_per_token, mmlu.

    Run free:
  2. 2
    Speculative decoding speedup

    Do — Use a 1B draft model to accelerate an 8B target. Measure tokens/sec for both.

    Expected artifact — ≥1.5× speedup on greedy decoding; tokens/sec table in spec.md.

    Run free:
  3. 3
    Distill a small student

    Do — Distill an 8B teacher into a 1.5B student on 10k completions. Compare student-before vs student-after on 30 prompts.

    Expected artifact — Win-rate of student-after over student-before ≥60%, reported in distill.md.

    Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.