Skip to main content
GACS will never ask for your seed phrase, private keys, or payment. free and ad-free.
GACS.app — Global Anti-Crime & Safety
Sign up / Log in
Back to lessonLab · Module 2 / 12
Lab

Tokenization & Text Processing

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

  1. Open a fresh terminal in a new folder, e.g. mkdir lab-2 && cd lab-2.
  2. Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
  3. Install the dependencies noted on each starter snippet (the small pip install … line above the code).
  4. Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · Train a BPE tokenizer from scratch
pip install tokenizers
save as: 02_01_lab.py
python
from tokenizers import Tokenizer, models, trainers, pre_tokenizers

tok = Tokenizer(models.BPE(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
trainer = trainers.BpeTrainer(
    vocab_size=8000,
    special_tokens=["[PAD]", "[UNK]", "[BOS]", "[EOS]"],
)
tok.train(files=["corpus.txt"], trainer=trainer)
tok.save("bpe.json")

enc = tok.encode("Tokenization is the first layer of an LLM.")
print(enc.tokens)
print(enc.ids)
Run with: python 02_01_lab.py
02 · Compare tokenizers on the same string
save as: 02_02_lab.py
python
from transformers import AutoTokenizer

text = "def fibonacci(n): return n if n < 2 else fib(n-1)+fib(n-2)"
for model_id in ["gpt2", "meta-llama/Llama-3.2-1B", "Qwen/Qwen2.5-0.5B"]:
    t = AutoTokenizer.from_pretrained(model_id)
    print(f"{model_id:35s} -> {len(t.encode(text)):3d} tokens")
Run with: python 02_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

  1. 1
    BPE on a real corpus

    Do — Train a 4k-vocab byte-level BPE tokenizer on a 5MB text dump (Wikipedia or your own notes). Save to bpe.json.

    Expected artifact — A bpe.json file ~150KB. Encoding 'Tokenization is fun' returns ≤6 tokens.

    Run free:
  2. 2
    Cross-tokenizer cost comparison

    Do — Take a 2000-character Python file. Tokenize with GPT-2, Llama-3, Qwen2.5, and DeepSeek tokenizers. Compute tokens/char for each.

    Expected artifact — A table showing token counts; the modern code-tuned tokenizers should beat GPT-2 by 25–40%.

    Run free:
  3. 3
    Round-trip integrity

    Do — Encode and decode 100 random sentences from a dataset. Assert decode(encode(s)) == s for each.

    Expected artifact — Console prints '100/100 round-trip passed'.

    Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.