Skip to main content
GACS will never ask for your seed phrase, private keys, or payment. free and ad-free.
GACS.app — Global Anti-Crime & Safety
Sign up / Log in
Back to lessonLab · Module 10 / 12
Lab

Serving & Deployment

Hands-on lab with starter code you can copy + run, and guided steps to produce the artifact for this module.

Setup

  1. Open a fresh terminal in a new folder, e.g. mkdir lab-10 && cd lab-10.
  2. Create a Python 3.11 venv: python -m venv .venv && source .venv/bin/activate.
  3. Install the dependencies noted on each starter snippet (the small pip install … line above the code).
  4. Copy each starter into a file with the suggested name, then run it and capture the output.

Starter code

01 · Serve a model with vLLM
save as: 10_01_lab.sh
bash
pip install vllm
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
    --port 8000 --max-model-len 8192 \
    --gpu-memory-utilization 0.9 \
    --enable-prefix-caching
Run with: bash 10_01_lab.sh
02 · OpenAI-compatible client call
save as: 10_02_lab.py
python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
out = client.chat.completions.create(
    model="Qwen/Qwen2.5-1.5B-Instruct",
    messages=[{"role":"user","content":"What is KV cache?"}],
    max_tokens=200, temperature=0.2,
)
print(out.choices[0].message.content)
Run with: python 10_02_lab.py

Guided steps

Work through each step in order. Stop when your output matches the expected artifact — then push your work to your lab folder.

  1. 1
    Serve with vLLM

    Do — Launch vLLM with your merged model and hit /v1/chat/completions from a Python client.

    Expected artifact — 200 OK response; tokens/sec printed from a 100-token completion benchmark.

    Run free:
  2. 2
    Throughput benchmark

    Do — Send 64 concurrent requests and measure p50/p95 latency and total tokens/sec.

    Expected artifact — bench.csv with columns concurrency, p50, p95, tps; a 3-line conclusion in bench.md.

    Run free:
  3. 3
    Prefix-cache win

    Do — Compare throughput with and without --enable-prefix-caching for a workload that shares a long system prompt.

    Expected artifact — ≥2× tokens/sec improvement with prefix caching, documented in cache.md.

    Run free:

Authoritative sources

Independent primary sources used to check and corroborate the guidance on this page.