Overview
I applied speculative decoding to Qwen2.5-7B-Instruct served with vLLM on an NVIDIA DGX Spark, using Qwen2.5-0.5B as the draft model, and measured what it does to latency and to accuracy. I also tested an FP8 KV cache and rejected it.

Setup
- Hardware
- NVIDIA DGX Spark (GB10)
- Serving
- vLLM 0.30
- Target model
- Qwen2.5-7B-Instruct
- Draft model
- Qwen2.5-0.5B
- Workload
- Synthetic prompts at 512, 2k and 8k input tokens, 128 output tokens, batch sizes 1 to 32, seed 0
Results
- Latency. Time per output token (TPOT) fell 73–75% on batch-1 synthetic benchmarks. End-to-end latency fell 71% at 512-token input.
- Accuracy. MMLU-Pro was unchanged at 42.1% (1000-question subset). HumanEval was 83.5% against an 82.3% baseline.
- Where it fades. The gain shrinks as batch size and context grow; at batch 32 with 8k input the cell is KV-capacity-limited.


What I rejected
I evaluated an uncalibrated FP8 KV cache as a second optimization. Its accuracy collapsed (MMLU-Pro 26.8, HumanEval 2.4 in the e4m3 setting), so I rejected it instead of reporting the latency alone.
