Lee DongWon

LLM Inference Optimization on DGX Spark

Qwen2.5-7B-Instruct on NVIDIA DGX Spark, served with vLLM.

Speculative decoding with a 0.5B draft model cut time per output token by 73–75% at batch size 1, with accuracy unchanged or slightly better.

The project in 22 seconds. No sound.

Overview

I applied speculative decoding to Qwen2.5-7B-Instruct served with vLLM on an NVIDIA DGX Spark, using Qwen2.5-0.5B as the draft model, and measured what it does to latency and to accuracy. I also tested an FP8 KV cache and rejected it.

A draft model proposes five tokens; the target model accepts four and replaces one
Speculative decoding: the small model guesses ahead, the large model verifies in one pass (illustration).

Setup

Hardware
NVIDIA DGX Spark (GB10)
Serving
vLLM 0.30
Target model
Qwen2.5-7B-Instruct
Draft model
Qwen2.5-0.5B
Workload
Synthetic prompts at 512, 2k and 8k input tokens, 128 output tokens, batch sizes 1 to 32, seed 0

Results

  • Latency. Time per output token (TPOT) fell 73–75% on batch-1 synthetic benchmarks. End-to-end latency fell 71% at 512-token input.
  • Accuracy. MMLU-Pro was unchanged at 42.1% (1000-question subset). HumanEval was 83.5% against an 82.3% baseline.
  • Where it fades. The gain shrinks as batch size and context grow; at batch 32 with 8k input the cell is KV-capacity-limited.
Heatmap of TPOT reduction by batch size and input length, from 74.7% at batch 1 to 15.4% at batch 32 with 8k input
TPOT reduction against the baseline, by batch size and input length.
Summary figure: TPOT by batch size and input length for baseline, FP8 KV, speculative decoding and both; TPOT reduction heatmap; modelled decode bandwidth; accuracy on MMLU-Pro and HumanEval
Full measurements: TPOT by batch size, reduction heatmap, modelled decode bandwidth, and accuracy.

What I rejected

I evaluated an uncalibrated FP8 KV cache as a second optimization. Its accuracy collapsed (MMLU-Pro 26.8, HumanEval 2.4 in the e4m3 setting), so I rejected it instead of reporting the latency alone.

Bars for MMLU-Pro and HumanEval: baseline and speculative decoding level, FP8 KV cache far lower
Accuracy for the baseline, speculative decoding, and the rejected FP8 KV cache.

More projects