Memory-Efficient KV Cache Quantization for Reasoning LLMs (EMIL Lab)

Overview

Studying memory-efficient KV cache quantization for reasoning LLMs (in progress).

Large Reasoning Models (LRMs) solve hard problems by writing a long chain of thought before the final answer, often hundreds to thousands of tokens. Every generated token adds keys and values to the KV cache, so for these models the cache grows with the length of the reasoning itself, not just the prompt. KV cache memory then limits how long a model can think, how many requests fit on one GPU, and whether it can run on smaller devices.

Status: in progress. Details will be shared after publication.

Motivation

• In long-context inference, the KV cache becomes the main memory cost, and it grows linearly with sequence length and batch size [2].
• Reasoning models generate long outputs during decoding [3], so the cache keeps growing token by token during generation.
• Quantization lowers the bytes per stored key and value while keeping every token, which makes it complementary to methods that drop tokens [1, 2].

Problem

How can the KV cache of a reasoning LLM be stored at low precision during long chain-of-thought decoding, while keeping its final-answer accuracy?

• Errors from quantized keys and values can build up over a long reasoning trace.
• One wrong intermediate step can change the final answer, so accuracy has to be measured on the final answer, not only on perplexity.

Research Direction

• Quantize the KV cache for long chain-of-thought decoding in reasoning LLMs.
• Keep reasoning accuracy close to the full-precision KV cache while reducing KV memory.
• Builds on public KV cache quantization work [1, 2].

The specific method, models, benchmarks, and results are not public yet.

My Role

• Undergraduate Research Intern, EMIL Lab, Kyung Hee University (advisor: Prof. Sungha Choi), Jul. 2026 – Present.
• Studying memory-efficient KV cache quantization for reasoning LLMs.
• Specific contributions will be listed after publication.

Tech: PyTorch, KV-cache quantization.

Status

In progress. Details will be shared after publication.

References

Background work only; not results from this project.

1. Z. Liu, J. Yuan, H. Jin, S. Zhong, Z. Xu, V. Braverman, B. Chen, X. Hu. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML 2024.
2. C. Hooper, S. Kim, H. Mohammadzadeh, M. W. Mahoney, Y. S. Shao, K. Keutzer, A. Gholami. KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization. NeurIPS 2024.
3. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948.