Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations

Brian Siyuan Zheng, Lee DongWon, Orevaoghene Ahia, Jonathan Hayase, Yejin Choi, Noah A. Smith
July, 2025
Paper BibTeX Code
tokenization
Lee DongWon
Lee DongWon

PhD student in natural language processing