tokenization | Lee DongWon https://moovingsloth.github.io/tag/tokenization/ tokenization Wowchemy (https://wowchemy.com)en-usFri, 11 Jul 2025 00:00:00 +0000 https://moovingsloth.github.io/media/sloth-icon-512.png tokenization https://moovingsloth.github.io/tag/tokenization/ Sampling from Your Language Model One Byte at a Time https://moovingsloth.github.io/publication/byte_sampler/ Fri, 11 Jul 2025 00:00:00 +0000 https://moovingsloth.github.io/publication/byte_sampler/ Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations https://moovingsloth.github.io/publication/broken_tokens/ Thu, 10 Jul 2025 00:00:00 +0000 https://moovingsloth.github.io/publication/broken_tokens/ SuperBPE: Space Travel for Language Models https://moovingsloth.github.io/publication/superbpe/ Wed, 09 Jul 2025 00:00:00 +0000 https://moovingsloth.github.io/publication/superbpe/