1 result · Clear filters · Feed
-
Medium 6.5
tokenizers BpeBuilder Buffer Overflow via merge token
tokenizers (Hugging Face) is affected by an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs). When loading a tokenizer.json via Tokenizer::from_file/from_str, the builder sizes a scratch buffer to the longest vocabulary ke…