One interesting fact I stumbled upon recently is that GPT2Tokenizer library and ...

nextworddev on April 5, 2023 | parent | context | favorite | on: OpenAI Tokenizer

One interesting fact I stumbled upon recently is that GPT2Tokenizer library and Tiktoken library produces the same number of tokens for `text-davinci-003` model, despite GPT2Tokenizer being GPT2 and text-davinci-003 being GPT3.5.

For code, however, Tiktoken library and GPT2Tokenizer produce different tokenizations.

shevis on April 5, 2023 [–]

Key difference here is in tokenization encoders. The newer models make use of the `cl100k_base` encoding.