Hey. Thank you! However has the fix not been deployed yet? Still shows broken UT... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

egorfine on April 10, 2023 | parent | context | favorite | on: OpenAI Tokenizer

Hey. Thank you! However has the fix not been deployed yet? Still shows broken UTF-8.

> a single user-perceived character might span into multiple tokens

Is this the way it works as designed or is this a bug?

Consider applying for YC's Fall 2026 batch! Applications are open till July 27.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact