openai/tiktoken
tiktoken is a fast BPE tokeniser for use with OpenAI's models.
Stars: 17,438Language: Python
Give AlbumentationsX a star on GitHub — it powers this leaderboard
Star on GitHubtiktoken is a fast BPE tokeniser for use with OpenAI's models.