What a token is
Language models don't read letters or words. They read tokens: pieces of text from a fixed vocabulary, often a whole short word, sometimes part of a long one, a space plus a word, or a single punctuation mark. In English a token is about four characters, or three quarters of a word, but code, numbers and other languages split differently. That is why counting characters only gives a rough guess, and why this tool runs the real tokenizer.
The token map shows it plainly. Paste a sentence and every block is one token, with spaces shown as dots, so you can see where the model's view of your text breaks.
Which encoding to pick
OpenAI's models use two encodings, and both are exact here:
- o200k_base: GPT-5, GPT-4.1, GPT-4o and the o-series (o1, o3, o4-mini). It has a 200,000 token vocabulary and is much more efficient for non-English text and code.
- cl100k_base: GPT-4, GPT-4 Turbo, GPT-3.5 Turbo and the older embedding models.
Claude, Gemini and Llama use their own tokenizers, which aren't published for the browser. For those models the tool counts with o200k_base and labels the result as an estimate: a good guide to the size of a prompt, not an exact figure.
Will my prompt fit the context window?
A context window is the most tokens a model can handle in one request, and it has to hold your input and the model's answer together. Pick a model, set the room you want to keep for the answer, and the bar shows how much of the window your text takes and how much is left. If the text runs over, the tool tells you by how many tokens.
Leave real room for the answer. A summary needs a few hundred tokens, a long report or a block of code can need several thousand, and a request that fills the window leaves the model nothing to reply with.
Splitting a document into chunks
Retrieval pipelines, embeddings and long document workflows need text in pieces of a set size. Set the chunk size and the overlap in tokens, and the tool cuts the text exactly where the tokens fall, with each chunk sharing its overlap with the next so no sentence loses its context. Download the chunks as a zip of text files, or as one JSON file with each chunk's index, its first token and its text.
Private, and fast on long files
The tokenizer runs in a Web Worker inside your browser, so a whole book counts without freezing the page, and none of your text is ever uploaded. The encoding files load once, the first time you count, and your browser keeps them. I built this alongside other free tools for teams building with AI, like the invisible character remover for cleaning model output.