Counting tokens exactly for OpenAI, Claude and Gemini

How to count tokens exactly for OpenAI with tiktoken, why Claude and Gemini counts elsewhere are estimates, and the official count endpoints.

Summarize with
A type specimen card with baseline, x-height and descender guide lines and one curved brush stroke with a green anchor point

To count tokens, run your text through the model's own tokeniser. For OpenAI models you can do that exactly on your own machine, because OpenAI publishes its tokeniser, tiktoken, with the o200kbase and cl100kbase encodings. For Claude and Gemini, a third-party token counter gives an estimate, because both vendors document counting as an API call rather than a tokeniser you download; the exact count comes from that endpoint. The count decides what a request costs and whether it fits the model's context window. To count right now, paste your text into my free LLM token counter, which runs in your browser.

I'm a design engineer who builds AI features into products, so I count tokens most weeks: system prompts, retrieved documents, tool definitions and the room left for the answer. Here is how counting works, where counts go wrong and how to get each vendor's own number. The service behind this post is my design engineering work.

What a token is

Models don't read letters or words. They read tokens: pieces of text from a fixed vocabulary, often a short word, sometimes part of a long one, a word with its leading space, or one punctuation mark. OpenAI's tiktoken README describes the method, byte pair encoding, and notes that each token corresponds to about 4 bytes on average. Google's Gemini token guide gives a similar rule for its models: about 4 characters per token, and 100 tokens for about 60 to 80 English words.

Those averages break quickly. I ran a few strings through o200kbase and cl100kbase, the same encodings my tool uses:

TextCharacterso200k_basecl100k_base
"Hello, world!"1344
A 131 word support prompt (the tool's sample)762160162
A one line Arabic greeting321028
A three line TypeScript function692020
The number 12345678901044

"Hello, world!" splits into "Hello", ",", " world" and "!". English prose lands close to the rule of thumb. The Arabic line is where encodings differ most: cl100kbase needs nearly three times the tokens of o200kbase for the same sentence. If your product serves Arabic, Hindi or Bangla speakers, a character count tells you very little.

How to count OpenAI tokens exactly

OpenAI's models use published encodings, and tiktoken's own model table maps them:

  • o200k_base: GPT-5, GPT-4.5, GPT-4.1, GPT-4o and the reasoning models o1, o3 and o4-mini.
  • cl100k_base: GPT-4, GPT-3.5 Turbo and the text-embedding-3 and ada-002 embedding models.

In Python it takes three lines:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")
print(len(enc.encode("Your prompt here")))

In JavaScript, the js-tiktoken package does the same, and it's what my tool runs in the browser. tiktoken.encoding_for_model("gpt-4o") picks the encoding from a model name.

Two things make a raw text count lower than what the API reports.

Message formatting. A chat request wraps each message in role and boundary tokens. OpenAI's cookbook recipe on counting tokens with tiktoken adds 3 tokens per message for its listed models, 1 more when a message has a name, and 3 for priming the reply. The same page says to treat its function as an estimate, because the way messages are counted may change from model to model, and it is marked as archived.

Tools, images and files. OpenAI's token counting guide says local tokenisers work for plain text but not for images, files or tool definitions, which add tokens that are hard to count locally. For a full request, it offers POST /v1/responses/input_tokens, which takes the same input as the Responses API and returns input_tokens.

One more caution. OpenAI's model list now leads with GPT-6 models, and tiktoken's table has no GPT-6 entry as I write. Until OpenAI lists their encoding, I treat a local count for those models as a close guide and confirm it with the input tokens endpoint.

Why Claude token counts are estimates, and how to get Anthropic's number

Anthropic documents token counting as an API call, not a library you run locally. So a "Claude token counter" that works offline in your browser is running some other tokeniser and calling the result Claude. Mine does this openly: it counts with o200k_base and labels the result as an estimate.

For Anthropic's own number, use the token counting endpoint: POST https://api.anthropic.com/v1/messages/count_tokens, with your API key and the same model, system prompt, messages and tools you plan to send. In the Python SDK it's client.messages.count_tokens(...). Per Anthropic's docs:

  • It accepts system prompts, tools, images and PDFs, though images and PDFs must be sent as base64.
  • It is free to use, with its own requests per minute limit: 5,000, 10,000 or 20,000 depending on your usage tier, separate from message limits.
  • Anthropic itself calls the result an estimate: the tokens used when you create the message may differ by a small amount.
  • Claude 4.7 and later models use a newer tokeniser that produces about 30 percent more tokens for the same text than earlier models. Recount when you change models; don't reuse old numbers.

That last point matters for any estimate. Nobody publishes how far an o200k_base count sits from a given Claude model's count. What I do is send a few of my real prompts to the count endpoint once, compare with the browser estimate, and keep that margin in my budget.

How to count Gemini tokens

Google's token guide documents a countTokens method: client.models.count_tokens in the Python SDK, countTokens in JavaScript, and a POST to the model's :countTokens endpoint over REST. The API reference says it runs the model's tokeniser on your content and returns totalTokens, with cached tokens counted separately. Firebase's guide to counting tokens says there's no charge for calling it and puts the quota at 3,000 requests per minute.

Media counts by its own rules. Per the token guide, an image of up to 384 pixels on both sides counts as 258 tokens, larger images are cut into 768 by 768 tiles of 258 tokens each, and audio counts at 32 tokens per second. No text tokeniser can tell you those numbers.

Context windows: will it fit?

A context window is the most tokens a model can handle in one request, input and output together. Anthropic's context window docs say everything counts: the system prompt, every message, tool results, images and documents, tool definitions, and the output including any extended thinking.

Current sizes from each vendor's docs, as read on 1 November 2026:

VendorModelsContext windowOutput limit
OpenAIGPT-6 Astra, GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna1.05M tokens128K tokens
AnthropicOpus 4.6 and later, Sonnet 4.6 and later, Haiku 5.5 and others listed1M tokens128k per request
AnthropicOther Claude models, such as Sonnet 4.5200k tokensSee the model table
GoogleGemini 41,048,576, shared by input, thinking and outputPart of the same window
GoogleGemini 3 and 2.51,048,576 input65,536

What happens at the edge differs too. On Claude, an input that alone exceeds the window returns a "prompt is too long" error; on Claude 4.5 and newer, if input plus max_tokens runs over, the request is accepted and generation stops when it reaches the limit.

A big window isn't a reason to fill it. Anthropic's docs note that accuracy and recall degrade as the token count grows, which they call context rot. In the products I build, I'd rather retrieve the right five passages than send the whole archive. My post on shipping AI features people trust covers how that shows up in the interface.

Common mistakes when counting tokens

  • Counting characters and dividing by four. Fine for a rough English estimate, wrong for code, numbers and most other languages.
  • Counting the user's text only. The system prompt, retrieved documents, tool schemas and earlier turns are often larger than the question.
  • Forgetting the answer. A request that fills the window leaves nothing for the reply. Keep room for the longest answer you expect.
  • Reusing counts across models. Encodings differ between vendors and even within one vendor's model families.
  • Trusting a "Claude" or "Gemini" count from a browser tool as exact. It's an estimate unless it came from the vendor's endpoint.

How my free token counter works

My LLM token counter is built for the moment you have a prompt and need an answer fast:

  • Exact counts with o200kbase and cl100kbase, the encodings OpenAI publishes, plus characters, words, lines and characters per token.
  • A context window budget. Pick a model or type your own limit, set the room for the answer, and a bar shows your text, the answer's room and what's free, or how far over you are.
  • Estimates labelled as estimates. For Claude, Gemini and Llama it counts with o200k_base and says so. The model list is editable, since vendors change their limits; check their docs.
  • A token map that shows every token as a block, spaces as dots, for the first 4,000 tokens.
  • A chunker for retrieval: set the chunk size and overlap in tokens, then download the chunks as a zip of text files or one JSON file.
  • Private by design. The tokeniser runs in a Web Worker in your browser. Only the encoding file downloads; your text never leaves your device.

It doesn't count message formatting, images or tool schemas. For those, use the vendor endpoints above.

Alternatives, honestly

  • tiktoken or js-tiktoken in your own code, for OpenAI counts inside a build step or a server.
  • OpenAI's input tokens endpoint, for whole Responses API requests with tools and files.
  • Anthropic's count_tokens endpoint, for Claude; free, with an API key.
  • Gemini's countTokens, for Gemini text and media; free per Firebase's guide.
  • The usage field on each response. Claude responses report what each request consumed in their usage field, which is the number to log.
  • Other web counters. Many exist; check whether they label non-OpenAI counts as estimates and whether your text is uploaded.

If you're designing the interface around these limits, my post on designing apps for AI assistants covers how products expose their data and actions to models.

Frequently asked questions

What is a token counter?

A tool that runs text through a model's tokeniser and tells you how many tokens it becomes. You need it to estimate cost, check the fit in a context window and size chunks for retrieval.

Is there an exact OpenAI token counter?

Yes. tiktoken and js-tiktoken run OpenAI's published encodings, so plain text counts are exact. Full chat requests add formatting tokens, and OpenAI's input tokens endpoint counts those too.

How do I count Claude tokens?

Send the request to Anthropic's free /v1/messages/count_tokens endpoint with the model you'll use. Browser tools can only estimate, and newer Claude models tokenise differently from older ones.

How do I count Gemini tokens?

Call countTokens through the Gemini API or SDK. It runs the model's own tokeniser and handles images, audio and video, which a text tokeniser can't.

How many tokens is 1,000 words?

For English, Google's rule of 60 to 80 words per 100 tokens gives about 1,250 to 1,670 tokens. Paste your own text for a real count; code and other languages differ.

Do tokens in the answer count toward the context window?

Yes. The window holds the input and the output together, including thinking tokens on models that think, so keep room for the reply.

Sources and method

I read every source on 1 November 2026. The token table above comes from my own run of js-tiktoken with o200kbase and cl100kbase, the library my tool uses. Model names and limits change often, so check the vendor docs before you rely on a number.

If you're building a product on a language model and want the prompts, limits and answers designed and built properly, book a free 30 minute intro call. After it, you get a written scope and one fixed quote.

Tags: AI, Tokens, OpenAI, Claude, Gemini, Free tools

Keep reading

Let’s look at your product.

A free 30 minute call. You leave with a clear next step, whether we work together or not.

Book a free intro call