mdtoken

Measure Markdown token counts and manage context budgets

GitHub MIT pip install mdtoken

mdtoken measures text with tiktoken encodings and provides configurable Markdown token limits. Use its Python API to inspect document size before assembling model context.

What it provides

  • Token counts using tiktoken encodings
  • Configurable Markdown token limits and pre-commit integration in the source project
  • A Python API for text and file processing
  • Claude model aliases use cl100k_base as a proxy, not an exact Claude tokenizer 
pip install mdtoken==1.0.0
Install the checked PyPI release
from mdtoken.counter import TokenCounter

counter = TokenCounter(encoding_name="cl100k_base")
tokens = counter.count_tokens("Your markdown content")
print(tokens)  # 3
Python API, verified with mdtoken 1.0.0

The source CLI uses mdtoken check. The PyPI 1.0.0 distribution omits its commands module, so that CLI currently fails. The Python example above runs with the installed release. Cost estimation is not provided.