Skip to content

dsom_token_auditor.py reference

Tiktoken-based token efficiency and context load calculation script.

Description

The dsom_token_auditor.py script compares non-DSOM "Bloated" context loads against optimised DSOM configurations (which utilise episodic resumes and progressive disclosures). It calculates savings percentages.

Script path

tools/dsom_token_auditor.py

CLI signature

uv run --with tiktoken python tools/dsom_token_auditor.py

Outputs

Prints an analysis report: - Token counts for bloated scenario structures. - Token counts for progressive disclosure scenario structures. - Percentage and absolute token savings achieved per execution turn.

Dependencies

  • tiktoken: Fast BPE tokenization library.

Internal Python API

count_tokens(text, model="gpt-4")

Counts tokens for a given string using optimised tokenisation with the specified model's encoder. - Arguments: text (raw string), model (defaults to "gpt-4"). - Returns: Integer representation of token count.

generate_bloated_context()

Generates a raw string simulating chat history and massive file loads.

generate_dsom_context()

Generates a raw string simulating episodic records and relative link references.

Deep State of Mind (DSOM) For My AI Protocol | Harisfazillah Jamel (LinuxMalaysia) | 2026-08-14 Standard: UK English | DBP-standard Bahasa Melayu Malaysia (Piawai) | GNU General Public License v3.0