Ingestion Latency and Contextual Mapping Velocities in DSOM

Artifact Level: L2 (Analysis) Status: Verified — Empirical benchmarks captured on T1 (Windows/NVMe) 2026-07-19.

Abstract

Traditional knowledge retrieval architectures introduce high latency due to remote API network calls, vector index lookups, and embedding pipeline processing. Under the Deep State of Mind (DSOM) protocol, the velocity of knowledge assimilation is bounded purely by local storage read throughput and the token processing capacity of the underlying model inference run. By utilising structured Open Knowledge Format (OKF) Markdown arrays within the isolated .agents/brain/ namespace, knowledge reading shifts from search operations to deterministic memory loading.

---

1. Assimilation Velocities (From Disk to Inference)

Because DSOM bypasses remote networks and embedding calculation queues, the speed of acquiring complete directory awareness matches the localised processing parameters of the agent environment.

       [ REPOSITORY STEP ]           [ PROCESSING PIPELINE ]           [ TIME HORIZON ]

    ┌────────────────────────┐
    │ Parse Directory Array  │ ────►  Standard POSIX file read   ────►  ~5–50 Milliseconds
    └────────────────────────┘
                │
                ▼
    ┌────────────────────────┐
    │ Filter Frontmatter     │ ────►  Regex match on OKF header  ────►  ~10–15 Milliseconds
    └────────────────────────┘
                │
                ▼
    ┌────────────────────────┐
    │ Memory Ingestion Block │ ────►  Direct Context Insertion   ────►  Instantaneous (On-Load)
    └────────────────────────┘

With this architectural alignment, scanning an entire logic tree—such as a specialised wing_dsom_core containing multiple structural fact rooms and declarative asset lists—takes only milliseconds before the files are fully present in the active memory block.

---

2. The Mechanics of Instantaneous Ingestion

The rapid mapping of directory state relies on explicit folder organisation and strict meta headers rather than deep scanning routines:

2.1 Immediate Frontmatter Indexing

The AI looks exclusively for the OKF YAML header:

---
okf_version: 0.1
type: engineering_matrix
title: "Target Room Architecture"
---

By reading only the metadata block first, the model validates the scope, data freshness, and relevance of a directory path within milliseconds, without wasting processing power on long plaintext analysis.

2.2 Predictable Token Footprints

The model executes the local calculation utility before parsing any text blocks:

uv run --with tiktoken .agents/skills/dsom-token-calculator/scripts/calculate-tokens.py [TARGET_PATH]

This script runs locally to instantly return exact workspace size limits, mapping layout constraints before files are fed into the main context engine.

2.3 No Database Lookups

Bypassing standard vector search indexes eliminates chunk optimisation loops and distance calculation delays, enabling near-instant document loading.

---

3. Knowledge Acquisition Performance Benchmarks

3.1 Architecture Comparison

Ingestion Parameter	Remote Vector / RAG API	Local DSOM OKF Directory
Discovery Latency	High (1500–5000ms): Network handshakes, semantic database vector distance checks, text fragment reassembly.	Ultra-Low (<50ms per file): Local filesystem reads inside bounded folder blocks (e.g., .agents/brain/).
Directory Scope Mapping	Unpredictable: Subject to chunk fragmentation and search algorithm thresholds.	Deterministic: Full visibility across directory tables via single-pass frontmatter sweeps.
Resource Efficiency	Consumes external memory, requires continuous network connections, and scales API costs with every request.	Fits entirely inside local hardware caches, enabling high performance on low-spec mobile setups (like Termux loops).

3.2 Empirical Benchmarks — T1 Node (Windows 11 / NVMe)

Measured via tiktoken cl100k_base across .agents/brain/ (38 files, 145,729 total tokens):

Metric	Measured Value
Files scanned	38
Total tokens (brain dir)	145,729
Total filesystem read time	910.048 ms
Total tokenisation time	337.177 ms
Combined pipeline total	1,247.225 ms
Average per-file read	~23.9 ms
Average per-file tokenise	~8.9 ms

Notable outliers (require gating):

File	Tokens	Read	Tokenise
palace_update_proposal_2026-07-17_0713.md	55,726	38.977ms	51.921ms
palace_update_proposal_2026-04-08_2154.md	34,534	31.192ms	35.700ms
palace_update_proposal_2026-04-08_1214.md	30,206	35.467ms	39.563ms

Advisory: The three palace_update_proposal files alone account for 120,466 tokens — 82.7% of total brain token mass. These are archival documents and must never be blindly loaded into active context. The 4,000-token gate correctly blocks them.

Termux Note: Android FUSE filesystem overhead was benchmarked to add a ~3.5× read latency multiplier over bare-metal NVMe (Samsung Note 10 simulation). The bare-metal read times become proportionally higher (e.g. 5.7ms on NVMe becomes ~20ms on Termux/Android). Actual Termux hardware runs confirmed this architectural scaling.

---

4. Context Optimisation over Mobile Pipelines

By optimising file ingestion for light workloads, directory mapping stays responsive even when operating over resource-constrained hardware fabrics:

• Progressive Disclosure Rules: The system filters out non-plaintext formats instantly, reading files by category block rather than opening the entire folder matrix simultaneously.
• Granular Task Boundaries: Keeping factual data rooms decoupled into clear directories ensures that only relevant information paths are added to the processing stream.

---

5. Advanced Markdown Optimisation Techniques

5.1 Strict Semantic Chunking via Markdown Headers

Standard chunking strategies split files at arbitrary character boundaries. Chunking by Markdown headers (##) guarantees every extracted piece retains its contextual parameters:

def split_markdown_by_section(md_text: str) -> list:
    """Split on H2 headers only. Safe for frontmatter."""
    import re
    # Split only on lines that START with ##, avoiding frontmatter
    parts = re.split(r'(?m)^## ', md_text)
    return [parts[0]] + [f"## {p}" for p in parts[1:]]

Caution: A naive split("\n## ") breaks YAML frontmatter blocks that contain ## inside string values. The regex above anchors to line start with (?m)^ to avoid this.

5.2 Active Context Manifest (Replaces Symlinks)

The symlink pattern proposed in earlier drafts is not suitable for this project due to: - Git on Windows (T1) stores symlinks as text files without core.symlinks=true and admin elevation. - Cross-platform breakage between T1 (Windows), T2 (WSL2/AlmaLinux), and Termux (Android FUSE).

Replacement: OKF active_context_manifest.md

---
okf_version: 0.1
type: active_context_manifest
title: "Active Context — YYYY-MM-DD"
description: "Declares the exact file paths the agent must load for the current task session."
---

## Active Files

- .agents/brain/task.md
- .agents/brain/wings/wing_dsom_core/hall_facts/room_tooling/closet.md

The agent reads this manifest on SOD, then loads only the listed files. Same cognitive architecture, zero portability risk.

5.3 Structural Deduplication (Governance)

Warning: Automated deduplication across closet.md and SKILL.md is high-risk. Automated structural deduplication can silently delete contextual nuance that an LLM depends on for correct reasoning.

Mandatory gates before any deduplication script runs: 1. Execute in --dry-run mode first, outputting a unified diff. 2. Require explicit human-in-the-loop approval before applying changes. 3. Scope is restricted to closet.md brain artefacts only. SKILL.md files are off-limits.

---

6. Token Quality Gate Enforcement

The dsom-token-calculator skill enforces two-tier gating:

Token Count	Agent Behaviour
< 4,000 tokens	Full file content dumped into active context.
≥ 4,000 tokens	Script raises [BLOCKED] flag. Agent generates OKF summary header and uses view_file with line-range targeting instead.

The per-section alerting capability (flagging individual ## sections that breach thresholds) is implemented in calculate-tokens.py v2 — see dsom-token-calculator skill for the updated script.

---

Architectural Caveats

[!CAUTION] The latency figures in Section 1 (5–50ms, 10–15ms) are theoretical architectural estimates. Empirical benchmarks (Section 3.2) show real combined pipeline times of ~32ms per file on T1 (Windows/NVMe). Termux/Android FUSE overhead is empirically benchmarked to be ~3.5× slower than bare-metal NVMe based on Samsung Note 10 performance profiles.

[!WARNING] The palace_update_proposal files in .agents/brain/ each exceed 30,000 tokens. Loading any one of them without the 4,000-token gate will consume the equivalent of the entire active brain budget. These files are archival-only and must be accessed via targeted view_file with explicit line ranges.

---

SOURCES

Document	Description
BYTE-CAPPED-EXECUTION-FRAMEWORK.md	Enforcement model for Byte-Capped Executions (Rule 10).
DSOM-TOKEN-EFFICIENCY-REPORT.md	Token efficiency audit proving 96.23% reduction via DSOM protocol.
PYTHON-UV-ENVIRONMENT-GUIDE.md	Mandated uv execution environment (Rule 16).
dsom-token-calculator SKILL.md	Executable skill for local token footprint measurement.

---

Deep State of Mind (DSOM) For My AI Protocol | Harisfazillah Jamel (LinuxMalaysia) | 2026-07-19 Standard: UK English | DBP-standard Bahasa Melayu Malaysia (Piawai) | GNU General Public License v3.0