Token Tracking in Production

Per-token prices dropped 280× since 2022. Production bills went up.

The gap is not the model. It is the workflow.

The Production Reality

Agentic systems consume more tokens than chat.

• Gartner: 5–30× more tokens than equivalent chatbot asks
• Stanford: coding agents up to 1,000× more than chat completions
• Production audit: 366K input tokens per average request across 10-turn conversations

Same task. Same outcome. Opus 4.7 used 2.9× more output tokens than 4.6.

Tokenizer changes can raise bills 35% overnight without touching the rate card.

Counting tokens is not enough. You need to know where they go.

What To Track Beyond Counts

A counter tells you how much you spent. It does not tell you why.

Track these instead:

Tokens per completed task One user action may trigger five LLM calls. Measure the full chain, not each call in isolation.

Quality gate metrics How many requests never reached the model because relevance was too low? Those are your cheapest wins.

Context utilization Models reason better with free context. Production systems often fill the window and pay for history they no longer use.

Call chain patterns Five sequential calls at 250 tokens each beats one call at 1,250 tokens. Consolidated designs win on latency and cost.

Quality Gates

Most savings come from calls you do not make.

pub const QualityGate = struct {
    relevance_floor: f64 = 0.005,
    max_context_files: usize = 6,
    max_output_tokens: usize = 500,

    pub fn shouldCallLLM(self: QualityGate, score: f64) bool {
        return score >= self.relevance_floor;
    }
};

Usage:

const gate = QualityGate{};

if (!gate.shouldCallLLM(combined_score)) {
    tracker.recordSkipped(.below_relevance_floor);
    return;
}

const context = selectTopN(ranked_results, gate.max_context_files);
try llm.review(context, gate.max_output_tokens);

A relevance floor of 0.005 skips roughly 57% of review requests in real codebases. Weak signal, weak answer, full price.

Search-First Pipeline

Put inference at the end of deterministic work.

Stage 1: Vector search      (~$0.0002)
Stage 2: BM25 reranking     (~$0.0001)
Stage 3: Static analysis    (~$0.0001)
Stage 4: Quality gate       (free)
Stage 5: Single LLM call    (~$0.03, only if gate passes)

Stages 1–4 cost $0.0004 combined. They decide what is relevant, rank it, resolve symbols, and check whether an LLM call is worth making.

pub const PipelineStage = struct {
    source_char_budget: usize = 600,
    output_token_budget: usize = 200,

    pub fn trimContext(self: PipelineStage, context: []const u8) []const u8 {
        return if (context.len > self.source_char_budget)
            context[0..self.source_char_budget]
        else
            context;
    }
};

Input caps matter more than output caps. Production overspend lives on the input side.

Consolidated Design

The naive multi-call chain:

1. Classify intent        → ~150 tokens
2. Plan subsystems        → ~250 tokens
3. Query subsystem A      → ~300 tokens
4. Query subsystem B      → ~300 tokens
5. Synthesize answer      → ~250 tokens
                           Total: ~1,250 tokens, 5 calls

The consolidated version:

1. Classify intent        → ~80 tokens (tight tagger)
2. Fan-out to subsystems  → 0 tokens (deterministic map)
3. Synthesize answer      → ~200 tokens
                           Total: ~280 tokens, 2 calls

The classifier returns a tag from a fixed vocabulary. Each tag maps to downstream calls in plain code. No LLM in the routing step.

pub const IntentRouter = struct {
    pub fn route(intent_tag: []const u8) []const Subsystem {
        if (std.mem.eql(u8, intent_tag, "revenue.query.ytd"))
            return &.{ .crm, .analytics };
        if (std.mem.eql(u8, intent_tag, "account.lookup.contact"))
            return &.{ .crm };
        return &.{};
    }
};

Use the model to classify. Use code to route.

Task-Level Tracking

Extend basic counters to capture the full picture:

pub const TaskMetrics = struct {
    llm_calls: u64,
    skipped_calls: u64,
    input_tokens: u64,
    output_tokens: u64,
    deterministic_cost_usd: f64,
    llm_cost_usd: f64,

    pub fn costPerTask(self: TaskMetrics) f64 {
        return self.deterministic_cost_usd + self.llm_cost_usd;
    }

    pub fn skipRate(self: TaskMetrics) f64 {
        const total = self.llm_calls + self.skipped_calls;
        if (total == 0) return 0;
        return @as(f64, @floatFromInt(self.skipped_calls)) / @as(f64, @floatFromInt(total));
    }
};

Record one TaskMetrics per user action. That is where architectural problems become visible.

Real Economics

Routable structured queries: $0.274 per multi-turn Sonnet conversation vs $0.0009 via deterministic retrieval. A 300:1 ratio on the slice that does not need full reasoning.

Model choice is a 2–5× lever. Architecture is closer to 10×.

Production Patterns

Treat inference as the expensive final step. Search, regex, and switch statements are cheap. Reserve the model for judgment over unstructured input.

Use LLMs to code solutions, not to solve problems. A model that writes a 50-line classifier costs tokens once. A model that acts as the classifier costs tokens on every request forever.

Set ROI targets per call. “AI-assisted code review” is too coarse. “Review 6 files with relevance > 0.005, return 500 words” is measurable.

Enforce budgets in code, not in prompts. Per-source character limits. Per-call token caps. Quality gates before every inference step.

Instrumentation Matters

Track to find architectural problems, not just to report costs.

Quality gates save 90%+ of unnecessary calls. Consolidated designs cut token chains by 4×. Input trimming stops the slow drift that tokenizer changes hide.

Build the counter first. Then build the pipeline that makes most calls unnecessary.

That is where production token economics actually change.

Implementation

Count tokens before you call the API.

zig-ai-tokenizer — BPE token counting and cost estimation in Zig.

git clone https://github.com/Stdubic/zig-ai-tokenizer.git
cd zig-ai-tokenizer
./scripts/fetch-vocab.sh
zig build
./zig-out/bin/zig-ai-tokenizer count "hello world"
# 2 tokens

Cursor Integration

Add to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "zig-ai-tokenizer": {
      "command": "python3",
      "args": ["mcp/server.py"],
      "cwd": "/path/to/zig-ai-tokenizer"
    }
  }
}

Reload Cursor. Then ask:

Use count_tokens on: "Write a Python Fibonacci function"
Use estimate_cost with gpt4 and 500 output tokens

Two MCP tools: count_tokens and estimate_cost. Full setup: github.com/Stdubic/zig-ai-tokenizer/docs/cursor-setup.md