McRitchie Studio
Say Hi 👋
Agents Builders
← All Tasks
Sizing

Fix cache_read token inflation

Shipped
fix-fan-out-usage-double-count

Created

Jul 08, 13:39

Started

Jul 08, 13:42

Completed

Jul 08, 16:55

DevOps handoff

Type

Bug

Shape

backend

Worktree Slug

fix-fan-out-usage-double-count

Repositories

mcritchie-studio

Release Slug

—

Branch

feat/fix-fan-out-usage-double-count

Pull Request

https://github.com/amcritchie/mcritchie-studio/pull/459

Local URL

—

QA URL

—

Production URL

—

observability,telemetry,usage-accuracy

Acceptance Criteria

  • tokens_in returns fresh tokens, excludes cache_read
  • intelligence token charts de-inflated for new data
  • actual_size buckets on cost not token totals
  • backfill recomputes actual_size from correct cost
  • regression tests lock token and sizing conventions

Expected Test Plan

  • [unit] Result#tokens_in excludes cache_read (input+cache_creation only)
  • [unit] derive_actual_size buckets on total_cost via cost thresholds
  • [integration] TaskEvent tokens_in fresh-token convention; actual_size from cost end-to-end

Checks Run

  • [unit] Result#tokens_in excludes cache_read (input+cache_creation only); cost still prices cache_read
  • [unit] derive_actual_size buckets on total_cost via ACTUAL_SIZE_COST_THRESHOLDS boundaries
  • [integration] ship persists a cost-derived actual_size round-tripping through the DB
  • [full-suite@f402b405f162ce3d8196f1fa879c7e6747a6b413] bin/rails db:test:prepare test test:system green
  • [rubocop@f402b405f162ce3d8196f1fa879c7e6747a6b413] bin/rubocop clean
  • [post-deploy] bin/rails tasks:backfill_actual_size_from_cost on mcritchie-studio-qa → ok (2026-07-08T16:34:06Z)
  • [post-deploy] bin/rails tasks:backfill_actual_size_from_cost on mcritchie-studio → ok (2026-07-08T16:54:01Z)

Agent Context

RE-SCOPED from falsified fan-out premise (see activity notes). Cost is already correct (Phase 1). Real bug: AgentSessionUsage::Result#tokens_in (lib/agent_session_usage.rb:35-37) includes cache_read (~98% of tokens) -> TaskEvent.tokens_in 30-50x inflated vs activity path, /intelligence token charts inflated, actual_size (measured_tokens_total bucketed by ACTUAL_SIZE_THRESHOLDS, task.rb:833-852,25-30) pins ~every task to XL. FIX (operator-approved): (1) drop cache_read from Result#tokens_in (cache_read stays own bucket for pricing); (2) switch derive_actual_size to bucket on total_cost (GT-correct) with cost-calibrated thresholds; (3) backfill recompute actual_size from cost for existing tasks. Write failing tests first. Cost thresholds: calibrate from real task total_cost distribution during build.

Testing phases

durable timing

How long this task spent in each testing phase — Build → Local Certification → CI → Review → Operator Acceptance.

Build

about 1 hour

Completed

Local Certification

22 minutes

Completed

CI

—

Missing

Review

about 1 hour

Completed

Operator Acceptance

—

Missing

Stage Timeline

Who handled each stage, the time it took (measured), and the model / tokens / cost reported (best-effort) — plus who's on it right now. — means the agent didn't report that metric.

Sizing Avi · PO MEDIUM Dev MEDIUM Actual LARGE ≠ forecast
  1. Created → Designed
    C Chikorita
    Chikorita
    Model
    —
    Duration
    —
    Tokens
    —
    Cost
    —
    Completed Jul 08, 13:39 · about 5 hours ago
    api
  2. Designed → Building
    C Chikorita
    Chikorita
    Model
    claude-opus-4-8
    Duration
    3 minutes
    Tokens
    4,450,253
    Cost
    ~$3.21
    Started Jul 08, 13:39
    Completed Jul 08, 13:42 · about 5 hours ago
    cli
  3. Building → Submitted
    B Bayleef
    Bayleef
    Model
    claude-opus-4-8
    Duration
    about 2 hours
    Tokens
    562,171
    Cost
    ~$39.69
    Started Jul 08, 13:42
    Completed Jul 08, 15:13 · about 4 hours ago
    cli
  4. Submitted → Reviewed Review Activity
    C Carl
    Carl primary
    A Alex
    Alex light
    Model
    claude-opus-4-8
    Duration
    about 1 hour
    Tokens
    8,061,553
    Cost
    ~$5.23
    Started Jul 08, 15:13
    Completed Jul 08, 15:58 · about 3 hours ago
    cli
  5. Evolve
    B Bayleef
    Bayleef
    M Meganium
    Meganium
    triggered by
    C Carl
    Carl
    Model
    —
    Duration
    about 1 hour
    Tokens
    —
    Cost
    —
    Started Jul 08, 15:13
    Completed Jul 08, 15:58 · about 3 hours ago
  6. Reviewed → Assembled
    S Steffon
    Steffon
    Model
    claude-opus-4-8
    Duration
    36 minutes
    Tokens
    20,038,382
    Cost
    ~$13.20
    Started Jul 08, 15:58
    Completed Jul 08, 16:34 · about 2 hours ago
  7. Assembled → Shipped
    A Avi
    Avi
    Model
    claude-opus-4-8
    Duration
    21 minutes
    Tokens
    5,792,010
    Cost
    ~$3.40
    Started Jul 08, 16:34
    Completed Jul 08, 16:55 · about 2 hours ago

Conversation

QA review feedback, agent handoffs, and follow-up notes for this task.

Comment about 5 hours ago

PREMISE FALSIFIED before building. Empirical check: 0 sidechain assistant lines across ALL 157 local session transcripts (and this heavily-fanned-out session has 1009 assistant lines, all isSidechain=false). Subagents run as SEPARATE session files — their tokens are never folded into the parent transcript. So the scoped fix "skip sidechain/delegated turns in AgentSessionUsage.sum_claude_usage" is a NO-OP; it would change nothing and NOT fix the ~2.3x overcount. The real cause is a different mechanism (candidates: dual AgentAction[hook] + AgentActivity[transcript-diff] pricing of the same turns; baseline/attribution over-counting across task moves; cache_read token semantics). Needs re-diagnosis against ground truth (transcript-true cost vs board-logged cost per session) BEFORE any fix. Holding the build.

Comment about 4 hours ago

RE-DIAGNOSED (evidence, 3 real sessions vs UsagePricing ground truth). COST IS NOT OVERCOUNTED — Phase 1 fixed it: every cost surface reconciles x1.00 (close-diff = GT exactly; the old ~2.3x was cache_read lumped into input at full rate, now gone). The residual CURRENT bug is TOKENS, not dollars: AgentSessionUsage::Result#tokens_in (lib/agent_session_usage.rb:35-37) INCLUDES cache_read (~96-98% of all tokens), so TaskEvent.tokens_in is ~30-50x inflated vs the activity/action path (which excludes cache_read). Effects: (1) /intelligence token charts (total_tokens, tokens_by_task/stage/trend, model_breakdown) inflated 30-50x; (2) actual_size derives from measured_tokens_total (task.rb:833-852) bucketed by ACTUAL_SIZE_THRESHOLDS -> nearly EVERY task pins to XL, defeating the sizing signal. FIX: drop cache_read from Result#tokens_in (one line -> fixes charts + consistency), then re-base actual_size (either recalibrate thresholds for fresh-token magnitudes OR bucket on total_cost which is already GT-correct). Dual-booking, close-diff baseline, per-action cost all RULED OUT with evidence. This task's original 'fan-out double-count' premise is dead; re-scope to the cache_read token-inflation + actual_size fix.

Sealed-bid sizing

Edit →

Alex (PM)

—

Avi (PO)

MEDIUM

Dev

MEDIUM

Actual

LARGE