Token usage estimates (model and effort)

Token usage estimates (model and effort)

This page tells you what running the toolkit costs in model tokens, so you can budget a run before you start - especially at scale (grading a whole marketplace). It is deliberately explicit about how the model you pick and the effort (thinking budget) you allow trade off against output quality and token cost.

The single most important thing to understand first is that the toolkit has two regimes, and they could not be more different in cost.

The two regimes

1. The deterministic core - zero model tokens, reproducible

The grade itself is produced by a deterministic, model-free gate. scripts/check.mjs (the gate), scripts/tier-report.mjs (the tier and burndown), and scripts/evaluate.mjs in its --format=html|md, --report=migration, --report=release, terminal, and --json modes run no model at all. They are plain Node over the plugin’s files.

• Model tokens: 0. There is nothing to estimate. A run costs the same whether you have a token budget of zero or infinite.
• Reproducible. The grade, the findings, and the gate exit code are deterministic - the same plugin at the same commit produces the same verdict every time. (One caveat for byte-level comparison: a rendered HTML or MD report stamps the current date, so its bytes differ from one day to the next even when the findings are identical. The verdict is reproducible; the rendered artifact is not byte-identical across days. The --json and terminal outputs carry no date.)
• Fast. As a real anchor, grading wshobson/agents (82 plugins) took about 16 seconds and 0 model tokens.

If all you want is the conformance grade and a rendered report, your token budget is zero. This is the regime you will use most, and it is free.

2. The model-assisted layer - real model and effort variance

Two surfaces use a model, and only these:

• The advisory reports. evaluate.mjs --report=review and --report=behavioral render advice produced by the askit-reviewer and askit-quality-grader subagents. The advice is decorated onto the deterministic report by an allowlisting applyAdvisory().
• The authoring skills. The askit-build-* family drafts components (a skill, a subagent, an MCP wiring) with a model.

A load-bearing property bounds your risk here: the advisory layer structurally cannot move the deterministic grade or the gate exit code. A cheaper or lower-effort model on the advisory path degrades the quality of the advice, never the pass/fail verdict. So “go cheap on advice” is a safe lever, not a correctness risk.

Operation catalog

Operation	Command	Regime	Model tokens
Gate (grade)	check.mjs <path> [--profile ...]	deterministic	0
Tier + burndown	tier-report.mjs <path>	deterministic	0
Render report	evaluate.mjs <path> --format=html|md	deterministic	0
Migration plan	evaluate.mjs <path> --report=migration	deterministic	0
Release readiness	evaluate.mjs <path> --report=release	deterministic	0
Review advisory	evaluate.mjs <path> --report=review --advisory <f>	model-assisted	model + effort dependent
Behavioral advisory	evaluate.mjs <path> --report=behavioral --advisory <f>	model-assisted	model + effort dependent
Author a component	the askit-build-* skills	model-assisted	model + effort dependent
Adversarial review	N review agents over a diff	model-assisted	model + effort dependent
Craft review (skill builder phase 2)	the askit-build-skill improve-mode phase 2 (one askit-reviewer dispatch)	model-assisted	one advisory run; default Sonnet 4.6 at medium effort, Opus 4.8 at high for a decision-critical pass, never Haiku (ADR 0037)

The advisory --advisory <file.json> itself is produced by a subagent run (the model-assisted cost); evaluate.mjs then renders it deterministically. The render of an existing advisory file is free; producing the advisory is the cost.

Model and effort vs quality - the explicit tradeoff

For the model-assisted layer, two dials drive both quality and token cost: which model you run, and how much effort (extended-thinking budget) you allow. Match them to the task; do not pay for depth a task does not need, and do not starve a task that needs it.

Model

Model	Judgment quality	Relative token cost	Best for
Opus 4.8 (claude-opus-4-8)	Highest - catches subtle, cross-cutting issues; strongest adversarial reasoning	Highest per equivalent task (more thorough, more output)	Adversarial review, behavioral grading on nuanced skills, authoring a complex component, anything where a missed issue is costly
Sonnet 4.6 (claude-sonnet-4-6)	Strong - a good default; occasionally misses the deepest cross-cutting cases	Middle	Most advisory reviews and authoring; the sensible default when you have many targets and want quality without Opus cost
Haiku 4.5 (claude-haiku-4-5-...)	Solid on bounded, well-specified tasks; weakest on open-ended judgment	Lowest, fastest	Mechanical or bounded passes, first-draft authoring, large-scale runs where you accept shallower advice

The quality gap between models is widest on open-ended judgment (is this skill’s description actually good? does this review find the non-obvious bug?) and narrowest on bounded, well-specified work (summarize these deterministic findings; draft a skeleton from a clear spec). Spend the model budget where judgment is open-ended.

Effort (extended-thinking budget)

Effort	Output tokens	Quality effect	Use when
Low / none	Fewest	Fine for mechanical tasks; shallow on hard reasoning	Summaries, format conversions, bounded drafts
Medium	Moderate	A balanced default	Most advisory reviews and authoring
High	Most	Real gains on hard reasoning (adversarial verification, behavioral nuance); diminishing returns on easy tasks	A correctness-critical review, a subtle behavioral grade, a hard design call

Effort multiplies output tokens (the model thinks more before answering). On an easy task that multiplication buys little; on a genuinely hard one it is where the quality comes from. The classic waste is high effort on a mechanical task; the classic false economy is low effort on a nuanced judgment.

The combined rule of thumb

• Scale, bounded advice you will skim: Sonnet (or Haiku) at low-medium effort. The grade is already deterministic and free; the advice is a bonus.
• One target, decision-critical advice: Opus at high effort. You are paying for the catch you would otherwise miss.
• Never: Opus + high effort to render a summary, or Haiku + low effort for a behavioral grade you will act on.

Measured data points

These are MEASURED from this repository (not estimates):

What	Model	Effort	Tokens	Notes
Gate, one plugin	n/a	n/a	0	deterministic
Gate, 82-plugin marketplace	n/a	n/a	0	~16s wall-clock, deterministic
Render HTML or MD report	n/a	n/a	0	deterministic projection
4-lens adversarial review of a ~160-line diff	Opus 4.8	high	~232k total	4 agents, ~51k-65k each (a useful anchor for “a thorough pre-merge review”)
Review advisory, one skill (clean target)	Opus 4.8	high	~63k	multi-pass verification; 118s wall-clock (run R1, batch 2026-06-10)
Review advisory, one 15-skill plugin (failing gate)	Sonnet 4.6	medium	~67k	the batch’s largest target; 150s (run R2)
Review advisory, one skill	Haiku 4.5	low	~39k	single bounded pass; 49s (run R3)
Behavioral advisory, one skill (12 derived cases)	Sonnet 4.6	medium	~36k	56s (run R4)
Behavioral advisory, one skill (8 derived cases)	Haiku 4.5	low	~33k	24s (run R5)
Review advisory, 86-skill collection (sampled 13)	Opus 4.8	medium	~77k	121s (run R6); instructed sampling is the cost lever at collection scale
Review advisory, 49-skill plugin (failing gate, full triage)	Sonnet 4.6	high	~103k	303s, 70 tool uses (run R7); the measured ceiling so far
Review advisory, one 4-skill plugin	Haiku 4.5	medium	~55k	49s (run R8); ~40% over Haiku/low with materially deeper advice
Review advisory, the SAME 4-skill plugin (A/B vs the row above)	Opus 4.8	high	~100k	123s (run R9); ~1.8x the Haiku/medium cost, and the only run of the pair to catch the target’s factual/legal content errors
Review advisory, the SAME 4-skill plugin (completing the triple)	Sonnet 4.6	high	~72k	321s (run R10); matched Opus/high on every verified content error plus two more verified majors, at ~70% of the Opus token cost
Review advisory, the SAME 4-skill plugin (completing the triple)	Haiku 4.5	high	~64k	108s (run R11); 1 finding - smelled one error but confabulated the correction and asserted false consistency; high effort did not fix the capability gap

The advisory rows come from the first two measurement batches (eight runs, 2026-06-10), recorded run by run in the historical evaluation-run record at docs/internal/eval-runs/eval-runs.md in the repository. Tokens are the harness-reported total for the dispatched advisory subagent (reading the target, running the free deterministic baseline, judging, emitting the advisory JSON); effort is instruction-directed depth.

Three measured lessons worth more than the raw numbers:

• Total cost is dominated by target size, not model tier. The Sonnet/medium review of a 15-skill plugin out-cost the Opus/high review of one skill, and the floor across the whole matrix was ~33k: reading the target dominates the bill. The model and effort dials move the quality of the advice far more than its total token cost - which makes “spend the model budget where judgment is open-ended” cheaper advice than it sounds.
• The quality gradient matched the predicted ordering. Opus/high found a real cross-component defect in a clean, self-conformant target; Sonnet/medium delivered the most findings per token on a messy real plugin (including the only major findings of the batch); Haiku/low was accurate but minimal on review, and on behavioral produced an all-pass verdict from a soft case set - live evidence for the “never Haiku + low effort for a behavioral grade you will act on” rule below.
• Effort moves tool use and wall-clock more than raw tokens, and the Haiku low-to-medium step is high-leverage. Sonnet/high on a defect-rich plugin used 70 tool calls over 5 minutes for ~103k tokens (the ceiling so far); Haiku at medium effort cost ~40% more than Haiku at low and produced materially deeper advice (veracity spot-checks it did not attempt at low). For scale runs on Haiku, instruct medium effort by default. One honesty note from the same batch: a strong model at high effort can still be confidently wrong - one high-effort advisory mis-triaged real link defects as checker false positives - so advisory recommendations about the gate itself are verified against ground truth before being acted on (the verification rule lives in the eval-run methodology, docs/internal/eval-runs/METHODOLOGY.md).
• Model tier changes WHAT KIND of finding is reachable, not just depth (the same-target model triple). On an identical small plugin at identical instructed effort, Opus/high (~100k) and Sonnet/high (~72k) both caught the target’s factual/legal content errors (a wrong statute name in a compliance skill, an inverted grammar rule taught as fact, a command whose scorecard contradicts its skill) - and Sonnet found two additional verified majors at ~70% of the Opus cost. Haiku, at BOTH medium and high effort, could not: at medium it reported “no veracity gaps detected”; at high it smelled one error but confabulated the correction (citing a statute that also does not exist) and asserted consistency that is false. The boundary is categorical (Haiku-vs-frontier), not gradual: if the review must vouch for factual, legal, or domain claims, budget at least one frontier model (two as a panel beat either alone - their verified finding sets only partially overlap), and never treat a Haiku-tier “verified” as verification, at any effort.

Authoring (askit-build-*): MEASURED, 2026-07-26

Six real authoring runs, measured while the builders authored the 25 examples that shipped in v1.8.0. One activity, two outputs: the teaching artifact and the cost datum.

Component size	What was authored	Model and effort	Tokens	Wall
Bounded	a hook golden set (3 golden + 1 anti, 4 runnable scripts)	Sonnet 4.6 / high	106k (see caveat 2)	25 min
Mid	a skill golden set (3 golden + 1 anti)	Opus 5 / xhigh	161k	18 min
Mid	a workflow golden set (3 golden + 1 anti)	Sonnet 4.6 / high	118k	18 min
Larger	an MCP golden set (3 golden + 1 anti, 3 working servers)	Opus 5 / xhigh	173k	20 min
Bounded, batched	5 builders, 1 golden each	Sonnet 4.6 / high	97k total	13 min
Bounded, batched	4 builders, 1 golden each	Sonnet 4.6 / high	100k total	15 min

Budget from this: a single golden-plus-anti set for one component type runs roughly 100k to 175k tokens, with the larger, runnable-artifact-heavy types (MCP) at the top. Treat 175k as the working ceiling for one component type. A batch of bounded single goldens amortizes to roughly 20k each.

Four caveats, because a cost figure quoted without them will be wrong when you use it:

1. The effort cells are not the ones originally planned. The plan called for Sonnet/medium and Opus/high. The Agent tool exposes a model parameter but no effort parameter, so subagents inherit the parent session’s effort. What is recorded is what was measured: Sonnet/high and Opus/xhigh. Two distinct cells, not the named ones.
2. The hook row covers its first round only and is deliberately not extrapolated. Its revision round completed after the figure was taken, and the ratio between harness-reported and transcript-derived totals ranges 0.30 to 0.56 across the six runs, so no stable conversion exists to extrapolate with.
3. Orchestration cost is in no row. Prompt authoring, review, independent verification, and direct fixes were not metered. A real budget adds them.
4. Do not read a batched row as a unit price. 97k for five bounded goldens amortizes fixed reading cost across five; one dispatched alone costs more.

Revisions dominate, and they are the reason to budget a range rather than a point. Of the runs above, one needed a revision round, earned by a real defect (two hook goldens emitted output the runtime silently discards). A single-pass number understates the real cost of getting a component right.

Measured advisory range: across every recorded run, one advisory pass has landed between 33k and 103k total tokens. This line is maintained by the eval-run pipeline (node scripts/eval-run.mjs --aggregate <date>) from the run record, and only ever widens: narrowing a measured claim means retiring a recorded run, which is a human decision.

How to estimate your run

1. The grade and any rendered report: budget 0 tokens. This covers check.mjs, tier-report.mjs, and every evaluate.mjs format except the two advisory reports. At any scale.
2. Advisory reports: budget N targets x (one advisory run) at your chosen model and effort. Measured (batches 2026-06-10): one advisory run lands in roughly 33k-103k total tokens, with the position in that range set mostly by target size and instructed depth (a single skill sits near the floor; a defect-rich multi-skill plugin at high effort sets the ceiling) and only secondarily by model tier. Budget ~40k per single-skill target, ~70k per plugin-scale target at medium effort, and ~110k for a high-effort full triage of a large plugin, then re-check against your own first run. At collection scale (50+ skills), instruct sampling - it held an 86-skill review to ~77k.
3. Authoring: budget per component, dominated by the component’s size and how many revision rounds you allow.

Worked example: grading a 200-plugin marketplace for conformance and rendering an HTML report for each costs 0 model tokens (deterministic, a few minutes of compute). Adding an Opus/high-effort review advisory for the 10 plugins that failed is 10 x (one advisory run) - a small, opt-in, bounded spend on top of a free grade.

Assumptions

These are the assumptions behind every number on this page. They are valid only while they hold; check them against your run.

• “Tokens” means total model tokens (input + output) for the operation, unless a row says otherwise. The 4-lens anchor is the sum of all four agents’ reported usage.
• “Effort” means the extended-thinking budget, the reasoning the model does before it answers. More effort means more output tokens.
• The deterministic regime is genuinely zero model tokens, not “near zero”. It runs no model. This is a structural property of the gate (Design Principle 3), not a measurement that could drift.
• The advisory layer is bounded by the deterministic findings it decorates. It explains, prioritizes, and adds behavioral or review judgment, but it cannot invent a pass or a fail. So its worst case degrades advice quality, never the grade.
• Model judgment quality is ranked Opus > Sonnet > Haiku on open-ended tasks, narrowing to near-parity on bounded, well-specified tasks. This reflects the model family’s general capability ordering, not a per-task guarantee.

Caveats

• Ranges, not point estimates. Model-assisted cost is workload-dependent: the number of findings, the size of the skill, the depth of advice requested, and the number of revision rounds all move it. A single number would mislead; budget with a range and a ceiling.
• This page tracks tokens, not dollars. Per-model pricing changes; convert tokens to cost with current pricing at run time.
• Re-measure when models change. The capability and cost ordering above is current as of this page’s date; a new model generation can shift both. Treat the measured table as the source of truth and the unmeasured ranges as provisional.
• The anchor is one data point. The ~232k 4-lens figure is a single review of a small diff at Opus/high effort. A larger diff, more lenses, or more verification rounds scale it up; a cheaper model or lower effort scales it down. Use it to calibrate, not to predict exactly.
• Guidance, not a guarantee. These estimates help you plan; they are not a service-level promise.

How to keep this current

The honest way to fill the unmeasured ranges is to record real runs. The companion practice is a historical evaluation-run record, which now exists at docs/internal/eval-runs/ in the repository: for each advisory or authoring run, it logs the context (what was evaluated), the skill or plugin name and pinned version, the date, the model and effort, the measured token usage, the wall-clock, and a pointer to the output. The advisory rows above came from its first batch (five runs, 2026-06-10); a handful more runs across models and effort levels turns each remaining provisional range into a measured one.

When you add measured rows, mark them MEASURED and cite the run, and move the corresponding range out of “not yet measured”. Keep the deterministic rows at 0 - that does not change.