utilities

model-cost-benchmark

Runs the same task across several models and reports measured cost, tokens, and latency so a cheaper model can be adopted with evidence.

Target Audience: Vault owners and operators deciding whether a task, prompt, or skill can run on a cheaper model without losing acceptable output quality.

Difficulty: beginner
Input: conversation
Output: document
Needs: none
Used by: CEO / Founder, Product Manager, Operations Manager

What it needs

task — The prompt to benchmark, inline or as a file
models — Comma-separated provider:model specs (default claude:haiku,claude:sonnet,claude:opus)
results_dir — Directory produced by the run route
instructions — Cacheable prefix tokens per run (system prompt + auto-loaded context + skill)
output_tokens — Output tokens per run, measured from a benchmark run
batch_size — Runs sharing one cache window; 1 (default) means isolated scheduled runs where the cache never hits
results_dir — Directory to clean, passed via --cleanup

What you get

benchmark-results (json)
lever-comparison (text)
metrics-summary (markdown)
pruned-dir (directory)
recommendation (markdown)
stdout

Ask it like this

Can this skill run on Haiku instead of Opus?
Compare Haiku, Sonnet and Opus on this prompt and tell me what it costs
Benchmark this task across Claude and Gemini
Is Opus actually worth it for this job?