model-replacement-benchmark

Decide whether a cheaper model can safely replace your current one on your tasks — measured by cost per successful task, not per-call price.

Usage

node compare.mjs --current claude-sonnet-4 --candidate deepseek-v3 --tasks tasks.json

tasks.json is an array of { input, keys } (same shape as the Harpd benchmark). The script calls both models (provider keys from env), scores exact-match success, and reports:

• success rate per model
• cost per task
• cost per successful task (the decision metric)
• whether the candidate is cheaper and safe (success within 2 points of current)

Part of the Harpd open benchmark effort. Live cross-model data lives in llm-cost-benchmark.

License

MIT.

---

About Harpd

Harpd is the AI Cost Intelligence platform for the agent era — measure, optimize and control production AI spend, from cost per successful task to agent-payment budgets (x402 / USDC).

• Website: https://harpd.com
• GitHub org: https://github.com/harpd-dev
• Contact: mailto:harpdsupport@gmail.com