Cost-per-token dashboards report what a call cost and stop there. They cannot tell a successful call from a wasted one. Verdryx grades the output, exact match, regex, a production outcome tag, or an LLM judge, and prices the result that actually matters: cost per resolved case, not cost per call. On the last live run: $0.00042 to resolve a case, $0.00025 spent on one that went nowhere.
This is a simulation, but shaped by the real run: 176 real Claude calls read back as outcome-tagged traces. A shift's worth of support cases gets graded, a real cost per resolved case comes out the other side, and a drift check runs against a stored baseline before one OTLP span closes the loop.
Traces and eval sets go in. Exact match, regex, an outcome tag straight off the gateway, or an LLM judge decide what happened, and the judge's own calls get priced like everything else. What comes out: a cost-per-outcome report, a drift verdict against a stored baseline, a SQLite file, and an event or a span for whatever is already watching your stack.
Resolved, abandoned, escalated: real outcome tags straight off the TokenFuse gateway turn into a dollar figure per case, not a total nobody can act on.
A flat threshold catches the obvious drop. A two-sample significance check beside it, Welch's t plus a bootstrap interval, catches the small, consistent regression a threshold tuned for noise would miss. Point it at a baseline whose source run is gone and it fails loudly, not silently.
An llm_judge case's Score.cost_usd is a real dollar figure, priced against the same book TokenFuse uses, not the zero placeholder the other three graders leave behind. Quality measurement gets an honest price too.
The eval runner, graders and drift math import nothing but the standard library. Anthropic and Parquet support are opt-in extras, not a tax on the common path.
217 tests run against a deterministic StubLLMAdapter, no network, no API key. It's what Verdryx's own CI runs, and what --model stub gives you too.
Grades an operator's own agents against an eval set or a production outcome tag. The README puts it plainly: it “never manipulates outputs, never crafts adversarial prompts, and never attacks anything.”
Cost-per-token dashboards answer a question nobody asked: what did the call cost. Verdryx answers the one that matters: what did getting it right cost, and did that number just get worse.
| Verdryx | cost-per-token dashboards | “eyeball the transcripts” | |
|---|---|---|---|
| Unit | Correct case | Token | Vibe |
| Catches quality regressions | Drift vs baseline, significance-tested | No | Sometimes, too late |
| Knows what an abandoned attempt cost | Yes, $0.00025 in the live run | No | No |
| Needs a platform | pip install, one SQLite file | A SaaS contract | Nothing but tears |
Verdryx prices the traces TokenFuse writes: outcome tags and spend data share the same request path, so cost and quality can never quietly drift apart. Its quality_drift events ride the same bus as Platform's Agent Passport contract. Mockryx can require an off-path reaction from Verdryx as part of a pre-prod drill.
No network, no key: --model stub is deterministic. Then price the traces it left behind.