What should a reduced-refusal model evaluation measure?
Record refusal rate, task success, formatting, latency and actual cost as separate measurements. Use tasks representative of the application you plan to build.
A repeatable comparison protocol
- Fix the task. Use the same inputs, output format, reasoning setting and output budget for each exact model ID.
- Separate refusal from failure. Record an explicit refusal separately from a timeout, empty response, invalid JSON, wrong answer or truncated generation.
- Keep every attempt. Repeat your tests and retain unsuccessful outputs, not just the best answer.
- Measure useful work. Compare task acceptance, instruction following, tool-call validity and cost per successful task, not token price alone.
- Re-test changes. A hosted alias or model update can change behavior even when your client configuration stays the same.
Evaluate coding outputs in isolation
Generated code is untrusted. Run acceptance tests in a disposable sandbox without API keys, host credentials or network access. Pin the test suite, record a candidate hash, and keep the tests read-only. Use operating-system isolation in addition to a virtual environment.
Report cost per accepted result
For ordinary uncached text usage, estimate cost as input tokens × input price / 1,000,000 + output tokens × output price / 1,000,000. Reported cache reads and writes use their own published rates. The final Usage record, not time spent waiting or a headline token limit, is the billing reference.
Example: at $0.20 per million input tokens and $0.60 per million output tokens, 10,000 uncached input tokens and 2,000 output tokens cost an estimated $0.0032. This is an illustrative calculation, not a quoted price for a particular model.
For model selection, divide the total cost of all measured attempts by the number of acceptable results. A lower token rate is not automatically a lower cost per completed task.
What RouterHub’s published evaluation shows
RouterHub’s public coding evaluations on GitHub include prompts, raw outputs, settings, candidate hashes and sandbox-scored tests. The GLM results record both the default-effort attempts and the low-effort cohort, so you can compare runs with their original settings.
The published Flask upload task reported three low-effort attempts for each of three GLM entries passing all 17 tests. A separate default-effort abliterated cohort returned no code within its budget. Compare these cohorts separately, using their recorded settings.