> For the complete documentation index, see [llms.txt](https://docs.thecolliery.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thecolliery.org/benchmarks/coaltipple/readme.md).

# CoalTipple—routing savings

This harness measures the **final output correctness** of CoalTipple routing—*what actually reaches the user after the model routes a task*—not just whether the routing decision was correct.

> Two complementary benchmarks. The internal routing dogfood (`dogfood/results/`, `dogfood/results2/`) scores the **decision** ("did the main escalate / delegate / self correctly?"). This one scores the **delivery** ("after it routed, is the answer right?"). Decision-correct does not guarantee output-correct—this closes that gap.

## What it tests (the design)

Every task here is **hard** (a cheap tier cannot do it reliably → forces escalate-up) **and subtle** (an eyeball "looks right" verdict misses the real failure → it only shows under an objective check). Two binding conditions:

1. **Escalate-up ONE RUNG at a time (the staircase), never jump to the top model.** If every main just called the strongest model, the output would be trivially good and we'd be measuring the top model, not CoalTipple. One rung (cheap→mid, mid→heavy, …) measures the real question: **does +1 rung deliver, or must it climb further?**—per (task difficulty × main tier). The top available tier has no rung above → it does the task itself (the ceiling baseline). *(With Fable disabled, Opus is the top.)*
2. **The little sibling cannot do the work, and cannot verify it either.** Because the cheap main cannot self-verify a subtle failure, verification is the **objective gold** (tests / answer key / rubric judged by a strong model)—*never* the main's eyeball. This reveals whether the escalated output reaches the user correct **without leaning on a verify the cheap main can't perform.**

## How to run

* **Division of labour:** *you* copy [`PROMPT.md`](broken://pages/w1qVpByaxmlYaj7Upeeq) and fire it at each model; *each model writes* its deliverables to `dogfood/output/<model-id>.md` (local, self-labelled); *the reviewing session reads + scores them*—no copy-back. Same workflow as the decision dogfood, scoring the OUTPUT instead.
* `PROMPT.md` bundles all 5 CORE tasks (one per group—`coding`, `math`, `knowledge`, `domain`, `creative`) and has the main do them one at a time within the session. All-5-in-one-session is convenient but carries a context-bleed + limit-risk tradeoff; for the cleanest per-task numbers, split into one task per session.
* Fire **Haiku + Sonnet** (the real +1-rung rows) and **Opus** (the ceiling, since Fable is off). Restart between models. Log anything you cap.

## Scoring (objective gold—not the main's verdict)

`node score.mjs <Tn> [file]`—see [`TASKS.md`](broken://pages/qbjByplf2aAqW1XvgYmX) for each task's gold:

| Method                    | Tasks                           | How                                                                                                                     |
| ------------------------- | ------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| **Automated**             | T1 crypto, T5 facts             | `score.mjs T1 <impl>` runs the vectors + static-checks `timingSafeEqual`; `score.mjs T5 <text>` runs the fact-checklist |
| **Rubric + strong judge** | T2 proof, T3 research, T4 legal | `score.mjs T2\|T3\|T4` prints the rubric; a reasoning-tier model scores it (the cheap main can't)                       |

* **Output correctness %** = correct deliverables / total, per model tier.
* **Climb second pass:** when the +1-rung output FAILS the gold, re-run escalating one *more* rung and record whether that clears it—the staircase economics (is one rung enough, or how far must it climb?).

## Output

Raw per-model deliverables → `dogfood/output/` (local). The headline numbers (output-% per tier + climb depth) roll up into [`RESULTS.md`](/benchmarks/coaltipple/readme/results.md) and the repo `README.md` table, **dated** (every published benchmark carries its test date).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.thecolliery.org/benchmarks/coaltipple/readme.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.