> For the complete documentation index, see [llms.txt](https://docs.thecolliery.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.thecolliery.org/benchmarks/skill-determinism/readme.md).

# SkillDeterminism

Measures the half of a shipped skill's behaviour that **nothing else checks**: a `SKILL.md` body, its `references/*.md`, and a hook-injected coordination cue are all natural-language prose. Their code-borne behaviour has tests; their **prose-borne** behaviour ships on the author's belief that it reads the same way to every reader. This benchmark measures whether it actually does—by having independent readers read the SAME text and checking whether they extract the SAME set of directives.

**This is not a correctness benchmark.** It measures agreement between readers, never whether the readers are right. Run 14 (below) is the result that makes this distinction load-bearing: a text can reach zero variance while every reader agrees on the wrong answer. Report both, always.

## Layout

| File                 | What it is                                                                                                                                                |
| -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `PREREGISTRATION.md` | The divergence metric, the pass/fail rule, the hypothesis under test, and the declared limitations—written before the hypothesis-test run (Run 15) fired. |
| `tasks.md`           | The frozen walker instruments, verbatim, per run—a stranger can re-run any of them against the named blob.                                                |
| `RESULTS.md`         | The digest. **The only link target** for an org row or a README.                                                                                          |
| `results/`           | Dated detail records with full per-rail tables and the five-part stamp.                                                                                   |
| `results/raw/`       | The frozen prompt files, byte-identical to what was dispatched (hashes in `tasks.md`).                                                                    |

## Standing rails

* **Metric:** `divergence(rail) = 1 − (walkers matching the modal answer / N)`, **MAX across rails, POOLED across tiers.** MAX because a mean lets one broken rail hide behind healthy ones; pooled because a file stable only on the strong tier has already failed the promise it exists to keep — the weakest plausible model must follow the contract too.
* **Not decorrelated evidence.** Every walker here is a fresh `blind-ic`/`Explore` leaf, and the platform injects the umbrella governance stack at spawn regardless (measured 2026-07-26). Between- walker spread is real data; agreement across walkers is never cited as independent confirmation.
* **Four instrument shapes exist and are NEVER pooled into one cross-run number:** lane EXECUTION (Runs 11-a-kind), lane SELECTION (Run 12-a-kind), a hook-injected coordination cue (Runs 13/14/17/18 —four sequential carves of the same clause), and this benchmark's own prose-index hypothesis test (Run 15). Each is tabled on its own terms; a rail from one is never averaged against a rail from another.
* **A prediction is written down before the walkers fire, every time.** A hypothesis confirmed after the fact is not evidence; the value of this method is that it can be wrong in public.
* **Novelty is a separate, weaker claim than the measurement itself.** A market survey (2026-08-01, \~18 candidates) found no tool doing all four of: tests a *shipped instruction file* · for *determinism across repeated identical reads* · *across model tiers* · *as a shipping gate*. The closest was Anthropic's own skill-authoring guidance (3/4—it recommends cross-tier testing and states outright that no built-in runner exists). State this as "not found in a real search", never as "nobody does this".


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.thecolliery.org/benchmarks/skill-determinism/readme.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.