decorrelate
Three verifiers agreed. You still only had one.
Adding verifiers feels like buying confidence. If they share a model, a temperature and a prompt scaffold, they fail in the same direction — so "3 of 3 agree" can be one error counted three times, at triple the price.
- alpha
- 49 tests passing
- verified vs published examples
- MIT
The report
What a redundant panel looks like
$ decorrelate report runs.jsonl
verifiers 3 skeptic-1, skeptic-2, skeptic-3
findings 40
contested 6 findings where the panel split
mean pairwise φ 0.803 redundant
Fleiss κ 0.800 observed 0.900 · chance 0.501
N_eff 1.15 of 3
█████████░░░░░░░░░░░░░░░
→ You are paying for 3 checks and getting 1.15. That is 2.6× the cost of the confidence you have.
spent $0.48 on verification · roughly $0.30 of it bought no additional independence
pair φ κ agree shared
skeptic-1 ↔ skeptic-2 0.80 0.80 90% 40 ← same model, same lens
skeptic-2 ↔ skeptic-3 0.80 0.80 90% 40 ← same model, same lens
skeptic-1 ↔ skeptic-3 0.80 0.80 90% 40 ← same model, same lensThe same 40 findings, the same $0.48, judged instead by three genuinely different questions on three different model families: N_eff 1.61 instead of 1.15, and 16 contested findings instead of 6. The contested count is the interesting one — those are the only findings where the votes actually did anything.
The number
N_eff — effective independent verifiers
This is Kish's design effect, borrowed from survey sampling. If N raters were independent, the variance of their mean vote would be p(1-p)/N. When raters correlate with average intra-class correlation ρ, it is p(1-p)/N · (1 + (N-1)ρ). That bracketed term is the factor by which correlation inflates your uncertainty, so dividing it out gives the number of independent raters that would have produced the same precision.
N_eff = N / (1 + (N - 1) · ρ)| ρ | 3 verifiers give you | Reading |
|---|---|---|
| 0.00 | 3.00 | genuinely independent |
| 0.40 | 1.67 | overlapping |
| 0.75 | 1.20 | one verifier wearing three hats |
| 1.00 | 1.00 | you have one verifier |
ρ is estimated as the mean pairwise phi coefficient across your verifiers. Negative ρ is clamped to zero — anti-correlated verifiers are not more than independent in any useful sense, and reporting N_eff > N would be sampling noise dressed up as a finding.
Alongside it you get Fleiss' κ with its observed and expected components kept separate — high agreement plus high chance agreement means everyone says yes to everything, not everyone is right — and per-pair φ and Cohen's κ, sorted worst first and flagged when a pair shares a model or a lens.
Verified
The statistics are checked against somebody else's numbers
A tool whose whole value is a number has to be right about the number. The test suite reproduces the canonical Fleiss worked example — 10 subjects, 14 raters, 5 categories — to three decimal places on all three published figures.
published, and reproduced
observed and chance agreement, both matched
tests, including every degenerate case
- Hand-computed 2×2 cases for Cohen's κ and φ.
- A rater with no variance — mathematically undefined correlation, reported as 0 and flagged as degenerate rather than emitted as NaN.
- A panel where chance agreement is total — κ reported as 0 rather than NaN.
- An unbalanced panel — rejected outright, because Fleiss requires a fixed rater count and quietly averaging over a ragged matrix would be a wrong answer that looks fine.
The fix
Four interventions, in order of return
1 · Vary the lens
Three different questions beat three identical ones, and it costs nothing. The lens banks are built to one constraint: no two lenses may be answerable by the same reasoning over the same lines.
2 · Vary the model
Cross-family beats cross-version. Routing two lenses to two Opus versions shares far more prior than routing one to Opus and one to Sonnet, so assignment is round-robin across families, not ids.
3 · Prefer an oracle
A test, a compiler, a linter, a reproduction script: deterministic, zero tokens, no shared priors. The planner marks which lenses have a deterministic equivalent — usually more than half.
4 · Asymmetric thresholds
Unanimity plus an oracle for destructive findings; one vote for cheap-to-fix nits. Uniform verification is the most common way these systems get expensive without getting more correct.
$ decorrelate lenses security
1. authz claude-opus-5
Can a caller who should not reach this code path reach it? Name the caller and the path.
catches: missing or wrong authorization, not input handling
oracle grep for the auth middleware/decorator on this route before asking a model
2. input claude-sonnet-5
Is any attacker-controlled value used here without validation or escaping? Quote the value and its sink.
catches: injection and deserialization, independent of who is calling
oracle a taint-analysis pass or a targeted fuzz case
3. session claude-haiku-4-5
Is session or identity state read, trusted, or mutated without a freshness check?
catches: stale-token and confused-deputy bugs the other two lenses do not look for
high / destructive unanimous (3 of 3) plus an oracle where one exists
medium majority (2 of 3)
low / cheap to fix 1 of 3, or skip verification entirelyDomains: security, correctness, performance, migration, research, generic. The planner refuses to pad past the bank size rather than duplicating a question — inventing a fourth lens by rewording the third is the exact thing the report measures.
Limits
What it does not do
It does not judge whether your findings are correct. It measures verifier independence. A panel can be perfectly independent and uniformly wrong.
ρ is estimated from your sample. With 5 findings the number is noise. The report states how many findings every verifier actually judged — treat fewer than about 20 as directional only.
It needs recorded verdicts. For the static version — N structurally identical verifiers visible in the source — use graphlint.