Run 2026-09-13_x4_critic_sonnet

Preregistered decision rule (PREREGISTRATION.md section 5)

• H1 (planted recall 8/12 vs decoy false positives 1/6, Fisher one-sided): p = 0.0656 -> FAIL
• H2 (sampler enrichment, hypergeometric): B3 p = 0.482, B6 p = 1.000 -> FAIL
• H3 (S0 two-note recall vs B4 one-note recall, Fisher one-sided): p = 0.1102 -> FAIL

Decision: NULL (signal requires H1 and H2 both passing).

Run summary

Total measured cost: $41.135067

Models: - cards: claude-haiku-4-5-20251001 (alias haiku) - critic: claude-sonnet-5 (alias sonnet) - critic_sonnet: claude-sonnet-5 (alias sonnet) - generator: claude-sonnet-5 (alias sonnet) - match_gold: claude-haiku-4-5-20251001 (alias haiku)

Exploratory finds (non-planted survivors, manual review only): 36

Oracle recall on planted: 66.7% [39, 86]; permutation p = 0.0009

T1. Corpus

notes	cards	cards_per_note	bridges	decoys	leakage_failures	corpus_sha256
60	328	5.47	12	6	0	f17c4f086430

T2. Sampler enrichment; base rate 0.65% of 52887 cross-note card pairs are planted

arm	draws	planted_hits	distinct_bridges	expected_hits	p_hypergeom
B1	100	1	1	0.654	0.482
B3	100	1	1	0.654	0.482
B6	50	0	0	0.327	1

T3. Oracle set: generator and critic without the sampler; mean cosine to gold on planted = 0.428

S0 group	units	NONE rate	non-NONE rate	survivor rate (post critic + dupgate)
planted	12	8.3% [1, 35]	91.7% [65, 99]	58.3% [32, 81]
decoy	6	83.3% [44, 97]	16.7% [3, 56]	16.7% [3, 56]
random	42	81.0% [67, 90]	19.0% [10, 33]	9.5% [4, 22]
recall on planted (gold match)	12			66.7% [39, 86]

T4. Per arm: NONE rate, critic kill rate, survivors, recovered planted bridges, measured cost

arm	units	NONE	critic kill	dup	survivors	bridges reachable	bridges recovered	recall	cost usd	usd per recovered
B1	100	84.0% [76, 90]	12.5% [3, 36]	0	14	1	0	0.0% [0, 79]	9.7561	n/a
B3	100	81.0% [72, 87]	21.1% [9, 43]	0	15	1	0	0.0% [0, 79]	10.0033	n/a
B4	24	0.0% [0, 14]	16.7% [7, 36]	0	20	12	4	33.3% [14, 61]	4.3559	1.089
B6	50	96.0% [87, 99]	0.0% [0, 66]	0	2	0	0	0.0% [0, 0]	4.2125	n/a
S0	60	66.7% [54, 77]	40.0% [22, 61]	0	12	12	8	66.7% [39, 86]	7.4075	0.9259

T5. Label-permutation null over the S0 oracle set

statistic	observed	expected under null	p	shuffles
survivors among planted S0 units	7	2.402	0.0009	10000

T6. Recombination vs single-note reflection on the same bridge notes

arm	bridges reachable	recovered	recall	fisher p (S0 > B4)
S0 two notes	12	8	66.7% [39, 86]	0.11
B4 one note	12	4	33.3% [14, 61]

T8. Recall after critic, per critic (same generations, before the duplicate gate)

critic	model id	correct recoveries	surviving	killed	decoy answers	decoy surviving	kill rate on random
sonnet	claude-sonnet-5	8	5	3	1	1	50.0% [22, 78]