Source: https://oliviercouthaud.com/proof · Status: Prototype · 2026 · Tags: TypeScript, Evals

# proof

A small TypeScript library that checks whether an AI system's choices about what to raise carry any information, and refuses to print a number when they don't.

## What I wanted

I was building an assistant that decides for itself what to bring up with me, and I wanted to know if it was any good. The usual way to test an AI is to ask whether its answers were right. That doesn't fit here. A system that chooses what to raise doesn't fail by being wrong, it fails by raising everything, or nothing, and looking fine either way. Only a denominator shows that, meaning the cases it considered and did not raise, and almost nobody keeps those.

In about a week of building that assistant I caught five results that were about nothing. Every one was clean and well formed and looked publishable, and every one became a rule in this library. The assistant works on my personal data, so I'm leaving it out. What's here is the harness I pulled out of it in July 2026, version 0.1.0. It's TypeScript with no runtime dependencies, and the code isn't public.

### Five rules, each from a result I caught first

**Keep the control group.** Record every candidate the judge considered, flagged or not, and measure whether the flagged set goes wrong more often than the unflagged set. In the library this is ClaimLedger, and the headline number is discrimination, not accuracy.

What I caught first: a judge that flagged 19 of 21 items. Its accuracy looked fine. It was a mood, not a prediction: ask "is this at risk?" about something that's already open and the answer is always yes. I replaced the yes or no with a probability, and made the flag line a threshold I own.

**Unknown is an answer.** Whatever checks what really happened is allowed to say it can't tell, and that is never scored as a failure. In the library this is the unknown outcome and coverage.

What I caught first: a run reported no signal, and then coverage showed that 11 of its 13 failure labels really meant "we never looked there". A promise settled in person scored the same as one dropped on the floor. The negative result was a statement about the observer, not about the judge.

**Put the leak guard in the API.** The judge never receives the world, only what evidenceAt(subject, asOf) hands it for a past date. Leaking the future then has to be deliberate.

What I caught first: a candidate query that, given a past date, only ever returned subjects who never came back. That population has a 0% event rate by construction. It reported 0 of 60 across every configuration, so the answer was fixed before the judge said anything.

**Refuse to certify.** Below the evidence bar the report says NOT ENOUGH EVIDENCE instead of giving a percentage, and it counts distinct subjects, not rows. In the library this is sufficiency.

What I caught first: a +18 point lift over 19 judgments, which was really 7 subjects. And a sign that flipped across three runs on overlapping data (-0.03, -0.06, +0.07), which is what noise looks like.

**The evidence must not shrink.** Stores merge per row instead of rewriting the whole file. In the library this is mergeWriteJsonl.

What I caught first: a daily cycle wrote ten predictions, deleted them, and reported success. Three stores were open at once, each rewriting the whole file, and the last one won. Nothing crashed, and every metric afterwards stayed well formed. NOT ENOUGH EVIDENCE had become impossible to tell apart from "the evidence was deleted".

## How it works

Here is the whole path a claim takes, from what the judge is shown to the verdict at the end.

**What counts as a leak?**

A backtest puts the judge at some earlier date and asks it to predict what happens next, when I already know what did happen. A leak is any detail in what the judge sees that wasn't available on that date, such as a comment count fetched today. It doesn't cause an error. It just makes the judge look better than it was.

Every candidate the judge considers goes into a ledger, flagged or not, because the unflagged rows are the control group. A claim can't be written without a falsifier, which is the observation that would prove it wrong, and a check date after the as-of date. Otherwise the write throws.

Reality then answers each claim with confirmed (the predicted bad thing happened), refuted or unknown. The thing that answers is an oracle I supply. Unknown, and anything still pending, stays out of every rate.

**What is the oracle?**

It's the part that answers "what actually happened?" for a claim and its check date. Sometimes that's a plain lookup, like whether a ticket's closed date comes before the check date. Sometimes it has to dig through messy records and honestly can't see. For those cases it can answer unknown, and it can mark a verdict as resting on having seen nothing.

Figure: Diagram. How a claim moves through it. Backtests return the claims as an array. ClaimLedger is the live path, where I call resolve() later. There is a fifth, optional check, versusBaseline, that compares the judge with a cheap heuristic. It isn't drawn.

The report asks four questions, in a fixed order. Is the oracle actually observing things? Do the raw scores separate the outcomes at all? Does the flagged set fail more than the unflagged set? Are there enough distinct subjects to say so? I put them in that order because asking them the other way round is how a broken oracle gets reported as a null result about the judge. I ran into exactly that.

**What do separation and lift measure?**

Separation ignores the flag line completely. It's the average score on things that turned out fine minus the average score on things that went wrong, so if it's near zero then no flag line anywhere can help. Lift comes after you draw a line: how much more often the flagged items went wrong than the ones you let pass, in percentage points.

There are two ways in. The live way is ClaimLedger: I write claims as they happen and call resolve() once reality can answer. The other is backtest, which stands the judge at past dates across history I already have and grades it today. Before spending a model call on that, baseRate checks that the outcome actually varies, because if it happens 100% of the time every predictor scores the same. The backtest never grades a claim whose check date hasn't arrived, skips subjects that were already settled, and counts everything it excludes, because a silent filter is how a sample gets rigged.

Figure: Diagram. The leak guard in my GitHub example. An issue's comment count and labels are today's values, so handing them to a judge standing weeks in the past would quietly improve its score.

One more piece is a small check on the claim "no single input caused this". isRealJoin wants at least two strands of at least two different kinds, more than a day apart, each dated and sourced. A candidate that fails is dropped, not downgraded, and if that leaves nothing today then nothing is the honest output. The precedent is a derived-state signal I'd shipped as "unfakeable" that turned out to be produced by a string-truncation bug. I deleted it instead of patching it.

## Results

There are two example scripts. The first is a world of 60 made-up support tickets. 14 of them never get closed; the rest are closed 5 to 24 days after they are opened. Every prediction asks the same thing: will this ticket still be open 14 days later? Predictions are made every 21 days along a ticket's life, at most three per ticket, which gives 88 per predictor. Three predictors take part. The alarmist flags everything. The coin gives a random score. The judge reads how long the ticket has been sitting, with score = 1 minus days open divided by 60, and flags anything below 0.35.

I put the 88 predictions per predictor into the slider below. If you have scripts off, the table shows the setting I used, a flag line of 0.35.

| Predictor | Flagged | Open | Not flagged | Open |
|---|---|---|---|---|
| Alarmist | 88 | 57 | 0 | 0 |
| Coin | 31 | 18 | 57 | 39 |
| Judge | 30 | 28 | 58 | 29 |

### Move the flag line

The judge holds a lift of between +34 and +57 points at every flag line the harness will certify, from about 0.06 up to 0.88. Push the line further right and the unflagged side drops below 8, and it refuses. The coin's lift is zero or negative at almost every setting. At 0.16 and 0.17 it shows +2 points, which is pure noise, and the separation line still reads backwards. The sufficiency check is a count and not a significance test, so it can't catch that by itself. The separation line is what does. The alarmist, wherever you put the line, has nothing on one side, and the harness says NOT ENOUGH EVIDENCE.

Figure: Diagram. The same results at the flag line I used. The first three are the 60 made-up tickets. The last is 206 real issues from the Next.js repository, 527 predictions in all.

The point of the first example is that the alarmist has the highest raw catch count and is worth nothing. With nothing unflagged it has no control group, so the harness refuses to score it. The coin is backwards, with separation -0.08. The judge shows a +43 point lift, but I built this world so that ticket age predicts stalling, and then gave the judge ticket age. It shows the harness can tell the three apart. It doesn't show that any real judge works.

verdict     NOT ENOUGH EVIDENCE — 88 flagged / 0 unflagged resolved. Any lift here is anecdote.verdict     NO SIGNAL — flagged 58% vs unflagged 68%. The selection carries no information; subtract it.verdict     LIFT +43pp — flagged 93% vs unflagged 50% (flat baseline 65%), over 60 subjects.

The second example uses real data. I pulled 400 issues from the Next.js repository, opened between January and April 2026, and 206 of them were usable: most of the rest had already been closed within three days of opening, before the first prediction date, so there was nothing left to predict. That gave 527 predictions, with the question "will this issue still be open in a fortnight?" The oracle is a lookup, whether closed_at comes before the check date, so a negative verdict is much more likely to be about the judge than about the instrument. The judge is a hand-weighted function. Age dominates, a reproduction in the issue helps, a very short report helps, and an outside reporter makes it linger. It's up against an incumbent, a one-line heuristic that only looks at age, and the comparison flags the same number of issues for both so neither wins by being trigger-happy.

coverage    527/527 observedsignal      separation +0.15, right direction: the estimate carries informationselection   flagged 160/165 · unflagged 320/362 (control) · 206 subjectsverdict     LIFT +9pp — flagged 97% vs unflagged 88% (flat baseline 91%), over 206 subjects.baseline    too close to call (9pp vs 9pp) — a model that only matches a cheap heuristic is not earning its cost

So the judge has a real signal and a real lift, and the one-line heuristic gets the same lift for free, which is what the tool reports. The lift is also small against a high base rate: both groups stay open about nine times in ten. I find a result like that more useful than one that flatters the judge.

The first run of this example was refused. I'd pulled the newest issues, which spanned twelve days, and the horizon is fourteen, so no check date had passed and the harness wouldn't score an empty set. Reaching for older issues isn't a workaround. You can't grade the future.

**Why count distinct subjects, not rows?**

If I ask about the same ticket every three weeks, the answers aren't independent, since a ticket that's stuck now is probably stuck next time too. A hundred rows about seven tickets is closer to seven experiments than a hundred. So the harness bounds what it will say by the number of distinct subjects.

## What I ran into, and the limits

The leak guard is an API shape, not a guarantee. A determined caller can reach around evidenceAt, and all the library does is make leaking deliberate rather than accidental. The merge-on-write storage can't express a real delete, so it's safe for stores that mark rows closed or superseded and wrong for one that needs to remove them. The sufficiency check is a count, 8 per side and 8 distinct subjects by default, and those two defaults are rules of thumb you can change. It is also not an accuracy benchmark. There's no golden dataset, and ground truth is whatever actually happened. If your system produces four gradeable items a month, it will tell you that, every time you ask.

Both examples score with hand-written functions, and grading a language model with the harness is the next test. When I pulled it out in July 2026, it reported NOT ENOUGH EVIDENCE about the central claim of the system it came from. That's the intended behaviour, and it's why I trust it.

## Where it stands

The GitHub example only uses fields fixed at creation time. Comment counts and labels change after an issue is opened, so using today's values would be cheating. The tests cover the write-time guard, the unknown handling, the subject count, the storage merge, the leak guard and the join check. Rebuilding those fields as of each date from the issue timeline is the next piece, and it costs an API call per issue. The library is not published as a package.

- **Stack**: TypeScript, no runtime dependencies; tests run with tsx
- **Modules**: ledger, backtest, metrics, join (cross-reference check), jsonl (merge-on-write store)
- **Status**: Prototype, version 0.1.0
- **Code**: Not public