Blind, or It Does Not Count

For two weeks my system was winning. Then I found the reason, and it had almost nothing to do with the system.

The thing being tested is a design-reasoning skill. It forces a coding agent through a slower loop. Frame the problem. Structure it. Compose it. Critique it. Leave an artifact at every step.

The claim is that this produces better interfaces than letting an agent go straight to code. Claims like that are cheap. Everyone selling an AI tool makes one. So the project grew a harness: competing builds, blind judges, gold-standard reference screens to compare against.

The early results were favourable. They were also wrong.

The brief was doing work nobody asked it to do

Each reference screen carries a style register. Quiet, or dense, or playful, whatever the original designer chose. The briefs never told either builder what that register was. It had not occurred to anyone that it mattered.

It mattered enormously.

A build that landed near the reference's register read as better to a judge. A build that landed far from it read as worse. Layout, hierarchy, and copy barely entered into it. The comparison was not measuring craft. It was measuring a coincidence of tone, and the disciplined loop produced that coincidence more often.

The fix is small. Supply the register to both arms as neutral brand facts. Then both builders know what they are writing toward, and the judges are left comparing execution.

The fix is a few lines. What it implied was expensive. Every prior round had measured the wrong thing. Two weeks of favourable numbers had to be thrown out and re-run.

I re-ran them. Not out of unusual virtue. The whole premise of the project is that self-certified quality is not quality. A result that needed squinting at would have made the project the exact thing it argues against.

Then a builder admitted it had been reading ahead

The second problem was not subtle.

One of the baseline builders disclosed that it had read archives from earlier rounds. The baseline arm is supposed to represent what you get without the discipline. A baseline that has seen how previous comparisons went is not a baseline. Six builds were contaminated.

There was a cheap option available. Split the invocation. Argue the exposure was marginal. Keep the numbers.

I superseded all six and rebuilt them under confinement instead. Contamination is unquantifiable, which is the actual problem with it. And a record with one convenient exception in it is not a record.

The tooling was lying too, and quietly

Then a crop flag turned out to have never worked.

Reference screens were supposed to be cropped from the top. That is where the decisions live: the header, the primary action, the first band of content. The command-line flag doing the cropping silently did nothing. Every gold image generated for weeks was cropped from the center instead.

No error. No warning. Just a different picture than the one everyone believed was on the table.

That one is the most instructive, because no bad judgment produced it. The documentation was read. The command was written. The command reported success. Catching it required looking at the output as an image rather than as a return code.

A related transform, meant to strip a footer strip, turned out to be lossy. It destroyed working-tree evidence on four screens before a dry-run step went in. That is the tax on speed inside a measurement pipeline. The pipeline is code. Code nobody has attacked is code nobody knows.

What the numbers said once the method held

The last matched run produced twenty builds and thirty blind review forms.

Blind preference came back 24 to 6 in favour of the disciplined loop. Eight of ten scenarios. Mean weighted score 124.1 against 113.2. Sixteen builds cleared the ship gate against four.

That result is believable in a way the earlier ones were not, and the margin is not the reason. The reason is that the comparison's controls are now known, because they had to break twice before anyone could see them.

One number matters more than the preference split. Unsupported claims came out at parity. An earlier round had the disciplined loop generating more of them, which is exactly the failure you would predict. Give a model a box labelled justification and it will fill the box. Watching that regress and then correcting did more for confidence in the gate than the win rate did.

The part still open

Fourteen of the thirty review forms flagged that builds might have been identifiable. Not from the design. From residue in the code: comments naming the rules the loop applies, which a judge could read as a signature.

I have not fixed it yet. Until I do, the 24 to 6 carries an asterisk, and I would rather say so here than let the number travel without one.

Possible unblinding is not unblinding. The effect might be nothing. It might also be the third time this harness has flattered its author without asking permission, which is roughly the base rate so far.

What the exercise actually taught

The goal was to measure whether a design process works. Most of the time went to discovering ways the measurement was tilted. Every tilt pointed the same direction.

That direction is not a coincidence, and it is not dishonesty. You build the apparatus using the same assumptions that produced the thing being tested. So the apparatus inherits them. A confound you would spot instantly in someone else's experiment is invisible in your own, because it looks like the setup.

Which is the argument. Not that blind evaluation is a nice discipline for teams with spare time. That an unblinded judgment about your own work carries so little information that acting on it is closer to guessing than to knowing.

You would not accept a vendor's benchmark of their own product. Do not accept yours.

Grade it blind, or admit it has not been graded.