One critic has exactly one blind spot
4 min readT. Krause
- evaluation
- critics
- gauntlet-graph
Give an agent a builder and one fresh critic per piece of work and the output gets markedly better. That is the Gauntlet Loop, and it works. But run it for a while and the failures start to rhyme: whatever the critic was not looking for is what survives.
This is not a criticism of the critic. It is arithmetic. A reviewer asked to judge a landing page against "as good as Linear's site" will form an opinion about typography, spacing and copy density, hold the page up against that opinion, and report the biggest gap. It will not notice that the page takes four seconds to paint on a cold cache, because nobody asked and because it never loaded the page twice.
Attention is the scarce resource, not capability
The obvious fix is to ask for more: also check performance, and accessibility, and whether the copy survives translation. Try it and you will watch the critic do the first thing properly and the rest in passing. A critic told to check five qualities produces one real judgement and four sentences that sound like judgements.
That failure mode is worth naming precisely, because it looks like success. You get a report with five sections. Four of them are confident. Only one of them is grounded in anything the critic actually did.
Narrow critics, arranged
The Gauntlet Graph's answer is not a better critic. It is more of them, each deliberately crippled: one node, one quality dimension, nothing else. A critic that has been told it is responsible for keyboard traversal and nothing else will drive the whole interface with a keyboard, because that is all it has to do with its context window.
The arrangement matters as much as the narrowness. The nodes form a directed graph, and a piece is only done when it traverses the whole graph clean in a single pass. Not four of five. Not "passed with minor notes". Clean, in one run.
That rule sounds harsh until you have watched the alternative. Partial credit is how a fix for trial four silently breaks trial two, and how the report at the end says both passed — because they did, separately, at different times.
The reset is the expensive part, and the point
Any failed trial sends the single biggest remaining gap back to the builder and puts the piece back at the graph's entrance. Every earlier trial runs again.
This is the most expensive rule in the method and the one people most want to relax. Don't. The whole class of bug it catches is the one where a change made to satisfy a late critic violates something an early critic already approved — and there is no way to catch that except by asking the early critic again.
It also means your cost is not linear in the number of trials. A piece that fails its fifth node re-runs five nodes, not one. Budget for that before you start a run, not while you are watching it.
Why the graph grows
A fixed set of trials is a fixed test suite, and anything measured against a fixed test suite eventually games it. Not maliciously — just by gradient descent. The builder learns the shape of what gets approved.
So after every clean traversal, a fresh graph engineer inspects the piece, identifies its weakest surviving quality, and adds exactly one harder node aimed at it. Nodes are never removed and never softened. The bar can only ratchet upward.
The practical consequence is that a piece has no natural stopping point. It can keep improving until you decide it is good enough. That is a feature if you are watching, and a way to burn a weekend's budget if you are not. Stop pieces individually as they clear your actual bar rather than waiting for the run to finish, because it will not finish.
When one critic is the right answer
None of this makes the Loop obsolete. If the work has one dominant quality — a piece of prose that either sounds like the writer or does not, a function that either passes the suite or does not — a single critic against a sharp bar is faster, cheaper and just as correct.
The graph earns its cost when the ways a piece can be wrong are genuinely independent of each other. A dashboard can be beautiful and unusable. An API can be consistent and slow. A game can be polished and boring. Those are not one judgement with several parts; they are several judgements, and the moment you hand them to one reviewer you have chosen which of them gets taken seriously.