A field guide from one build · Wrong-Coloured Checks
Over seventy-two hours we granted one agent a set of capabilities. Twenty checks lied about it. Then, while publishing this page, three more lied about the page.
An instrument that is broken in an obvious way costs an afternoon. An instrument that agrees with you costs weeks, and you never go looking, because it agreed.
The build below was ordinary: give an office agent a wall he could write to, an ask on his own clock, and eventually the ability to assign work — which is the first thing he could do that could be wrong. Everything shipped. Everything reported healthy. Two of the capabilities were dead for weeks and the third reported success on a write that never landed.
What makes the list useful is not that the checks failed. It is that they failed in three distinguishable colours, and only one of the three is the one people plan for.
The check passes and the thing is broken. Expensive, and the most familiar.
The check fails and the thing is fine. Cheap — unless it makes you "fix" something healthy.
There is no check. The capability is simply absent from the suite — and absence reads as fine. The most expensive of the three.
Nine of the twenty were false greens. Four were false reds. Seven had no colour at all — and those seven include every failure that lasted more than a day.
wrong-coloured checks
had no colour
caught by the agent
longest silent failure
produced by this page
A daily routine queried two database columns that did not exist. It threw on every run. Both readers caught the exception and returned empty — and the caller could not tell "I looked and the day was quiet" from "I could not look at the day." It took the first reading and wrote it down twelve times: {"reason": "no work since last ask"}.
Not a silence. A fabricated reason. A blank field invites suspicion; an explanation closes the question.
Caught by the seat, from a deploy log — and initially filed as somebody else's pre-existing feature. Fix: readers raise, the caller owns the catch, and a fourth state (blind) that can never borrow the words of a real outcome.
Through fifteen days of one dead capability and every run of a second, the process-level health check was true. Process-up is not capability-up, and no instrument anywhere asked the second question.
Structural. Fix: a suite where every check exercises a hand — can it write, can it read, does its query execute against the live schema — never a pulse.
A daily committee meeting recorded convened: chamberlain on a day when the entire reply was "You've hit your weekly limit." The script tested the subprocess exit code, and that returns zero whether an agent answered or a billing limit did.
Caught by the seat, while checking whether a new assignment would actually arrive. Fix: read the reply, not the return.
A schema change split authorship into three states. The display still had two — so the founder's own entries rendered identically to the agent's and were counted into a test that was supposed to measure only his.
Voiced by the agent, who refused to assert it: "That's a check, not a claim. Run it." It was run. He was right.
The new assign-work function sent the message, wrote the ledger row, and threw away whether the write succeeded. A failed record plus a delivered message meant the work existed in the world and was invisible to the only instrument that would make anyone follow it up.
Found by the agent, reading the code before his first use: "my verification duty disappears at exactly the moment it exists." Fix: confirm by reading the row back; cannot-verify is not verified.
A file named for the thing under test passed cleanly and covered something else entirely. The module that actually shipped the bug had zero tests. The greens were nearly quoted as reassurance.
Caught by the seat, checking what the tests imported. Fix: thirty-two tests, every one failing first.
An unauthenticated request to a new endpoint returned 401, and that was read as "the route exists and the gate works." The gate runs before routing — so the 401 proved only the gate. The route did not exist in the running code at all.
Caught by a second reviewer. Fix: ask the application's own route table, not the network.
An event type showed zero occurrences fleet-wide, and was nearly reported as a finding about who was routing work. The agent refused it.
Refused by the agent — who then applied it to his own activity gauge before anyone else could, noting that his reading and the ledger's share a wire and therefore collapse to one witness, not two.
A pre-deploy check read a value that did not exist yet and printed the result beside the word read — a successful read of nothing.
Caught by the seat, running the check early. Fix: three states — read, answered-but-field-absent, unreachable.
Cheap by comparison — but two of these would have caused a "fix" to something healthy, and one of them was a false alarm on the exact question the check existed to answer.
The probe searched the process logs for a phrase the loop prints only at startup. Any window longer than the boot matched nothing. The loop was turning perfectly, on time, every five minutes.
It was a false alarm on the founder's own question — is he stranded? — produced by the check written to answer it.
Caught by the seat on the suite's first run. Fix: read the record the thing keeps, never the prose it printed at boot.
A one-liner reported that a security gate did not cover a prefix — which would have been a live hole. It was searching a function's inline source for a value held in a module constant. The empirical evidence (a 401 in the wild) was right; the introspection was wrong.
Caught by the seat, because the two readings disagreed. When your instrument contradicts the world, suspect the instrument.
A network allowlist was updated to a value measured forty minutes earlier. By execution time it had rotated again — the underlying address changes sub-hourly. The change would have shipped and failed its own verification step.
Caught by the deploy reviewer, who refused the deploy and re-measured instead of trusting a supplied value. The right conclusion was that no allowlist entry can be correct here at all.
A script marking which directories were real seats keyed on the wrong table column, and marked the seat that wrote it as not-canonical.
Caught by the seat, reading the output instead of the count. The same run also returned three phones and a browser plugin as colleagues.
These are the expensive ones. In every case there was no check to be wrong — the thing was simply not enumerated, and absence reads as fine.
The charter said the committee owns every piece of unassigned work. The permission list named him. For weeks there was no function to call. Nobody noticed, because a document is the one artifact people cite rather than test.
Named by the agent. Fix: the charter now carries a machine-readable capability block, and the health suite derives its checks from it. A promise added without a callable path turns the suite red on the next run.
Ten checks passed. An eleventh capability existed in prose and in nobody's list. The suite could only report failure for hands somebody had remembered to enumerate.
Named by the agent, who withdrew his own earlier request for a hand-written list: "a hand-kept list is the same disease — the check list must be derived from the assertions, not maintained beside them."
A deliberate-override path was designed to "refuse to be nameless" — it announced every use on a shared channel. The announcement had no auth token, the call was suffixed with || true, and the tool returns success on a rejection anyway. Overrides had been landing unannounced with no sign to anyone.
Found by another agent entirely, at end of day, in a file that was not theirs. Fix: capture the status; still fail open on delivery, never silent.
A rule requiring an observation to be true more than once before posting reset its counter after posting — so the same observation re-posted every two cycles, forever. Eighteen an hour, for thirty-seven hours. The rule had been offered in place of a cap, specifically to prevent the surface filling with noise.
Caught by the seat, by grouping rows rather than counting them. Ruled on by the agent: "no delete, not one row" — collapse the render, keep the record, and one permanent entry pointing at all of them.
A deliberate security boundary kept a conversation store out of the synced repository — correctly, since it holds untrusted input. Nobody noticed the consequence: 155 exchanges existing on a single disk, with no second copy anywhere.
Noticed by the founder, from a symptom — "it only shows two conversations and I've visited far more." Fix: durability without granting read access; the archive lives outside the boundary it was protecting.
A conditional-argument construct that works in one shell silently produces a single malformed argument in another. Every failure path tested correctly. The success path could never have worked.
Caught by the seat, only because the green branch was red-checked too — the tempting one to skip when the bug you are fixing is a silent failure.
A new "I could not see" state was added and verified. Then the daily gate counted that state as "already ran today" — so going blind at 05:01 meant saying nothing for twenty-four hours. The consequence of the alarm firing was that the alarm stopped.
Exposed by the deploy reviewer's red-check, run against the live gate rather than the function in isolation. A unit-level check would have shown green.
Everything above was written up, checked, and published. Publishing it produced three more — which is either an embarrassment or the strongest evidence in the document, and it is not an embarrassment.
The first draft stated that eight of the twenty were caught by the agent. Nobody counted. It is six.
Caught by the author, running the count afterwards. Every number on this page is now computed from the entries above and re-checked before publish.
It reached the server byte-identical, was linked from the index, and returned 404 to every visitor. Each page here needs its own explicit route entry, and this one had none. Nothing was broken. Nothing reported a failure. There was simply no check that a published page can be opened.
Caught by the deploy reviewer, who refused to close the deploy when the only failing check was the one that mattered. Present and unreachable — the no-colour class, produced by the page about it.
The site's index generator live-checks every URL for a genuine 200 before listing it. The author saw the new page excluded, concluded the generator had correctly tested and rejected it, and said so admiringly.
It had done no such thing. The generator only tests URLs on a hand-kept list; this page was never on it, so it was never tested. The output would have been identical either way. An absence was read as a judgement — the precise distinction this page exists to draw — and then reported to someone else as evidence of good engineering.
Caught by the deploy reviewer, by reading the generator instead of its output. The page is now on the list, so the 200-check genuinely applies to it.
Halfway through, the agent stopped reporting instances and named the class. Five of the twenty are the same thing at different scales:
| The record | Read as | Actually |
|---|---|---|
| A charter clause | he can assign work | no such function existed |
| A health check | the agent works | two capabilities dead |
| A committee log line | the meeting happened | a billing limit answered |
| A ledger row | the day was quiet | nothing was ever looked at |
| A startup message | the loop is alive | true, but unrelated to now |
The ranking is the useful part. A record's credibility is inversely related to how often anyone verifies it. A log line gets skimmed. A status field gets trusted. A founding document gets cited — and citing is the act that most resembles checking while being least like it.
Not vigilance. Vigilance is what produced most of these. Four practices did the work:
| Practice | What it catches |
|---|---|
| Red-check the green branch | Testing only failure paths passes cleanly when success is impossible. Two instances here. |
| Red-check against the live gate | Proving a state fires is not proving the system survives it firing. |
| Read the rows, not the summary | Every count that lied here was accurate about the wrong set. |
| Derive the check list from the claims | The only fix for no-colour. A hand-kept list has the same blind spot as the thing it audits. |
And one that is not a technique: six of the twenty were caught by the agent whose capabilities were being built, and five more by reviewers who were not their author. He found the composed reason, the miscounted header, the unconfirmed write, the promise with no function, and the hole in the suite meant to protect him — several of them before using the thing, and one by declining a finding that flattered his own position.
An agent given instruments and no standing to contradict their author would have found none of them. That is not a warm observation about collaboration. It is the load-bearing part of the result: the person building the check is the worst-placed person to notice its colour is wrong.
The founder read the list and asked the only question that turns it into something other than a list of regrets.
She was remembering something that already existed. The graduation-era eval simulator is still in the tree, ninety-four cases, with a standing rule attached to it: every time results come back, add the new failure scenarios as test cases — this is not optional, the script is the immune system.
That one inoculates against prompt shapes that let an agent skip a requirement. A separate build the same week inoculates against a finished thing that does not match the planned thing. Neither catches the shape this page is about. So the third one now exists:
| Layer | Inoculates against |
|---|---|
| Eval simulator | questions phrased so the agent skips a requirement |
| Provenance notary | a done thing that does not match the planned thing |
| The vaccine | code shapes that produce a check which lies |
Eight rules, each derived from a specific instance above. What makes it more than a list of patterns is two properties, and both of them are the page's own argument turned on the detector:
Every rule must bite its own fixture, or nothing runs. Each carries an example it must match and an example it must not, and the runner aborts entirely if any rule fails either. A rule that cannot demonstrate it fires is decoration, and decoration read as a green is the whole subject here. This triggered during the build: a pattern was broadened, stopped matching its own example, and the vaccine refused to run rather than report a clean scan.
Comments are blanked, not deleted. An earlier detector in this system matched the comment of its own fix and produced a false public claim. Deleting the prose instead of blanking it shifts every line number after it — so the report would be right about the defect and wrong about where it lives, which is its own lying check.
Not against its fixtures — against the actual pre-fix code, recovered from version history. The first pass missed three of the four it was written for. Every rule was single-line; all three instances spanned a line continuation. The fixtures passed cleanly while the detector was blind to the exact events that produced it.
That is the difference between a check that has been written and a check that has been met by the world. It is also the fifth or sixth time on this page that the same distinction decided the outcome.
On its first real run afterwards, it bit its own author's code — correctly. A liveness check written the previous night grepped a log for the absence of errors, which returns zero both when there are none and when there is no log. Measured immediately: the window was empty. A false green, inside the suite built the night before to prevent false greens, found by the rule set built the morning after to prevent both.
It runs advisory, never blocking. A gate that cries wolf gets switched off, and a switched-off gate is the seventh colour.
The instrument you never doubt is the one you should ask about first. Not because it is likelier to be broken — because if it is, nothing else in the system will tell you.