The agent said "Done". Why AI coding agents ship broken systems, and what actually stops it

A coding agent is very good at the file in front of it. It reads the function, changes it, runs the tests and writes "Done". The trouble is everything the file does not show: the other program that reads the same value, the Python version the agent did not run, the check that could never fail. Most of the broken things our agents have shipped were not bad code. They were correct code in a system nobody looked at as a whole. Here are five ways it goes wrong, all from one release day of Crossweft, and the check that caught each one.

1. The other side of a seam

A value written by hand in two places is a seam: a route the client calls and the server registers, a version string, the fields of a DTO, the action name on a button and the table that dispatches it. Change one side and every local check stays green. The compiler sees one language at a time, and the tests of each component test that component.

An agent has no reason to open the other side, because nothing tells it the other side exists. That is the idea behind Crossweft and our first post about it, Every hand-written seam needs a guard: write the seams down as a map, give each one a guard that reads both sides, and tell the agent after every edit which file on the other side to re-read. The newest guard follows every button to its handler and its API route; we wrote about what it found in A button that calls nothing still compiles.

2. A check that cannot turn red

A green check proves only the shapes someone thought of. In review before the release, an independent reviewer gave the new button guard a map we had not imagined: a handler that builds its URL as "/v1/" + kind. The check passed it, because a path that ends in / was read as "any route under /v1/". Every test was green, and the author's own probes were green, because they tested the shapes the author had in mind.

The fix was small. What mattered was the rule behind it: every check needs a negative control, an input that must make it fail. If you cannot make your check go red, you do not know that it works. We now write the red case first, watch it fail, and only then fix.

3. The environment the agent did not run

The release passed every test on the machine that built it. The public CI then failed on Python 3.9 on all three operating systems: 17 errors, one cause. A test helper called Path.write_text(newline=...), and that argument only exists from Python 3.10. The agent's machine has 3.12 and 3.14.

It was the second time this exact mistake reached our CI, so this time it became a test: a scan of the source tree that fails on any write_text( call with newline=. An agent remembers nothing between sessions. A repository can remember, but only in a form that runs.

4. Grading your own work

The author of a fix is the worst person to say the fix is complete. The author tests the model of the problem they already hold, and so does every probe they write. In the review above, the author's checks agreed with the author. A reviewer in a fresh context, with only the code and no explanation, found the gap.

So our rule is mechanical, not polite. On anything that will be hard to undo, a separate agent reviews the change before it ships: a fresh context, no access to the author's reasoning, and a verdict of PROCEED, HOLD or REJECT. Before this release that took four verdicts over two days. Two of them were HOLD, and the release was better for each.

5. "Done" as a word

An agent stops when it decides it is finished, and its word for that is "Done". Asking in the prompt for more care helps for a while and then fades as the context fills. A check that runs when the agent tries to stop does not fade.

The Crossweft stop hook runs the seam check before the agent may finish, and blocks the stop while a seam disagrees. Our other hooks do the same for red tests and for a handoff that hides unfinished work. They still caught gaps at the end of that same day, after the release: an old version of the tool installed for the wrong user, a skill file not regenerated after the upgrade. Each one was a minute to fix, because the hook named it.

What the five have in common

Each of these failures was invisible from the file the agent was editing, and each fix moved the knowledge out of the agent's head into something that runs: a map, a negative control, a CI matrix, a fresh reviewer, a hook. We wrote more about that move in Three ideas from agent news, turned into checks.

This has a price. A map has to be kept up to date, and a review round takes time. Static checks prove wiring, not behaviour. But the alternative is finding out from a user that the button does nothing.

If you work with coding agents across more than one language or component, the box beside this post has the four commands to start: install, initialise, find the seams that are already there, and let your agent map the rest.