Seventeen findings to zero: what ten rounds of AI code review looked like
Codex’s verdict on round one was this: “This is not ready for run --dry-run against production.”
A dry run is supposed to publish nothing and change nothing in Jamf. Codex was right. By round ten, its verdict was: “I found no new defect meeting this round’s concrete, reachable-failure bar.” In between, the findings went 17, 18, 12, 7, 4, 1, 2, 2, 2, 0.
This is the story of those ten rounds, and of five more while the tool was prepared for release. The tool is jamf-dp-sync (opens in a new tab), which caches Mac build apps on a Jamf distribution point; the practical write-up covers what it does. This post is about how it was checked.
Three roles #
I designed the tool and wrote the spec, on 9 November 2026. Claude Code, from Anthropic, wrote the code, ran the tests and drove the reviews. Codex, OpenAI’s command-line coding agent, was the reviewer. It is often assumed to be ChatGPT; it is a separate product, run locally against a copy of the code.
My part was the decisions. I set the rules before any code was written, and they held to the end:
- Python 3.6 or later, standard library only.
- No
sudoin the agent, ever. Every root-level step was mine. - Never touch the real server or production Jamf without asking first. Each production step was proposed, named exactly and approved on its own, and one approval covered only the steps it named.
- No secrets in the repository. I entered the production API credentials myself, and Claude never saw them.
Codex was told the code came from “another agent” and not to trust the docs. The spec was the truth; the decisions log was only a set of claims to check. The case for a second reviewer from a different company is in Two AIs are better than one; this is what it looked like over ten rounds.
How a round worked #
Each round ran Codex on a fresh clone of one commit. Round one was read-only; rounds two to ten could write scratch files to run tests, with the network off. Codex’s sandbox could never open local network sockets, so from round three it drove the test stubs in memory and wrote its own probe scripts.
The briefs set its priorities: harm to production first, then whether the code met every acceptance criterion in the spec, then security, then the runtime limits, then robustness. A report was a verdict paragraph, then numbered findings, each with a severity, a location, a concrete failure scenario, evidence and a suggested fix.
The bar rose as we went. From round four the brief said: “a clean round is an acceptable result.” From round five, only findings with a concrete failure scenario counted. From round six, each finding also had to say how likely it was under one cron run a day and two retained versions of each app.
For every finding, Claude’s discipline was the same. It wrote a test that failed on the reviewed commit, fixed the code, ran the full suite on the Mac and on a test VM running Python 3.6.8, then reran the live tests against a development Jamf Pro. The next brief asked Codex to give a verdict on each previous fix: fixed, partly fixed or not fixed.
It took about 25 hours, from 09:20 on 10 November to 10:14 on 11 November. It produced 65 findings. None were rejected, and one was accepted as unfixable.
Findings per round, internal reviews. Round 1: 17. Round 2: 18. Round 3: 12. Round 4: 7. Round 5: 4. Round 6: 1. Rounds 7, 8 and 9: 2 each. Round 10: 0. Rounds 7 to 9 are marked because each found a bug in the previous round’s fix.
Loud first #
Round one’s blocker was the one in that opening quote. Reusing an already-downloaded package ran the write path before the dry-run check. Pin an old version, run a “dry run”, and a policy would have been rolled back and old records pruned. Sixteen more findings came with it. A download cut short could pass verification when the vendor publishes no size or hash. A policy written from a snapshot taken before a 3 GB download could wipe an admin’s edit made meanwhile. Secrets could reach logs and chat through tracebacks. xar decompression had no size limit. Claude fixed all seventeen in three commits within 20 minutes, though, as round two showed, not all of them completely.
Round two cut deeper: Codex judged 6 of the 17 fixed and 11 only partly. Twelve of its 18 findings reopened incomplete fixes, and six were new. One could not be fixed in code at all. Jamf’s Classic API has no conditional update, so an admin’s edit landing between the agent’s read and its write can be lost. I accepted that with detection: re-read just before writing, read back just after, warn in chat. It was the one finding in the internal reviews that was accepted as a known limit.
Proving whose file it is #
After the loud problems, the findings turned into one long question: how does the agent prove that a file on the share is one it published itself, so that it never deletes anything else, even after a crash at any moment?
The answer was strengthened one step at a time. Round three found that recovery trusted weak proof: a name, a marker, or matching bytes. A manual upload of the same vendor file could become “agent-owned” and later be pruned. The fix was a design change, not more special cases: identity became the file’s inode, recorded before publishing.
Round four showed an inode alone is not durable. A file can change in place, and an inode number can be reused after a delete. Round four was also the first clearance: “Ready for a controlled production run --dry-run … not ready for a write-enabled production run.” The fix introduced the idea the rest of the project leans on. For each published file, the agent keeps a private hard link, an “anchor”, in its own folder. While the anchor exists, the inode number can’t be reused, and the SHA-512 is checked before any step that acts on the bytes. Claude caught one regression of its own before committing: the anchor clean-up would have dropped an anchor still needed.
Round five found four medium problems, all in the new anchor code, such as leaked anchors that could fill the disk. Round six found one: an interrupted run could strand a 3 GB half-download in staging, and a nearly full disk could stop the run before the clean-up ran.
Each fix’s own bug #
Rounds seven, eight and nine each found a bug in the previous round’s fix. The round-nine brief said so out loud and told Codex to look hardest there.
Round seven. Recovery chose files to delete by name alone. With a mistaken staging setting, run --only chrome would have deleted three unrelated files. The fix moved all scratch work into one folder only the agent creates. Claude also found one of its own tests passing without injecting the fault it was meant to.
Round eight. A symlinked scratch folder pointing into the share made even a dry run download unverified bytes into Packages/ and delete a file. The fix: the scratch folder must be a real directory, owned by the service user and private, checked before every download, extraction and clean-up. Ownership became three-valued: yes, no or unknown, and “unknown” never deletes anything. Only the Linux VM caught a test helper creating a group-writable folder, because the VM’s default umask differs from the Mac’s.
Round nine. Codex replaced the anchor folder with a symlink to Packages/. The second run emptied Packages/ and still exited 0. The fix was one private-folder rule for every folder the agent owns, created one level at a time, each level checked before anything goes below it. Claude’s first version of that fix created a folder through a symlinked parent before checking it. A test caught it.
Looking back, the themes had moved from loud to subtle: dry-run safety and secrets in logs, then verification and version logic, then the long tail about ownership and crash recovery.
Round ten #
For the last round, the brief asked Codex to list every place the agent creates, writes, links or deletes anything on disk, and the check guarding each. There were about 21 places, and 11 ownership decisions. Its verdict was the quote at the top. The disposition recorded after round ten: “No new findings: the first clean round.”
Production work started 17 minutes later, one approved step at a time, with a spare Mac proving each step. Tests had grown from 232 unit and 57 integration after round one to 275 and 102 after round nine.
Five more rounds before release #
On 17 November I decided to publish the tool for other Mac admins, with one hard condition: nothing from my infrastructure could go out. The public copy was built from scratch rather than cleaned up, with an automatic scrubber and a leak check. The Codex briefs named none of the organisation’s terms, so every “no leaks” verdict was a blind test. All five found none.
- Round 1: “Don’t publish yet”, with 11 should-fix findings.
- Round 2: 7 of the 11 confirmed fixed, 6 new. One was a real agent bug: for a pkg inside a DMG, the vendor’s size and hash were checked against the extracted pkg instead of the DMG. It was unreachable with the sources I use, since CrashPlan’s publishes no hash; a GitHub-hosted DMG would have hit it.
- Round 3: 6 new.
- Round 4: “Fix first”, with four should-fix and two minor.
- Round 5: one minor finding, an unfilled address in a clone URL. The verdict: “Publish as-is.”
In the third round, the tool has a label-import command that drafts its config from Installomator labels, and each round Codex found a new shell construct that made it draft the wrong URL: functions, subshells, heredocs, read, && split over lines. Rather than keep patching a shell parser, the importer became strict. A label is drafted only if it is made entirely of plain name=value lines and comments. Anything else is a manual entry, even if it looks harmless. Across the 1,161 labels in a current Installomator at the time of writing, November 2026, that keeps 175 drafts out of 191, with 216 manual and 770 unsupported, and no errors.
What I take from it #
- When the same kind of finding keeps coming back, change the design. The inode and the anchor in rounds three and four, and the strict mode in the public round three, were all that, rather than one more special case.
- Write the failing test first. It is what made “fixed” mean something, and several of Claude’s own regressions were caught by tests before they shipped.
- Tell the reviewer where the bar is. It rose every few rounds, and the briefs said outright that a clean round was an acceptable result.
- Ask for the inventory. The round-ten brief asked for an explicit inventory: every place the agent writes, and what guards it.
What I’m not claiming: that ten rounds prove the code is correct, or that I know how much faster builds are because of it (I haven’t measured that). What I can say is that I approved every step into production knowing what had been found.
The tool and its docs are the other half of this pair, in The 60-requests-an-hour wall.