feature · observed edit truth
An AI agent audit trail that does not take the agent's word for it.
Muster's workspace observer derives every file change from git and the filesystem, hashes the content before and after, and emits a receipt that reproduces on a machine that was never there. Measured live against Codex, the agent's own patch stream reported 0 of 5 real edits.
Head to head
Codex self-report vs Muster observation, on the same edits.
Two independent live runs on 2026-08-27, plus three earlier protocol probes. Every number below was re-measured in a fresh session rather than copied from a build report.
| Signal | Codex app-server (self-report) | Muster workspace observer |
|---|---|---|
| Edits reported, 5 live runs | 0 of 5 | Reported in both runs it was attached to |
item/fileChange/patchUpdated | 0 | n/a — does not use the channel |
turn/diff/updated | 0 | n/a — does not use the channel |
workspace.patch events | — | 1, diff verified by git apply |
| Detection latency | Never detected | 86ms (earlier run 75ms; budget 1000ms) |
| Source of truth | What the agent loop says it did | git baseline + filesystem, re-derived every cycle |
| Content integrity | Not hashed | sha256 before and after |
| Cross-machine reproducibility | Not applicable | Byte-identical receiptHash, two processes ~40 min apart |
Rerun it yourself — scripts/evidence/workspace-observer-live.mjs →
Why the gap exists
Coding agents edit through the shell, and the patch channel never sees it.
Every agent harness that records file changes gets them the same way: the coding backend emits a structured notification saying it modified a file, and the harness writes that down. It is the obvious design, and for a while it worked, because early coding agents really did apply edits through a dedicated patch tool.
They mostly do not any more. A capable coding agent given a refactoring task will reach for the shell — a python heredoc, sed -i, a generated patch piped into git apply, or a whole script it writes and then runs. From the structured patch channel's point of view, all of that is a single opaque commandExecution. The files change. The notification never fires. The harness's audit log stays empty and, crucially, gives no indication that anything is missing.
Muster measured this instead of assuming it. A wire-level probe drives codex app-server through initialize → thread/start → turn/start, timestamps every notification, polls the target file every 5ms so that patch-versus-disk ordering can be established, and lingers past turn/completed to rule out a shutdown race. Across three probes and two full head-to-head runs, item/fileChange/patchUpdated fired zero times while all the edits landed on disk. This finding is version- and model-specific, and Muster's own documentation says it should be re-measured against each Codex release rather than assumed permanent.
What the observer does instead
The workspace observer never subscribes to a claim. It runs on four invariants, each of which exists because of a specific failure it was caught by:
- Watch events are a trigger, never data. Every cycle re-derives full state from git and the filesystem from scratch. On macOS, recursive
fs.watchreports every change as "rename", coalesces create-then-delete into one event, invents ancestor-directory entries, and can hand back a null filename. Those payloads are unusable as data. Dropping every watch event costs latency, bounded by the poll interval — never correctness. git statusalone is insufficient. An agent that runsgit commitmid-turn makes its own edits invisible togit status --porcelain. So the observer pins a baseline commit at start and diffs the worktree against it, using status only to discover untracked files and to inherit.gitignore.- Strictly read-only. Never
git add, neverhash-object -w, never the index lock —--no-optional-lockseverywhere. The only writes go to a shadow tree under the system temp directory, deliberately outside the watched root, so the observer can neither trigger itself nor perturb yourgit status. - Only changes since
start(). Start seeds the baseline and never emits. Pre-existing dirty state is the world as the run found it, not something the run caused — which matters on any real branch with staged work in progress.
There is one stated blind spot rather than a papered-over one: a file created and deleted entirely between two detection cycles is invisible, because neither git nor the lossy macOS watch stream can recover it. The window is bounded by the debounce and poll intervals, and a caller that flushes at every tool-call boundary shrinks it to near zero for shell-driven edits.
Receipts, not log lines
Detecting the change is half of it. The other half is producing something a person who was not present can check. Each observed change carries a deterministic receiptHash computed from the content hashes and the change set. Two observer processes, running in different temporary directories roughly forty minutes apart on the same logical change, produced byte-identical receipt hashes. That is what turns a log line into citable evidence.
The receipts feed an append-only event spine with 21 typed event types, monotonic sequence enforcement, fencing tokens that reject stale writers, idempotency keys that detect conflicting receipts, and a full compensation and cancellation state machine with terminal invariants. One reducer-level rule refuses any payload containing secrets or model chain-of-thought — because an audit log you cannot show an auditor, in case it leaked a key, is not an audit log.
At the protocol boundary, source: "observer" is enforced at the type level, so a backend's self-report is structurally incapable of masquerading as observed truth. That is a small detail with a large consequence: the two categories of evidence can never be silently mixed.
How it was hardened
The observer and the board it feeds went through adversarial verification before shipping, which found and fixed eight real bugs. The worst were an O(N²) state fold that took a 10,000-event drive from 2073ms down to 224ms once fixed, a planner that proposed assignments the reducer would have rejected, and a work-in-progress saturation case that escalated as no_qualified_model — misreporting transient capacity pressure as a permanent capability gap to whoever read the audit output. Observer stress testing survived a 320-file mutation storm with gapless sequences and correct hash chains.
Open flags, stated plainly
- The
workspace.patch.diffvariant would broadcast raw file content unredacted over the stdio transport. Nothing emits it yet, and a redaction layer has to land before anything does. - The live evidence script needs an authenticated Codex CLI, so it is a manual evidence job rather than an unattended CI check.
- The Codex finding is version-specific. Re-measure it; do not assume it holds forever.