When Green Is Not Clearance: What a False-Pass Test Reveals About Coding Agents

Two controlled false-green cases exposed the gap between a successful command and trustworthy acceptance. The next question is whether protecting that gap saves developers work.

There is a particular kind of software failure that does not look like failure at all. A test command finishes. Its exit status is zero. A coding agent reports completion. Every component appears to agree.

But the agreement can be about the wrong thing.

I have been building ClaraGate, a bounded independent-clearance prototype for coding-agent workflows. Recent adversarial reproductions sharpened a distinction that I think matters well beyond any one tool: the observed success of a verifier is not the same as authority to accept the work.

1. Two ways a green result can mislead

In one controlled reproduction, the pytest output reported failing tests while the process returned exit status zero. In another, a newly introduced test configuration deselected the failing cases; the run reported success because the remaining selected test passed. Both outcomes were independently reproduced against the shipped alpha test environment.

These were deliberately constructed stress tests. They are not evidence that coding agents routinely use these techniques in production. Nor are they proof that ClaraGate already blocks every variant. They show why the meaning of a green command depends on more than its exit status.

The cases are also different. In the first, the process status failed to faithfully represent the failure messages visible in the output. In the second, the command's successful outcome reflected a reduced set of executed checks. A verifier that only observes exit zero can accept either outcome for reasons unrelated to the original acceptance intent.

2. A frozen test file is not a frozen verification system

What determines a pytest result extends beyond assertion bodies. Test selection, collection hooks, configuration files, plugins, environment variables, process behavior, and the runner itself can affect which checks actually execute and what result is returned.

That creates a specific boundary problem: a system can preserve a declared set of test files while still allowing an uncontrolled input to change the meaning of their execution.

In the current alpha, ClaraGate reruns a developer-designated Python/pytest command and compares defined verification inputs with a pre-established basis. That is a narrower guarantee than a sealed execution environment. The reproduced cases exposed gaps outside the protected surface or in the trust placed in a process return code. They are findings to address, not failures I am claiming have already been solved.

The design question is therefore not simply how to catch more edits. It is how to establish which inputs are authoritative, which can be changed by the coding agent, and what observation is sufficient to justify a clearance decision.

3. Legitimate test changes are the other half of the problem

Protecting acceptance criteria introduces a tension that an external engineer put plainly during technical discussion: what happens when the agent is supposed to add tests?

An agent implementing a new constraint may reasonably create new coverage. Automatically treating that as disqualifying would be a poor developer experience. But letting the agent unilaterally revise the acceptance conditions would recreate the self-attestation problem.

There is an important distinction here. Adding ordinary task tests is not necessarily the same as changing the protected acceptance basis. A useful workflow must distinguish additive evidence from an amendment to the conditions of clearance, and it must specify who can approve that amendment.

That distinction is potentially the product. It is also a potential cost center. If approval is required on most ordinary tasks, an independent gate could become another rubber stamp. Architectural independence would not, by itself, make the workflow valuable.

4. Measure the cost of the boundary, not just the number of catches

The next useful experiment is not a polished demonstration where every planted failure is caught. It is an observation of real coding-agent tasks and what engineers actually have to do before accepting them.

For each task, I want to distinguish whether tests were added, whether the governing acceptance basis changed, whether a hold was justified, whether the developer needed to investigate, and how long that investigation took. Just as important: what would the same developer have done with their existing CI and review process?

Two measures matter together: unjustified clearance, where a weak signal is accepted, and avoidable review burden, where a gate consumes time without adding useful evidence. Minimizing one by ignoring the other is not progress.

Negative controls are also important: deliberately failing fixtures can help reveal when a check has become blind. But no single negative control proves the whole evaluation path is complete, and a passing control is not a certificate of test adequacy.

5. What I am trying to learn now

ClaraGate's current development scope is intentionally narrow: a coding-agent alpha centered on OpenCode and Python/pytest, with a developer-designated rerun and protected verification inputs. I am working through adversarial findings and seeking independent developer feedback. The alpha is not a universal code-correctness checker, a production merge authorization system, or a guarantee against false PASS outcomes.

The commercial question is equally bounded: does independent completion clearance save developers more acceptance work than it introduces? That means testing against existing alternatives, including CI, independent test environments, and human code review—not assuming another gate is useful simply because it is independent.

I am particularly interested in the mundane, concrete answers. When was the last time an agent changed tests for a legitimate reason? When did green CI fail to answer the question a reviewer actually needed answered? And how much work would an independent clearance step have removed—or added?

Autonomy does not become more trustworthy merely because fewer humans press Merge. It becomes more useful when the evidence required for acceptance can be established without asking humans to reconstruct the entire result.

That is the boundary ClaraGate is trying to make practical.