---
title: "We measure what our AI harness sees, not what it prevents"
description: "16.6% of the 2,661 plans closed by our AI implementation agents had at least one verification failure recorded before closing. No interception rate is publishable, and 14 of our 48 guardrails leave no trace. What a harness lets us claim, what it does not, and the five errors we corrected in our own method."
author: "Paul-Antoine Tual"
published: 2026-11-04
language: en-GB
category: "Junyr Method™"
tags: ["AI harness", "agentic coding", "AI agents", "implementation agents", "guardrails", "observability", "measurement", "human in the loop", "methodology", "Junyr Method™"]
canonical: https://paulantoinetual.fr/en/blog/harnais-implementation-ia-mesurer-ce-quil-voit
french_version: https://paulantoinetual.fr/blog/harnais-implementation-ia-mesurer-ce-quil-voit.md
---

# We measure what our AI harness sees, not what it prevents

> 16.6% of the 2,661 plans closed by our AI implementation agents had at least one verification failure recorded before closing. No interception rate is publishable, and 14 of our 48 guardrails leave no trace. What a harness lets us claim, what it does not, and the five errors we corrected in our own method.

Of the 2,661 plans our AI implementation agents closed between August and October 2026, **443 (16.6%)** had at least one verification failure recorded before closing [1]. That is what our harness sees: no interception rate is publishable, and **14 of our 48 guardrails** leave no trace at all.

Junyr is built mostly by AI agents that write code under a single orchestrating session, and the harness is everything around them, in software and in procedure. This article reports what we can count, what we cannot, and the five errors in our own method that we corrected along the way.

- Before closing: 443 of 2,661 plans carry at least one recorded failure, and close to two thirds of them (63.7%) still had a red check when closed.
- After closing: 190 plans (7.1%) are cited by a later bug report, but only 6 to 35% of those links are right, which points to 11 to 66 plans.
- Guardrails: of 48, 9 keep a firing counter, 25 leave an indirect trace and 14 leave none.
- Next step: a telemetry plan was submitted to our registry on 4 November 2026. It awaits a human decision.

## The harness is everything that frames our implementing agents

Our harness is the set of software and procedural devices that frame the AI agents implementing Junyr, and a few rules of the method hold it together. We described the general idea in our article on harness engineering [2].

- Plan by default. Nothing is implemented without an explicit request, and a human approves every plan.
- One writing orchestrator at a time. A single session drives the implementing agents, as in our article on lean orchestration [3].
- No Git. A registry in a database serves as history, the approach we set out in our article on removing Git [4].
- Release by a human gesture. Agents prepare a candidate; a person puts it into production.

For a business leader, nine terms are enough to follow the rest.

| Term | What it means |
|---|---|
| Plan | The written description of a piece of work, approved by a human before any agent starts |
| Claim | The moment an agent takes ownership of an approved plan |
| Lane | A batch of work entrusted to one agent |
| Wave | A group of lanes launched together |
| Declared footprint | The list of files a lane announces it will touch |
| Pre-image | A reference copy of the declared files, taken before the lane starts |
| Barrier | The check that compares changed files with the declared footprint and can raise a STOP (a halt) |
| Freeze | Freezing the code trees and building in isolation |
| Promote | The human gesture that puts a frozen candidate into production |

A plan crosses seven stages, and 48 guardrails sit along them: Preparation 4, Claim 7, Lane 8, Review and barrier 9, Rule gates 7, Freeze and promote 8, Night and follow-up 5 (see Figure 1). Each guardrail leaves a trace of one of three kinds, and the squares in the figure show which.

> **[Diagram] The seven stages of the harness and the trace left by each of its 48 guardrails**
> Seven boxes in a row, linked by arrows, one small square per guardrail. Counter squares are filled navy, indirect trace squares are light grey with an outline, no trace squares are orange. Totals: 9 counters, 25 indirect traces, 14 with no trace. By stage, counter, indirect, none: Preparation 0, 3, 1; Claim 2, 4, 1; Lane 3, 2, 3; Review and barrier 2, 4, 3; Rule gates 1, 4, 2; Freeze and promote 0, 5, 3; Night and follow-up 1, 3, 1.
> Figure 1. Our harness has seven stages and 48 guardrails: 9 leave a firing counter, 25 an indirect trace, and 14 no usable trace. Source: Junyr, devflow registry and run logs, inventory of 4 November 2026, internal documents, not public.

Counting them was the easy part.

## 443 of 2,661 plans had a failure recorded before closing

Between August and October 2026, 443 of the 2,661 closed plans (16.6%) had at least one failed verification recorded before closing, and 63.7% of those 443 still carried a red check at the moment the plan was marked done [1].

- Where failures concentrate (see Figure 2): lint (style checks) 14.1%, pytest (tests) 13.1%, adversarial review 8.4%, live 6.7%, browser 6.6%, security review 3.2%, rules 2.9%, real database 1.9%, typecheck (type checks) 1.3%, concurrency 1.3%. The mechanical checks come first, and the adversarial review, built to find fault, ranks third.
- Three steps recorded zero failures: pre-image (759 rows), lane receipt and red-proof. A zero is ambiguous, since it can mean that nothing failed or that failures there are not recorded, and the record alone does not say which.
- How to read it: a recorded failure shows that a check fired during the work. Whether the defect would have reached production is unknown.

> **[Diagram] Failure rate before closing, by verification step, August to October 2026**
> Horizontal bars, failure rate in percent. Lint 14.1, pytest 13.1, adversarial review 8.4, live 6.7, browser 6.6, security review 3.2, rules 2.9, real database 1.9, typecheck 1.3, concurrency 1.3. The two reviews are highlighted. Three steps recorded zero failures: pre-image, lane receipt and red-proof.
> Figure 2. Before closing, lint (14.1%) and pytest (13.1%) fail most often; the two reviews, adversarial and security, come third and sixth. Each rate counts failures the harness recorded before closing. Source: Junyr, devflow registry, measurements of 4 November 2026, internal documents, not public.

## Coverage changed month by month, so every figure before closing is a lower bound

Recording covered 49.6% of plans in August, 99.9% in September and 100% in October, while the share of plans with a failure went from 7.2% to 36.9%, then to 9.0% (see Figure 3) [1].

Coverage settles one thing: August cannot be compared with the other months, because half the plans had no record and their failures were never counted. It does not explain September's peak. Doubling the coverage could roughly double the share of failures, and the share was multiplied by five; October, just as fully covered, falls back to 9.0%. We do not know why September stands out, and we draw no conclusion from it.

- August: coverage is too low to compare with the following months.
- September and October: coverage is close to complete, which makes them the only two comparable months. The drop from 36.9% to 9.0% is real in the records, and we make no claim about its cause.
- Consequence: any aggregate before closing is a lower bound, best read on September and October.

> **[Diagram] Recording coverage and share of plans with a failure, August to October 2026**
> Two lines over three months, axis 0 to 100 percent. Recording coverage: 49.6 in August, 99.9 in September, 100 in October. Share of plans with a recorded failure: 7.2 in August, 36.9 in September, 9.0 in October. Coverage is nearly complete from September, so August cannot be compared, and coverage does not explain the September peak.
> Figure 3. Recording covered 49.6% of plans in August and nearly all of them from September: August cannot be compared, and coverage does not explain the September peak. Source: Junyr, devflow registry, measurements of 4 November 2026, internal documents, not public.

September's peak stays unexplained.

## After closing, 11 to 66 plans probably carry a later defect

A later bug report cites 190 of the 2,661 closed plans (7.1%), but a citation is a weak signal, so we read a random sample of 30 links between bug reports and plans [1].

- The reading: 5 real defects, 10 simple references, 11 tooling or test issues and 4 unclear cases.
- The rate: 5 out of 30 is 17%, with a 95% confidence interval of 6 to 35%. Applied to the 190 cited plans, that gives 11 to 66 plans, or 0.4% to 2.5% of closed plans.
- The central point: 17% would give about 32 plans. We quote the range, because a single figure would claim more than 30 cases support.
- The earlier sample: a first set of about ten reports, judged "mostly real", was biased. On the random draw, the figure fell to 17%.

The range is the finding.

## No interception rate is publishable, and four conditions explain why

Our first visual divided intercepted defects by intercepted plus escaped ones and returned 60 to 70%. We withdrew it. What replaces it is shorter: the harness reports what it sees, and what it prevents stays unmeasured. Four conditions are missing for anything stronger.

- No counterfactual. Nothing compares the same plans with and without the harness, so we do not know what would have happened.
- No measurable causal link. Only 25 plans are both recorded as failing and cited by a later bug report. Among plans with a recorded failure, 5.5% were cited afterwards, against 7.5% for the others. That gap is not significant, and the ratio of the two rates, computed month by month, ranges from 0.41 to 0.88.
- Variable coverage. Recording covered half the plans in August and nearly all of them afterwards.
- Unreliable links. Only 6 to 35% of the bug-to-plan links are right.

What follows is a table we can reuse as it stands.

| What we can say | What we will not say |
|---|---|
| 16.6% of closed plans (443 of 2,661) had at least one verification failure recorded before closing. | "The harness intercepts X% of defects." |
| After closing, 11 to 66 plans probably carry a later reported defect. | "190 plans were defective." |
| Of our 48 guardrails, 14 leave no usable trace. | "September shows a degradation of quality." |
| We do not yet know which AI model is the most reliable in this setting: the information is not recorded. | "Models X leave the framework more often than models Y." |

## 48 guardrails, 14 without a trace: the weak point is observability

We sorted the 48 guardrails into three classes of trace, numbered in pipeline order: 9 keep a firing counter, 25 leave an indirect trace that we can only infer from other records, and 14 leave no usable trace [1]. The weak point is observability. A guardrail without a trace may work very well, but we can neither prove it nor measure it.

Seven activity counts give a feel for what the traces do show.

- Freshness: 24 stale plans out of 797 probed.
- Resubmissions: 1,169 resubmissions and 481 abandoned plans, with 85% of the resubmissions coming within 7 days. The cause is not recorded.
- Barrier: 37 STOPs across 13 runs, with 547 paths flagged.
- Rule gate: failed on 43 plans.
- Freeze: 81 frozen candidates never promoted (with an empty proof), 36 promoted and 7 ready.
- Deferred proofs: 163, of which 129 were late, including 105 from the browser and 21 from live checks. The remaining 3 are not attributed.
- Ground-truth replay: 90 FAIL on 2,336 lines, in a single run, so we do not know how representative it is.

Seen is not proven.

## Nobody can say which lane left its footprint

No table attributes a write outside the declared footprint to a specific lane, and the AI model that executed a lane is recorded nowhere. We cannot compare models, and we cannot count how many implementers write outside their zone.

For the six largest STOPs (90, 69, 83, 81, 50 and 35 files), we read the causes by hand.

- Unattributed: two STOPs, of 90 and 69 files.
- Residue closure: one STOP, of 83 files, where the orchestrator closes a mechanical residue.
- Following pre-images: one STOP, of 81 files, which belong to a later wave.
- Merge refused by the tool: two STOPs, of 50 and 35 files.

Our qualitative reading, which we have not demonstrated, is that most STOPs originate in orchestration: declared residues and overlapping waves. Counts obtained by text search are a floor.

## Five errors in our own method, corrected before publication

Each error below changed a number or a claim in this article, and the table gives the correction we applied.

| Error | Correction |
|---|---|
| An interception ratio of 60 to 70% in our first visual | Withdrawn, for lack of a counterfactual and of a measurable causal link |
| A first sample of about ten bug reports, judged mostly real | Replaced by a random draw of 30 cases: 17% real, 95% interval of 6 to 35% |
| A first visual covering only 14 verification steps | Replaced by the inventory of the 48 guardrails |
| A label, "blocked by review", that misled: a negative review also includes refuted premises | Renamed "negative review" |
| A search for corrective plans by title, which returned 0 results | Replaced by the signal "plan cited by a bug report" |

## What to ask the team that builds your tools with AI agents

If your company has agents build its business tools, the decision that follows from our numbers concerns delay and risk: do not accept a promised benefit before the instrument that would measure it exists. We give no amount in euros, because we have no measure to support one.

- Instrument before you promise. Ask what is recorded today, before any claim about a gain.
- Ask which controls leave a trace and which do not. In our case, the answer is 34 of 48 with a trace, and 14 without.
- Separate what is seen before delivery from what comes back after it. For us, that is 16.6% on one side and 11 to 66 plans on the other, two different quantities.
- Refuse any interception rate that comes without a counterfactual.

Our own next step is a telemetry plan, submitted to our registry on 4 November 2026 and reviewed by an adversarial agent.

- A table called `harness_firings` records, for each firing: the guardrail, the result, the cause, the run, the plan, the lane and the AI model.
- Writing is best effort and never changes the exit code of an existing tool. The report is read-only.
- The review corrected four points: isolate the tests on a working database, emit only from a live registry, derive only causes the barrier already computes, and state that only about ten of the 48 guardrails will receive an emitter, the others being labelled "not instrumented".
- The plan is not approved: approval belongs to a human.
- Two questions remain open: whether to record the model for every lane or only for sensitive lanes, and whether to make the freeze's stop reasons machine-readable.

A first usable dataset will need several waves of operation.

## Five things we do not know

- The interception rate of the harness.
- Which AI model implements best in this framework.
- How many implementers write outside their zone.
- How effective the 14 guardrails without a trace are.
- How representative the ground-truth replay is, since it comes from a single run.

Until then, we report what we see.

> This article rests on a single internal dataset: one project, one team, three months. There is no control group. The link between bug reports and plans rests on a sample of 30 cases, the replay on a single run, and the causes of STOPs were read by hand on the six largest only. The development database used contains the full registry, and production was not used. The figures were produced by AI agents and cross-checked by other agents (Haiku sub-agents for cross-checks, an Opus agent for the inventory), and have not been independently audited.
>
> Method: the source is the devflow registry (plans, verifications, bug reports, release candidates, deferred proofs). The tools are read-only SQL queries and run logs. The safeguards are a random draw of 30 cases, a 95% confidence interval and the retraction of the unfounded ratio.

Further reading: the same discipline of declared, measured and unknown applied to a model's speed, power draw and price, in [The frontier runs on gas. Most office AI does not have to.](/en/blog/ia-frontiere-gaz-ia-bureau-frugale).

## Sources

1. Junyr, devflow registry and run logs, measurements of 4 November 2026, internal documents, not public.
2. Junyr, [*Harness engineering: the software layer that turns a model into an agent*](/en/blog/ingenierie-harnais-agent-ia), blog article, 20 September 2026.
3. Junyr, [*Lean orchestration: five rules from the bill of a multi-agent run*](/en/blog/orchestration-frugale-agents-ia), blog article, 27 September 2026.
4. Junyr, [*Removing Git from the active workflow without losing rollback: our roadmap-driven approach*](/en/blog/retirer-git-pilotage-par-roadmap), blog article, 15 July 2026.

## Frequently asked questions

### Does an AI implementation harness reduce defects?

We cannot say. Our harness records the failures it sees, but we have no counterfactual and no measurable causal link between a guardrail and a prevented defect, so no interception rate is publishable.

- 443 of 2,661 plans (16.6%) had at least one verification failure recorded before closing.
- Plans with a recorded failure were cited by a later bug report 5.5% of the time, against 7.5% for the others, a difference that is not significant.
- Our first interception ratio, 60 to 70%, was withdrawn.

### What share of AI-built changes fail a check before closing?

16.6%: 443 of the 2,661 plans closed between August and October 2026 had at least one failed verification recorded. It is a lower bound, because recording covered only 49.6% of plans in August, and it is best read on September and October.

- Lint (14.1%) and pytest (13.1%) fail most often; the adversarial review comes third at 8.4%.
- Close to two thirds of those 443 plans (63.7%) still carried a red check when they were closed.
- Recording covered 99.9% of plans in September and 100% in October, which makes those two months the readable ones.

### Why is a harness's interception rate hard to measure?

Because four conditions are missing: a counterfactual, a measurable causal link, stable recording coverage and reliable links between bug reports and plans.

- Without a run that skips the harness, we do not know what would have happened.
- Only 25 plans are both recorded as failing and cited by a later bug report, and the 5.5% against 7.5% gap is not significant.
- Only 6 to 35% of the links between bug reports and plans were right in a random sample of 30.

### What does a company need to record to measure its harness?

For each guardrail, the result, the cause, the run, the plan, the lane and the AI model, written to one table on a best-effort basis.

- Our plan is a table called harness_firings, written without ever changing the exit code of an existing tool.
- Today, 14 of our 48 guardrails leave no usable trace and the AI model that ran a lane is not recorded.
- Only about ten of the 48 guardrails will receive an emitter; the others will be labelled not instrumented.

### Do AI models leave their authorised zone often?

We do not know. No table attributes a write outside the declared footprint to a specific lane, and the model that ran each lane is recorded nowhere.

- We read by hand the causes of the six largest barrier STOPs: two unattributed, one residue closure, one set of later pre-images, two merges refused by the tool.
- Our qualitative reading, not demonstrated, is that most STOPs originate in orchestration.
- Counts obtained by text search are a floor.
