Engineering agentic systems: golden rules, architecture and security of the “Human-in-the-Loop”

When a machine acts without a guardrail

On 1 August 2012, it took just forty-five minutes for the brokerage firm Knight Capital to lose approximately $440 million. A faulty software deployment had reactivated obsolete code on a stock-order router, which began flooding the market with erroneous transactions, without any human mechanism being able to stop it in time [1]. The company, one of the largest American market makers, never recovered. This catastrophe is not a story about artificial intelligence, but it recounts exactly what is at stake today: the cost of an automated execution devoid of real-time operational guardrails.

The evolution of AI has set in motion a profound structural transition: the shift from passive systems, geared towards generating responses, to autonomous agents oriented towards action. Unlike conventional language-processing interfaces, modern agentic architectures design and execute multi-step action plans, interact directly with digital and physical environments, and handle critical tools: production databases, financial transaction gateways, industrial systems. This deployment relies on emerging protocols: Anthropic’s Model Context Protocol (MCP) [3] and agent-to-agent (A2A) communication protocols [4]. They allow a system to discover new capabilities dynamically at execution time.

This increased autonomy comes with a considerable widening of the attack surface and of operational risks. Yet language models have a documented flaw: they are systematically overconfident. Work on their calibration shows that a model commonly verbalises a confidence of 90% or more where its actual accuracy is markedly lower, and that hallucinations frequently occur with a high displayed level of certainty [2]. An agent can therefore generate an erroneous action plan, or even a destructive hallucination, while declaring itself perfectly confident. To underestimate this failure mode is to reproduce, at software scale, the Knight Capital scenario.

To remedy these vulnerabilities, the integration of human-control protocols, Human-in-the-Loop (HITL), must not be conceived as a mere layer of ergonomic validation, but as a fundamental engineering constraint. The level of control must, however, be matched to the criticality of the operations, the reversibility of the actions and the regulatory imperatives.

Three supervision regimes: HITL, HOTL, HOOTL

The first architectural decision consists in choosing, for each type of action, who decides and when. The literature on autonomous systems distinguishes three supervision regimes, a taxonomy inherited from the debate on weapons systems and transposable to software agents [5].

Regime	Temporality	Autonomy	Recommended use cases	Cognitive load	Operational & HR challenges
Human-in-the-Loop (HITL)	Synchronous interruption: the agent freezes its execution and awaits explicit human approval.	Low: the agent proposes, but the physical execution of the action is blocked.	Critical and irreversible operations: transfers, database modifications, medical diagnoses.	High: continuous assessment, context analysis, manual validation.	Requires dedicated teams of approvers, available to avoid bottlenecks.
Human-on-the-Loop (HOTL)	Asynchronous supervision: the agent runs on its own but under continuous monitoring.	Moderate to high: the agent acts within predefined limits.	Medium criticality, high volumes: booking modifications, scheduling, status updates.	Moderate: alerts triggered when a threshold is crossed or on abnormal behaviour.	A single supervisor can steer and audit several agents in parallel.
Human-out-of-the-Loop (HOOTL)	After-the-fact audit: no real-time intervention.	Maximum: the agent is sovereign within its scope.	Low risk, highly standardised: call routing, answers to frequently asked questions.	Low: periodic analysis of the logs and re-training.	Requires regular compliance auditors and offline annotators.

None of these regimes is “better” in absolute terms: the right system combines several of them, action by action. What remains is to determine how, concretely, to structure the agent’s behaviour so as to preserve human authority.

The golden rules of Human-in-the-Loop

Rule 1. Separate the proposal of an action from its execution

An agent’s architecture must never grant it the ability to execute, directly and unilaterally, an action with irreversible effects on its environment. The agent must produce a documented action proposal (intention, parameters, estimated consequences, source data used) and submit it to an isolated execution engine. Physical execution remains suspended until a qualified operator has approved the transaction. This separation protects the infrastructure against malicious request injections as well as against the model’s erroneous inferences.

Rule 2. Structure the interruption points before writing the code

Human control points are not added after the fact, as a catch-up. Engineers must map high-risk transactions from the design stage. This structuring relies on quantitative intervention thresholds that automatically trigger the pausing of the system: spending ceilings, data sensitivity levels, regulatory criteria (GxP, HIPAA, GDPR).

Rule 3. Make the agent’s reasoning entirely transparent

The operator’s trust depends on the visibility of the steps leading to a recommendation. The agent must set out, in an intelligible manner, the data it relies on, the business rules applied and the alternatives discarded. This transparency is best delivered through accessible interfaces (of the low-code or no-code kind) so that non-technical teams instantly understand the reason for an escalation and correct the agent’s trajectory without reading raw code.

Rule 4. Codify the non-negotiable limits deterministically

This is the most counter-intuitive rule, and the most important. An agent built on a probabilistic model cannot guarantee the observance of a security constraint formulated solely in natural language. Non-negotiable limits (restricted access to production databases, budget ceilings, the scope of authorised tools) must be encoded deterministically in the software harness surrounding the model. The system incorporates strict schema validation (typically via a validator such as Pydantic) and imperative rules that block a non-compliant request before it reaches the model’s interface or an external API. The guardrail lives in the code, not in the prompt.

Rule 5. Design for progressive automation

The implementation must begin at the maximum level of human intervention. This conservative start makes it possible to record decisions, to analyse the divergences between the model’s proposals and human corrections, and to measure the system’s reliability precisely. As the metrics improve and the models become calibrated, the pause criteria are relaxed: the system evolves from synchronous control (HITL) towards supervision by exception (HOTL). Automation must be earned; it is never the starting point.

Rule 6. Anchor governance in five obligations: the “HEART” memo

To place human-machine collaboration within a responsible approach, it is useful to summarise the control obligations under a mnemonic acronym, HEART, which brings together five pillars, each independently recognised as a principle of AI governance:

• Human safety & accountability: human safety and ultimate legal responsibility must rest with a natural person.
• Explainability: the explainability of every structuring choice, in order to rule out the “black box” effect.
• Alignment: the continuous alignment of the agent’s operations with the business objectives and the company’s ethical charter.
• Review: the regular audit and reassessment of performance and emergent behaviours.
• Trackability: the absolute technical traceability of the machine’s actions and of human validation decisions.

(HEART is here a mnemonic grouping device, not an established normative standard: the five obligations, for their part, are classic governance requirements.)

Rule 7. Manage escalation levels dynamically

The interruption system must adapt to the nature of the request. When it freezes its execution, the agent must be able to distinguish a simple approval (binary validation), a structured request for clarification (when an essential variable is missing), or the delegation of execution to a third-party system. Operator feedback must be guided by structured feedback prompts (to avoid vague responses) and stored, so as to enrich the system’s historical learning context.

Rule 8. Strictly bound the agent’s iterations

To prevent resource exhaustion and to stop an agent from becoming trapped in fruitless correction loops in the event of repeated disagreement with the human, workflows must impose a strict iteration limit. Once crossed without acceptable validation, this limit must switch the system into a safe fallback mode: suspension of the task and alerting of a higher level of supervision.

Architecture frameworks and emerging standards

Moving from principle to production requires protocols capable of unifying the technical execution of tasks and compliance control. Five references structure the field today.

The 12 factors of agents (12-Factor Agents)

Inspired by the “Twelve-Factor App” methodology from SaaS development, the 12-Factor Agents guide, theorised by Dex Horthy (HumanLayer) in line with the lessons learned from orchestrating coding agents such as Claude Code and the CodeLayer IDE, defines the criteria for agentic applications that are reliable and operable in production [6].

Factor	Technical description	Role with respect to human supervision
F1. Natural Language to Tool Calls	Translating intention expressed in natural language into tool-call schemas.	Formalises, in an intelligible way, the action the agent is about to undertake.
F2. Own your prompts	Control, versioning and mastery of the instructions passed to the model.	Prevents semantic drift and guarantees behavioural stability.
F3. Own your context window	Engineering the size and structure of the data injected into the context.	Prevents overloading the model and limits decision-making hallucinations.
F4. Tools are just structured outputs	Modelling the agent’s connectors as strict output schemas.	Allows deterministic validation before any external communication.
F5. Unify execution state and business state	Synchronising the machine’s execution state and the state of the company’s databases.	Avoids decisions based on obsolete business data.
F6. Launch/Pause/Resume with simple APIs	Starting, freezing or resuming a flow via standard APIs.	Indispensable for suspending the agent during asynchronous human interruptions.
F7. Contact humans with tool calls	Integrating the request for human help as a function executable by the agent.	Allows the agent to call on human expertise as a tool, in the event of uncertainty.
F8. Own your control flow	Retaining deterministic control over the state-transition graph.	Avoids entrusting the routing logic to the model’s random interpretation.
F9. Compact Errors into Context Window	Simplifying and intelligently injecting system errors into the context.	Helps the agent correct its errors without systematic human intervention.
F10. Small, Focused Agents	Dividing the work into a constellation of specialised micro-agents.	Isolates tool access rights according to the principle of least privilege.
F11. Trigger from anywhere	Triggering and supervising the agent from any channel (Slack, SMS, email).	Brings approval requests closer to employees’ workspaces.
F12. Make your agent a stateless reducer	Eliminating volatile internal states in favour of a pure, memoryless reducer.	Facilitates resumption after interruption and guarantees the reproducibility of audits.
F13. Pre-fetch all context (honourable mention)	Proactively loading the required data before the decision-making process.	Improves speed and avoids untimely data requests mid-flow.

Three factors directly carry human control: F7 (contacting the human as one calls a tool), F6 (suspending/resuming cleanly) and F10 (small agents with granular rights). These are the technical foundations of HITL.

AESP: economic sovereignty remains with the human

As soon as an agent carries out financial tasks (subscribing to a service, settling an invoice, managing third-party assets), a tension arises between the speed of the machine and human sovereign control over the assets. A research paper published in early 2026, the Agent Economic Sovereignty Protocol (AESP), formalises a strict invariant: the agent is economically capable, but never economically sovereign [7]. Any movement of funds must pass through a deterministic rules engine with eight verification levels before reaching the settlement layer.

Security rule	Control exercised	Consequence of a violation
Per-transaction limit	The absolute ceiling authorised for a single payment order.	Automatic rejection or escalation to human validation.
Time-window limit	Maximum cumulative spending over a sliding time window.	Immediate suspension of the agent’s payment capabilities.
Address allowlist	Network addresses or target wallets certified in advance.	Blocking of any transaction outside the approved destinations.
Chain allowlist	Restriction to the authorised execution protocols.	Immediate rejection of the submission.
Method allowlist	Specifically permitted functions or smart-contract calls.	Blocking at the cryptographic layer.
First-payment review	Compliance check at the first interaction with a third party.	Requirement for reinforced human approval.
Minimum balance	Cash reserve to be preserved in the originating account.	Freezing of any commitment to new spending.
Budget limits	Overall envelope allocated to the agent’s operating cycle.	Temporary deactivation of the financial tools.

These execution controls are reinforced, at the infrastructure level, by bilaterally signed cryptographic commitments (via the EIP-712 standard): no party can unilaterally modify the terms of a transaction. The protocol also preserves the confidentiality of flows thanks to ephemeral addresses derived by HKDF, which prevent an external observer from correlating the payments of a single agent. At this stage, AESP remains a recent academic proposal, not an adopted industry standard: its value lies in the clarity of the invariant it posits, more than in any maturity proven in production.

Capability-bound agent certificates

In multi-agent production, architectures face a risk of drift that 2026 research calls the capability-identity gap: an agent, having received human authorisation for a given scope, dynamically modifies its capabilities at execution time (by importing new tools or by delegating to subordinate agents) without the central system detecting this privilege escalation [8]. Conventional frameworks authenticate who the agent is, but not what it can do.

The proposed countermeasure: capability-bound identity certificates, which cryptographically bind the agent’s identity to the exhaustive list of its declared tools, its underlying models and its limits (concretely, by extending an X.509 certificate with a fingerprint of the “capability manifest”). The mechanism rests on three security properties:

• G1. Capability integrity: any modification of the tool configuration immediately invalidates the active rights and requires explicit human re-authorisation.
• G2. Behavioural verifiability: all actions must be auditable after the fact to detect model substitutions or parameter tampering.
• G3. Interaction auditability: exchanges between agents and with humans are recorded in a tamper-proof ledger, providing the forensic analysis trail in the event of an incident.

KnowNo: disturbing the human only when warranted

HITL has an enemy: approval fatigue. An agent that requests validation at every step wears out its supervisors and loses all value. The KnowNo method [9], presented in the work Robots That Ask For Help (Ren et al., CoRL 2023), addresses this problem by drawing on conformal prediction, which offers a statistical coverage guarantee (the correct option appears in the selected set with a probability of at least 1 − α, where α is the error rate tolerated by the organisation) while minimising the interruptions imposed on the operator.

The principle, without equations, comes down to one idea: instead of asking the agent “are you sure?” (a question it answers poorly, cf. the overconfidence of models), its uncertainty is calibrated on examples validated by experts, and escalation occurs only when several actions remain statistically plausible.

Concretely, faced with a state x, the agent considers a set of options 𝒴 = {y_1, …, y_m}. A calibration set S_cal = {(x_i, y_i)} of n historical situations is available, where each y_i is the action validated by an expert. For each example, a non-conformity score is computed, the inverse of the probability the model assigned to the correct answer:

s_i = 1 − f̂(y_i | x_i)

where f̂(y | x) is the probability assigned by the model to the option y. These scores are ordered and the quantile threshold corresponding to the targeted tolerance level is derived from them:

q̂ = Quantile( {s_1, …, s_n} ; ⌈(n+1)(1−α)⌉ ⁄ n )

At execution time, faced with a previously unseen situation x_test, the agent builds the set of acceptable candidate actions by retaining only those options whose non-conformity score remains below the threshold:

C(x_test) = { y ∈ 𝒴 : 1 − f̂(y | x_test) ≤ q̂ }

The decision whether or not to interrupt the human is then governed by the cardinality of this set:

Set C(x_test)	System decision
a single option	Autonomous execution of the sole action y*.
two options or more	Interruption: escalation to the operator in the form of a multiple-choice question.

If the set reduces to a single option, the model possesses a certainty mathematically aligned with the required level of safety, and it acts without disturbing the supervisor. If it contains several valid options, the agent identifies a zone of ambiguity, freezes its execution and presents the pre-selected choices. The guarantee of task completion then holds provided that the human answers correctly: KnowNo converts the statistical coverage guarantee into a guarantee of success, at the cost of a single question, asked only when it is necessary.

Fourteen principles for the human-agent lifecycle

Finally, the viability of an agentic system is not measured by the machine’s performance alone: human-agent interaction is a design target in its own right. A 2026 academic synthesis [10] proposes fourteen principles spread across four stages of an agent’s operational life, from commissioning to the repair of trust after an incident.

• Stage 1: Initial scoping and expectations. Explicitly announcing the known capabilities, limits and failure modes (Set Accurate Expectations); calibrating anthropomorphism to the context (neutral in a professional environment); establishing a clear relational framework (triage assistant, co-pilot, adviser) before entrusting real responsibilities.
• Stage 2: Interaction and shared control. Dynamically negotiating the level of initiative according to risk, with instant interruption and resumption (Negotiate Shared Control); making intention transparent; adjusting social cues without inducing dependence; regulating proactivity to protect the operator’s concentration.
• Stage 3: Long-term collaboration. Adapting to the user’s state (fatigue, correction rate); consolidating in memory their objectives and values, not merely their past actions; actively preventing unhealthy dependence; favouring behavioural consistency (predictable errors) over erratic excellence.
• Stage 4: Failure and repair of trust. Supporting a shared repair (direct-manipulation interfaces for correcting the agent); adapting the recovery strategy to the type of failure (state rollback and audit for serious errors); combining cognitive repair (explaining the error technically) and affective repair (expressing contextual regret and reaffirming the new safety instructions).

Synthesis: three directives for decision-makers

To deploy autonomous agents in industrial workflows without reproducing a Knight Capital-style scenario, IT managers and systems engineers can structure their architecture around three directives.

1. Isolate security control within the execution environment. Rule out security filters implemented solely as semantic instructions (system prompts). The validation of tool-call parameters, the verification of network rights and the management of transaction budgets must be handled by deterministic modules written in an imperative language, external to the model.

2. Establish dynamic control points and tamper-proof traceability. Structure complex flows into asynchronously managed state graphs. Synchronous interruption freezes execution without loss of information while awaiting approval. Every proposal, every operator response and every final validation are recorded in a decision log with cryptographic proofs, to simplify compliance auditing.

3. Use the human as a contextual corrector and learning guide. Direct the supervisor towards high-value-added cases. By statistically calibrating uncertainty (in the manner of KnowNo), the system automates standardised, high-confidence tasks and calls on the human, in the form of a choice, only for ambiguous situations. Every human correction is stored as an adaptive-learning knowledge base, gradually reducing the future need for manual interruptions.

These three directives trace one and the same conviction, which lies at the heart of the MATIA Method™: in the age of agents, scarcity does not shift towards generation (which becomes abundant) but towards verification. An agent has value in a company only if its autonomy is defensible before a client, an auditor or a regulator. Human-in-the-Loop is not the brake on automation: it is what makes automation industrialisable.

To turn these principles into a governed agentic architecture in your organisation, see the MATIA Method™ and the AI Express Audit & Roadmap.

---

Sources

[1] Knight Capital Group, Form 8-K (realised pre-tax loss of approximately $440 million), SEC EDGAR, 2 August 2012; and SEC, Order / press release 2013-222 (faulty code deployment reactivating obsolete code on an order router; ~45 min at market open), 16 October 2013. https://www.sec.gov/newsroom/press-releases/2013-222 [2] Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in LLMs (arXiv:2502.11028, 2025), verbalised overconfidence of models (displayed confidence ~88-99% vs markedly lower actual accuracy); corroborated by Can LLMs Express Their Uncertainty? (arXiv:2306.13063). The figure sometimes cited of “87% confidence” cannot be sourced to a precise benchmark and should be regarded as illustrative. https://arxiv.org/abs/2502.11028 [3] Anthropic, “Introducing the Model Context Protocol” (open protocol for connecting models to tools and data), 25 November 2024; dynamic capability discovery documented in the specification (tools/list, resources/list). https://www.anthropic.com/news/model-context-protocol · https://modelcontextprotocol.io [4] Google, “Announcing the Agent2Agent Protocol (A2A)” (agent-to-agent interoperability, complementary to MCP), 9 April 2025; protocol contributed to the Linux Foundation, June 2025. https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/ [5] In-the-loop / on-the-loop / out-of-the-loop taxonomy, originating in the literature on autonomous systems: Human Rights Watch & Harvard IHRC, Losing Humanity: The Case Against Killer Robots, 19 November 2012 (adapted to software agents). https://www.hrw.org/report/2012/11/19/losing-humanity/case-against-killer-robots [6] Dex Horthy / HumanLayer, 12-Factor Agents, Patterns of reliable LLM applications (transposition of the “Twelve-Factor App” to agents; F11: “Trigger from anywhere, meet users where they are”; F13 as an honourable mention). https://github.com/humanlayer/12-factor-agents [7] Jian Sheng Wang, AESP: A Human-Sovereign Economic Protocol for AI Agents with Privacy-Preserving Settlement (arXiv:2603.00318, 27 February 2026), “economically capable but never economically sovereign” invariant, deterministic engine with eight checks, EIP-712, HKDF. Proposal preprint, not an adopted standard. https://arxiv.org/abs/2603.00318 [8] Ziling Zhou, Governing Dynamic Capabilities: Cryptographic Binding and Reproducibility Verification for AI Agent Tool Use (arXiv:2603.14332, March 2026), “capability-identity gap”, capability-bound certificates (X.509 extension), properties G1 (integrity), G2 (behavioural verifiability), G3 (auditability). Preprint. https://arxiv.org/abs/2603.14332 [9] Allen Z. Ren et al., Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (“KnowNo”), CoRL 2023 (arXiv:2307.01928), conformal prediction, non-conformity score 1 − f̂(y|x), escalation to the human if the prediction set is not a singleton. https://arxiv.org/abs/2307.01928 [10] Haiyi Zhu, Canwen Wang, Qing Xiao, Hong Shen (Carnegie Mellon University), Design Principles for Human-Agent Interaction (arXiv:2606.20630, 2026), fourteen principles spread across four stages (Initially, During Interaction, Over Time, When Things Go Wrong). https://arxiv.org/abs/2606.20630