Engineering agentic systems: golden rules, architecture and security of the “Human-in-the-Loop”

When a machine acts without a guardrail

The Knight Capital incident shows how automated execution can turn a local software fault into a major loss before a team can understand and stop the system, with approximately $440 million in realised losses accumulating in forty-five minutes on 1 August 2012 [1].

• A faulty deployment reactivated obsolete code on an order router and triggered erroneous trades.
• The event did not involve AI; it illustrates the broader risk of fast, repeated action that cannot be interrupted in time.
• For an agent, engineering must therefore govern the authority to execute as carefully as the quality of the recommendation.

Contemporary agents increase this operational exposure when they combine multi-step planning, tool calls and access to external environments, while protocols such as MCP and A2A facilitate connections to capabilities and communication between agents respectively [3][4].

• Exposed assets may include a production database, payment system, mailbox or industrial control.
• Discovering or adding a tool expands the available capabilities; the connection protocol does not itself grant permission to use it.
• A planning error, indirect prompt injection or stale business state can therefore produce a real-world effect at speed.

Human control must consequently be designed as an executable policy for authority, because a model’s stated confidence is neither a reliable measure of correctness nor evidence that an action complies with business rules [2].

• Assess risk by action type: impact, reversibility, detectability, interruption time and applicable obligations.
• Enforce prohibitions and limits in the software that authorises the action, independently of the prompt’s wording.
• Place human intervention where it can still change the outcome, and provide the context required to decide.

Three supervision regimes: HITL, HOTL and HOOTL

The choice between HITL, HOTL and HOOTL should be made action by action, according to the authority given to the machine and the point at which a human can still intervene, with this taxonomy from autonomous systems offering a useful frame without determining an application’s compliance by itself [5].

• Criticality assesses the scale and reach of an error.
• Reversibility measures whether the previous state can be restored without residual harm.
• Interruptibility establishes whether a human stop remains technically possible before or during the effect.

Regime	Timing	Agent authority	Suitable cases	Operational requirement
Human-in-the-Loop (HITL)	Synchronous pause before the effect.	The agent proposes; a separate mechanism authorises and executes.	Non-routine payments, sensitive changes, decisions whose effects are difficult to reverse.	Competent approver, readable context, deadline and cover procedure.
Human-on-the-Loop (HOTL)	Supervision during execution, with alerts and a stop control.	The agent acts within a predefined envelope.	High volumes, bounded and recoverable effects, anomalies detectable soon enough.	Telemetry, thresholds, a tested circuit breaker and named on-call responsibility.
Human-out-of-the-Loop (HOOTL)	Prior configuration and retrospective audit.	The agent acts without real-time intervention.	Standardised, low-impact tasks whose errors are detectable and repairable.	Minimal permissions, logs, sampling and rapid return to a more controlled regime.

The same workflow may therefore use several regimes without inconsistency, for example HOOTL to classify a request, HOTL to prepare a reversible change and HITL before a payment or deletion.

• Assign the regime to an action rather than to the agent as a whole.
• Prevent delegation or a newly available tool from bypassing the assigned regime.
• Reassess the choice after an incident, model change, new permission or metric drift.

The golden rules of Human-in-the-Loop

Rule 1. Separate an action proposal from its execution

For any action whose effects exceed the authorised autonomous envelope, the agent should produce a structured proposal and a separate component should decide whether that proposal may be executed.

• The proposal describes intent, parameters, data provenance, expected effects and rollback options.
• The authorisation engine checks permissions, business limits and state freshness at the last useful moment.
• The executor uses a token with limited scope and duration, then records the actual outcome rather than the intention alone.

Rule 2. Put interruptions where they can change the outcome

Effective checkpoints follow a risk map prepared before implementation and sit before an irreversible effect, with an explicit procedure for the absence of a human response.

• Thresholds may concern an amount, data sensitivity, destination, process deviation or a combination of signals.
• Expiry should lead to rejection or a safe pause, never to implicit approval.
• Under the AI Act, “high-risk” is a legal classification under Article 6, linked in particular to Annex I products or Annex III uses and subject to the stated conditions and exclusions; the provider’s or deployer’s role then determines the applicable duties [11].

Rule 3. Give the operator verifiable decision material

A useful approval request exposes the facts and rules needed for the decision without claiming to reproduce the model’s internal reasoning faithfully when the interface cannot verify it.

• Show source data, its date, the action parameters and the rules that fired.
• Distinguish retrieved facts, model inferences and unresolved assumptions.
• Present relevant alternatives, estimated effects and the means to reverse or contain the action.

Rule 4. Enforce limits in the software harness

A prompt can shape behaviour, but a security limit becomes binding only when an independent component checks it at the point of authorisation or execution.

• Validate schema and semantics: types, ranges, identifiers, destinations and consistency with current business state.
• Apply authentication, authorisation, least privilege, quotas, network filtering and secrets management outside the model.
• Handle idempotency, concurrency, timeouts, safe rejection and logging; Pydantic or a JSON schema covers only part of the control system.

Rule 5. Grant autonomy by action class

A new deployment with material effects should begin under close supervision and expand its autonomy only when production observations, fallback tests and the stability of its operating context support doing so.

• Record proposals, corrections, refusals, incidents and waiting times to establish a baseline.
• Measure action quality, escalation quality and recovery after error separately.
• Define thresholds for promotion to HOTL and automatic thresholds for returning to HITL.

Rule 6. Anchor governance in five duties: the “HEART” mnemonic

The HEART mnemonic groups five familiar governance concerns to make architecture reviews more systematic, without constituting a standard, certification or evidence of compliance.

• Human safety & accountability: assign decisions, controls and routes of redress to the people or entities responsible under the applicable law and governance structure.
• Explainability: provide justification proportionate to the use case and grounded in verifiable evidence.
• Alignment: continually test actions against business objectives, internal policies and external constraints.
• Review: reassess performance, incidents, drift and the adequacy of controls.
• Trackability: retain a sufficient, protected and governed record of proposals, authorisations, executions and corrections.

Rule 7. Match the escalation format to the need

An interruption should request precisely the decision the operator can make, because a generic approval transfers little information and encourages approval fatigue.

• Use binary approval when the context is complete and the decision rule is clear.
• Ask a structured question when a business variable is missing or several options remain plausible.
• Delegate to an authorised person or system when particular expertise or authority is required.

Rule 8. Bound attempts and define the fallback

Every agentic loop should have limits on iterations, time and cost, together with a safe terminal state, so that repeated disagreement or failure cannot consume resources indefinitely.

• Count attempts by task and by tool within an overall budget.
• Prevent automatic rephrasing from bypassing a human refusal or deterministic rule.
• When the budget is exhausted, pause, preserve useful state and alert the designated supervision level.

Architecture frameworks and emerging proposals

The following references do not share the same status: 12-Factor Agents is a practitioner guide, whereas AESP and capability-bound certificates are recent research proposals whose properties must be verified in the chosen implementation.

• Use these frameworks as design and review aids.
• Distinguish a protocol’s claimed property, the property actually enforced by code and the evidence available during an audit.
• Retain compensating controls where interoperability, revocation or logging remains incomplete.

The 12 factors of agents (12-Factor Agents)

Dex Horthy’s 12-Factor Agents guide adapts reliable-application practices to agents and stresses control of context, tools and execution flow, without implying that applying the list is sufficient to secure production [6].

• The factors structure the boundary between probabilistic outputs and deterministic execution.
• Pausing, resuming and contacting a human become explicit system operations.
• Focused agents make permission boundaries easier to enforce, provided the orchestrator actually applies them.

Factor	Technical description	Contribution to human supervision
F1. Natural Language to Tool Calls	Translate an intention into a structured tool call.	Makes the proposal inspectable before execution.
F2. Own your prompts	Version and test instructions.	Links behaviour to a known configuration.
F3. Own your context window	Select and structure injected data.	Reduces decisions based on noisy or stale context.
F4. Tools are just structured outputs	Model tools through strict schemas.	Permits deterministic validation of the request’s shape.
F5. Unify execution state and business state	Synchronise execution and business state.	Limits approvals based on an obsolete situation.
F6. Launch/Pause/Resume with simple APIs	Make start, pause and resume explicit.	Preserves state during an asynchronous decision.
F7. Contact humans with tool calls	Model human contact as an operation.	Structures the question, recipient and answer.
F8. Own your control flow	Keep routing under application control.	Stops the model from redefining authorisation points by itself.
F9. Compact Errors into Context Window	Summarise useful errors into context.	Supports correction without exposing unnecessary data.
F10. Small, Focused Agents	Narrow each agent’s role and tools.	Supports least privilege and action attribution.
F11. Trigger from anywhere	Accept several triggering channels.	Brings approval to the right operator under consistent authentication.
F12. Make your agent a stateless reducer	Derive the next action from explicit state.	Supports recovery, audit and partial reproduction.
F13. Pre-fetch all context (honourable mention)	Load required data before the decision.	Reduces unexpected data requests during the workflow.

AESP: bounding an agent’s economic capability

The AESP preprint proposes making an agent economically capable without transferring sovereignty over assets, using a deterministic eight-check engine, tiered escalation and cryptographic mechanisms [7].

• The engine expresses limits for transaction, period, destination, method, first interaction, balance and budget.
• EIP-712 commitments bind signatures to agreed typed data; their effect depends on those commitments being checked and executed.
• HKDF derivation aims to isolate contexts and reduce linkability, without providing anonymity by itself against metadata or operational practices.

Proposed rule	Control	Intended response
Per-transaction limit	Ceiling for one payment order.	Reject or escalate under policy.
Time-window limit	Cumulative spending over a rolling window.	Suspend payment capability.
Address allowlist	Authorised destinations.	Block unknown destinations.
Chain allowlist	Permitted networks or protocols.	Reject an out-of-scope submission.
Method allowlist	Permitted contract functions.	Block a disallowed call.
First-payment review	First interaction with a counterparty.	Require stronger approval.
Minimum balance	Reserve that must be preserved.	Refuse new spending.
Budget limits	Envelope for the operating cycle.	Temporarily disable financial tools.

AESP therefore provides a testable policy model, but its status as a preprint and the evaluation reported by its author do not amount to independent validation or evidence of production maturity.

• Review the threat model and full execution path before adoption.
• Test revocation, race conditions, recovery, oracle failure and paths that bypass the protocol.
• Treat reported performance and coverage figures as results of the proposal to be reproduced in the target environment.

Capability-bound agent certificates

Another research proposal cryptographically binds an agent’s identity to a capability manifest so that a tool, model or delegation falling outside the authorised scope can be detected [8].

• G1, capability integrity: a changed manifest should fail verification if every relying party checks the current version before acting.
• G2, behavioural verifiability: attestations and replay evidence support audit without making every probabilistic output automatically explainable or reproducible.
• G3, interaction auditability: a chain of evidence can make tampering detectable; its strength depends on storage, keys, timestamps and intermediate participants.

These certificates address a real authorisation-scope problem, but they remain a proposed architecture and do not replace runtime authorisation, revocation or tool isolation.

• Authenticate identity and verify the manifest at the last useful moment.
• Refuse tools or sub-agents outside the approved scope.
• Log policy decisions and protect the chain of custody for evidence.

KnowNo: calibrating escalation without promising certainty

KnowNo applies conformal prediction to robotic planning to build a set of candidate actions from calibration examples and ask for help when that set does not support autonomous execution under the chosen policy [9].

• The parameter α specifies a nominal error level and hence target marginal coverage of 1 − α.
• The standard guarantee assumes exchangeability between calibration examples and test cases, or the assumptions appropriate to the conformal variant used.
• It concerns coverage frequency across cases, not certainty that the action in a particular singleton set is correct.

For a state x, the agent considers 𝒴 = {y_1, …, y_m} and a calibration set S_cal = {(x_i, y_i)} of n situations whose correct action has been established; using KnowNo’s chosen non-conformity score, the calculation retains the following structure.

• For each example: s_i = 1 − f̂(y_i | x_i).
• For the target tolerance: q̂ = Quantile( {s_1, …, s_n} ; ⌈(n+1)(1−α)⌉ ⁄ n ).
• For a new case: C(x_test) = { y ∈ 𝒴 : 1 − f̂(y | x_test) ≤ q̂ }.

The cardinality of C(x_test) governs escalation, while imperative business thresholds retain priority over this statistical mechanism.

• If the set is empty, pause and escalate: none of the proposed options passes the selection rule.
• If it contains one option, execute it only when the action class is authorised, retaining the residual risk.
• If it contains two or more options, pause and ask a structured question presenting the admissible options.

KnowNo’s results show that such calibration can improve the trade-off between task success and requests for help in the environments studied, without guaranteeing every case or giving the supervisor knowledge they do not possess.

• A human answer helps only if the person understands the context, has the required authority and chooses correctly.
• A change in distribution, model, action list or policy requires renewed evaluation and often recalibration.
• Falling observed coverage or a rise in empty sets should suspend autonomy and trigger investigation.

Fourteen principles for the human-agent lifecycle

A 2026 academic synthesis organises fourteen human-agent interaction principles across four stages, underlining that effective control depends as much on the operational relationship as on raw model performance [10].

• Initial scope and expectations: state capabilities, limits and failure modes; establish an understandable role before granting responsibility.
• Interaction and shared control: negotiate initiative according to risk; make intent visible; allow interruption and resumption.
• Long-term collaboration: monitor fatigue and correction rates; remember useful goals; prevent dependency and favour predictable behaviour.
• Failure and repair: enable direct correction; match recovery to harm; explain the error and update controls.

This framework remains a research proposal to adapt to the setting, but it identifies four responsibilities often split across product, operations, security and management.

• Product designs understandable and proportionate requests for help.
• Operations provides availability, cover and response-time monitoring.
• Security and legal teams define limits, evidence and routes of redress for the applicable use.
• Management monitors human workload, incidents and the criteria for autonomy.

Three directives for defensible autonomy

An industrialisable agentic architecture makes every action authorisable, interruptible and auditable in proportion to its effects, while treating calibrated uncertainty as one statistical signal among the relevant business controls.

• Isolate execution: separate proposal, authorisation and effect; enforce permissions, budgets and constraints in components independent of the model.
• Design escalation and fallback: choose HITL, HOTL or HOOTL by action class; represent expiry, an empty conformal set, drift and incidents as explicit states.
• Measure before expanding autonomy: monitor coverage, corrections, escalation errors, recovery and operator workload, then revise policy on that evidence.

The Junyr Method™ consequently places verification at the centre of operational value: an agent’s autonomy becomes defensible when an organisation can explain its scope, demonstrate its controls and produce the record of a decision without conflating statistical assurance, technical security and legal compliance.

• A client should be able to understand who authorised what and on which data.
• An auditor should be able to connect configuration, decision, execution and outcome.
• A regulator assesses a specific system, intended purpose and actor role; no maturity level or HITL mechanism guarantees compliance by itself.

To turn these principles into a governed agentic architecture in your organisation, see the Junyr Method™ and the Junyr AI maturity audit.

---

Sources

[1] Knight Capital Group, Form 8-K (realised pre-tax loss of approximately $440 million), SEC EDGAR, 2 August 2012; and SEC, Order / release 2013-222 (faulty code reactivated an obsolete function on an order router for approximately 45 minutes after market open), 16 October 2013. https://www.sec.gov/newsroom/press-releases/2013-222 [2] Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in LLMs (arXiv:2502.11028, 2025), on verbalised overconfidence and distractor effects; corroborated by Can LLMs Express Their Uncertainty? (arXiv:2306.13063). https://arxiv.org/abs/2502.11028 [3] Anthropic, “Introducing the Model Context Protocol”, 25 November 2024; MCP specification for tool and resource discovery. https://www.anthropic.com/news/model-context-protocol · https://modelcontextprotocol.io [4] Google, “Announcing the Agent2Agent Protocol (A2A)”, 9 April 2025; contribution of the protocol to the Linux Foundation, June 2025. https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/ [5] Human Rights Watch & Harvard IHRC, Losing Humanity: The Case Against Killer Robots, 19 November 2012, for the in-the-loop / on-the-loop / out-of-the-loop taxonomy adapted here to software agents. https://www.hrw.org/report/2012/11/19/losing-humanity/case-against-killer-robots [6] Dex Horthy / HumanLayer, 12-Factor Agents: Patterns of reliable LLM applications. https://github.com/humanlayer/12-factor-agents [7] Jian Sheng Wang, AESP: A Human-Sovereign Economic Protocol for AI Agents with Privacy-Preserving Settlement (arXiv:2603.00318, 27 February 2026), preprint presenting the eight-check engine, EIP-712 and HKDF. https://arxiv.org/abs/2603.00318 [8] Ziling Zhou, Governing Dynamic Capabilities: Cryptographic Binding and Reproducibility Verification for AI Agent Tool Use (arXiv:2603.14332, version of 19 March 2026), preprint on capability integrity, behavioural verifiability and interaction auditability. https://arxiv.org/abs/2603.14332 [9] Allen Z. Ren et al., Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (“KnowNo”), CoRL 2023 (arXiv:2307.01928), conformal prediction and requests for help when the prediction set does not support autonomous action. https://arxiv.org/abs/2307.01928 [10] Haiyi Zhu, Canwen Wang, Qing Xiao, Hong Shen, Design Principles for Human-Agent Interaction (arXiv:2606.20630, 2026), preprint proposing fourteen principles across four stages. https://arxiv.org/abs/2606.20630 [11] Regulation (EU) 2024/1689 on artificial intelligence, notably Articles 3, 6 and 14 and Annexes I and III; classification and duties depend on the system, its intended purpose and the actor’s role. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689