Beyond vibe coding: four practices for verifiable software

What vibe coding actually covers

Vibe coding, a phrase coined by Andrej Karpathy in February 2025, describes a practice in which a person states an intention in natural language, largely accepts the generated code and iterates on the result without examining every line [1].

• Karpathy placed the practice in the context of personal, disposable projects.
• Simon Willison reserves the term for work in which the generated code goes unread, distinguishing it from AI-assisted development with understanding and review [2].
• Collins named it its 2025 Word of the Year, evidence of cultural reach rather than validation as a production method [3].

The suitable method therefore depends on the software’s lifespan and risk, because flexibility that helps a prototype becomes debt once the code serves customers, moves money, processes sensitive data or supports a lasting operation.

• For exploration: optimise for rapid learning and plan to discard the prototype.
• For retention: identify an owner, document decisions and add the relevant tests.
• For production: add security, operational and experience controls proportionate to the consequences of failure.

What the available studies really measure

The available evidence supports checking generated code, but each study examines a specific setting and none can independently calculate a particular project’s probability of failure, productivity gain or future release stability.

• Security: 45% of generated samples failed security tests in Veracode’s 2025 benchmark; that proportion aggregates more than 100 models, 80 tasks and four languages, and is not a default production vulnerability probability [4].
• Productivity: METR’s randomised trial observed a 19% slowdown among 16 experienced developers completing 246 tasks in repositories they knew well; METR describes the result as a snapshot of early-2025 tools in this setting [5].
• Delivery: DORA 2025 reports an association between greater AI adoption, higher throughput and higher instability; the report describes AI as amplifying existing practices rather than causing the same outcome in every team [6].

Together, these findings support one practical decision: teams should measure their own system and retain evidence from controls rather than infer quality from a benchmark, a persuasive demonstration or a model name.

• Define risks and acceptance criteria before generation begins.
• Test the code, architecture and environment that will actually be released.
• Examine failures, correct them and retain the results behind the decision.
• Require accountable human sign-off where the stakes warrant it.

UltraCoding as a delivery method

We use UltraCoding to mean an organisation of agent-assisted development in which generation is one stage and delivery rests on four coordinated practices, with security, accessibility and internationalisation controls running across them.

• Orchestrate distinct roles where the work can be divided usefully.
• Challenge the first solution through review focused on defects and assumptions.
• Test behaviour, cross-cutting qualities and the rendered interface.
• Decide from defined gates and explicit human judgement.

The method seeks quality that can be evidenced within a defined scope, without promising defect-free software or treating validation by another agent as independent assurance.

• Agents may share the same blind spots, training data or reasoning errors.
• Tests establish only the properties and scenarios they actually cover.
• Independent assurance requires sufficient separation of people, tools, interests or environments for the risk at hand.

Practice 1: orchestrate work that can run in parallel

A multi-agent architecture is useful when several paths can be explored separately, but its value must be evaluated on the target task rather than inferred from results achieved by a research system [7].

• An orchestrator divides the problem, assigns bounded briefs and consolidates the results.
• Specialists may address architecture, implementation, testing or documentation in parallel when their dependencies remain manageable.
• Interfaces, completion criteria and conflict resolution matter as much as the number of agents.

Anthropic’s field report supplies a useful but narrow reference point: its multi-agent system outperformed Claude Opus 4 alone by 90.2% on an internal research evaluation and used about 15 times the tokens of a chat interaction [7].

• The improvement concerns broad research questions with independent lines of inquiry.
• The 15× figure describes this architecture and comparison, not universal coding economics.
• Anthropic itself notes that many coding tasks offer less parallelism and involve more dependencies.
• A deployment decision therefore needs local evidence on quality, elapsed time, cost and rework.

Practice 2: make critical review traceable

Critical review asks a second agent to find errors, weak assumptions and edge cases in the first result, then ties each criticism to executable evidence or a human decision [8][9][10].

• A compiler, test or static analyser can confirm some defects repeatably.
• A critic agent can suggest missed scenarios, probe authorisation and look for leaks or injection paths.
• The generator addresses the findings, while an accountable person resolves disputes and accepts residual risk.

This separation can improve detection without automatically creating independent assurance, because two agents may share a model, provider, context or harness that reproduces the same biases.

• For a routine change: combine agent critique, automated tests and proportionate code review.
• For a sensitive component: add specialist security tools and a reviewer sufficiently separate from the producer.
• For a regulatory or contractual duty: determine the required controls and independence from the applicable rules and context.

Practice 3: test software across three layers

A useful test strategy covers expected behaviour, cross-cutting qualities and the rendered interface separately, so success in one layer cannot conceal defects that only another layer can expose.

• Behaviour: unit, integration and end-to-end tests for intended journeys and failure paths.
• Cross-cutting qualities: security analysis, secret detection, dependencies, authorisation, logging, internationalisation and applicable WCAG criteria [12].
• Interface: captures of critical states and viewport sizes, regression comparison and assisted visual inspection.

Vision-language models can identify and describe visible anomalies, while evaluating an experience remains a human activity informed by user research, usability testing and business constraints [11].

• Visual AI pre-sorts captures, suggests issues and speeds up documentation.
• Technical tests confirm repeatable states, interactions and accessibility criteria that can be automated.
• Representative people expose misunderstandings, expectations and compromises that an image does not reveal.
• The design owner decides whether the experience suits its actual audience and setting.

Practice 4: tie every release to explicit gates

The delivery loop repeats generation, challenge, testing and correction until agreed criteria are met, then sends non-automatable questions and residual risk to the person who owns the decision.

• Define mandatory tests, their environment and tolerances before work starts.
• Block release on agreed critical failures, such as a functional regression or a detected critical vulnerability.
• Record exceptions, their expiry, owner and remediation plan.
• Revisit gates as incidents, user feedback and the product evolve.

A gate makes the decision observable and repeatable within its scope, but it guarantees neither the absolute absence of vulnerabilities nor overall compliance, which requires controls matched to each system, actor and obligation.

• “No critical defect detected by the defined tools” is a verifiable criterion.
• “The software is secure” goes beyond what that result establishes.
• Production remains a risk decision supported by supervision, observability and rollback capability.

Scale the effort to the risk and the real economics

The depth of control should increase with lifespan, exposure and the cost of failure, while its economics should be calculated from local measurements rather than another application’s token multiplier.

• A disposable script may need only a short check inside a closed scope.
• A customer service, financial flow or personal-data process warrants more testing, separation and traceability.
• Useful measures include full cost, elapsed time, rework, escaped defects and incidents avoided without double counting.

Internal hosting may answer control or location requirements, but it does not make the economics favourable by itself: hardware, energy, availability, operations, security, upgrades and skills all belong in total cost.

• Compare equivalent models and quality levels on a representative workload.
• Include actual utilisation, off-peak operation, redundancy and staff costs.
• Check contractual terms and data flows before describing a solution as sovereign.
• Choose among APIs, dedicated hosting and a hybrid model through a documented total-cost analysis.

Questions a business leader can require answers to

A business leader need not select every tool to establish this discipline, but should ask who produces, who checks, what evidence triggers release and who remains accountable for the residual risk.

• Who reviews generated code, with what degree of separation and what blocking authority?
• Which tests cover behaviour, security, accessibility, internationalisation and interfaces?
• Which gates prevent release, and how are exceptions tracked?
• Which measures compare total cost, speed and quality after release?

The developer retains a central role: they frame constraints, select relevant evidence, adjudicate cases automation cannot settle and own the release decision.

• The agent generates and explores at speed.
• Tools provide observations limited to their coverage.
• Specialists interpret results in the technical and business context.
• The accountable person accepts, delays or rejects the release.

Move from generation to evidence

UltraCoding organises a simple idea: as code generation becomes easier, value moves towards explicit criteria, complementary controls and accountable human judgement that can defend a release.

• Use vibe coding to explore within a scope where failure remains inexpensive.
• Install the four practices when software must last or defects carry consequences.
• Scale controls to observed risks without turning a benchmark into an individual promise.

Junyr’s AI maturity audit is a free, no-commitment 30-minute video call that provides an initial position you can act on, followed by a one-page summary.

• Your position on the maturity scale.
• The main obstacle to address.
• The first sensible project to undertake.
• A one-page summary after the call.

Book the 30-minute audit.

---

Sources

[1] Andrej Karpathy, post that introduced the phrase vibe coding, 2 February 2025. https://x.com/karpathy/status/1886192184808149383

[2] Simon Willison, “Not all AI-assisted programming is vibe coding (but vibe coding rocks)”, 19 March 2025. https://simonwillison.net/2025/Mar/19/vibe-coding/

[3] Collins Dictionary, “Collins’ Word of the Year 2025: AI meets authenticity as society shifts”, 6 November 2025. https://blog.collinsdictionary.com/language-lovers/collins-word-of-the-year-2025-ai-meets-authenticity-as-society-shifts/

[4] Veracode, 2025 GenAI Code Security Report. https://www.veracode.com/wp-content/uploads/2025_GenAI_Code_Security_Report_Final.pdf

[5] METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, 10 July 2025. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/

[6] DORA, State of AI-assisted Software Development 2025. https://dora.dev/research/2025/dora-report/

[7] Anthropic, “How we built our multi-agent research system”, 13 June 2025. https://www.anthropic.com/engineering/multi-agent-research-system

[8] OpenAI, “Finding GPT-4’s mistakes with GPT-4”, 27 June 2024. https://openai.com/index/finding-gpt4s-mistakes-with-gpt-4/

[9] Madaan et al., Self-Refine: Iterative Refinement with Self-Feedback, 2023. https://arxiv.org/abs/2303.17651

[10] Shinn et al., Reflexion: Language Agents with Verbal Reinforcement Learning, 2023. https://arxiv.org/abs/2303.11366

[11] Lu et al., UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design, CHI 2025. https://arxiv.org/abs/2502.12561

[12] W3C, Web Content Accessibility Guidelines (WCAG) 2.2. https://www.w3.org/TR/WCAG22/