Choosing an AI model without betting on the next champion

By Paul-Antoine TUAL, AI Transformation Leader, Croissance et Transitions. August 2026, revised September 2026.

This fifth instalment in the “Markets & AI” series reaches a less dramatic conclusion than the end of a race: model performance is still improving, but businesses should treat selection as a reversible engineering choice rather than a bet on one lasting winner.

• Public leaderboards remain useful for finding candidates, but cannot justify a purchase on their own.
• Internal evaluations connect quality to your documents, costly errors and actual deadlines.
• Durable value moves towards the corpus, rules, controls and integration that you own.

---

1. A benchmark shortlists candidates; it does not choose your system

Older general tests have lost some discriminatory power as scores have risen, while more realistic evaluations have shown that the benchmark itself can become the main source of uncertainty.

• Stanford’s 2025 AI Index records saturation on several historic tests and points readers towards harder, newer evaluations [1].
• SWE-bench Verified was created in 2024 as a human-checked subset of 500 software tasks [2].
• In February 2026, OpenAI stopped using it to measure frontier coding capability, citing flawed tests and model exposure to training solutions [3].
• In July, OpenAI also withdrew its recommendation of SWE-bench Pro after estimating that about 30% of its tasks were broken [4].

Arena complements fixed tests with blind human preferences, yet its own method shows why two adjacent ranks do not automatically represent two distinct levels of quality.

• Its ranking method reports a score, confidence interval and rank spread instead of presenting the order as certain [5].
• A vote measures preference for one answer in the submitted context; it does not directly measure contractual compliance, API stability or the cost of a complete process.
• Style, reasoning effort and tool access must be compared under equivalent configurations.

A useful internal evaluation is a table of real tasks annotated before testing, then run blind on each model with the same instructions and tools.

• Sample: 20 to 50 cases covering routine requests, difficult cases and high-impact errors.
• Quality: factual accuracy, rule compliance, completeness and the rate of answers needing human rework.
• Service: latency, availability, rate limits and behaviour when a tool fails.
• Economics: cost per accepted task, including retries and human review.

2. Versions move faster than purchasing decisions

Official supplier logs show closely spaced releases, previews, scheduled withdrawals and API changes, making integration stability a purchasing criterion in its own right [6].

• A preview can be excellent for exploring a capability and too changeable for a critical process.
• A stable identifier, deprecation policy and migration period have measurable economic value.
• A modest score gain does not always offset a regression in latency, structured output or tool calling.

The proximity of a product announcement to a financial event may illuminate a supplier’s communications strategy without proving that financial markets set its technical timetable.

• A coincidence of dates establishes a sequence, not causation.
• A supplier announcement is a primary source for the product and an interested source for claims of superiority.
• Contractual guarantees, migration documentation and production measurements are more actionable for a buyer than the launch narrative.

A reversible architecture limits the effect of this volatility without requiring a complex router from day one.

• Isolate model calls behind an interface that you own.
• Version prompts, output schemas and the evaluation set alongside the application.
• Qualify a second supplier for critical tasks or prolonged incidents.
• Avoid proprietary features when their benefit does not cover the future migration cost.

3. Open weights widen the choice and bring an operating burden

In May 2026, Epoch AI estimated that the best open-weight models lagged the closed frontier by an average of four months on its composite index, a real convergence that still depends on the metric and the availability of evaluated models [7].

• The study measures an aggregate index and a time lag, rather than universal equivalence in every business domain.
• Epoch says private benchmarks and unreleased closed models may cause its estimate to understate the real gap.
• The proximity is nevertheless enough to make open solutions credible for more workloads.

Choosing between a closed API and open weights allocates control, risk and operational work differently.

• Managed API: quick deployment, elastic capacity and supplier-managed updates, with greater supplier dependence.
• Private hosting: control over version, location and the technical chain, with infrastructure and on-call costs.
• Managed open model: a useful middle ground when the contract clarifies the version, data handling and portability.

Specialisation should therefore follow diagnosis of the problem rather than a licence preference.

• Improve context first when an error comes from missing or outdated information.
• Add tools when the model must consult a system, calculate or act under supervision.
• Consider fine-tuning when the required behaviour is stable, repetitive and measurable at sufficient volume.
• Self-host when location, availability or economic constraints justify it through a documented total-cost calculation.

4. The right denominator is cost per accepted task

Stanford measured a more than 280-fold fall in the price of inference at GPT-3.5 performance between November 2022 and October 2024, but that trend predicts neither the volume consumed nor the cost of your application [1].

• A request may charge separately for input, output, reasoning, tools and some multimodal capabilities.
• Two models can use different token volumes to complete the same task.
• A low unit price loses its advantage when the model requires more retries or human review.

Discount mechanisms change the economics of a workflow only when its architecture matches their operating conditions.

• OpenAI says its Batch API costs 50% less than synchronous processing in exchange for asynchronous completion within a window of up to 24 hours [8].
• Caching reduces the cost of repeated prefixes; Google distinguishes automatic implicit caching from paid explicit caching with storage [9].
• These savings suit deferrable work and repeated long contexts, rather than unique or interactive calls without repetition.

A pilot should produce legible unit economics before deployment, with separate measurements so that the source of an overrun remains visible.

• Numerator: API, infrastructure, tools, observability and human labour.
• Denominator: accepted case, resolved ticket or approved document, depending on the process.
• Distribution: median and high-percentile cost and latency, rather than an average alone.
• Threshold: maximum budget and minimum quality that trigger human handling or a model switch.

Decide without waiting for a champion: four disciplines

A robust purchasing policy connects each model to a use, evidence, cost and exit route, allowing the business to benefit from progress without rebuilding the system after every leaderboard update.

• Measure locally: maintain real cases, annotated criteria and a history of regressions.
• Buy reversibly: decouple the application, qualify a fallback and negotiate data return.
• Manage the economics: track cost per accepted result, tokens by stage and human review time.
• Build your own assets: improve the corpus, rules, connectors and validations that survive a supplier change.

The decision to specialise a model comes after this basic discipline because most defensible advantages are first built in the data and evaluation process.

• A clean, versioned corpus can support a closed API, a retrieval system or a fine-tuned model.
• Explicit rules make outputs testable and incident reviews easier.
• A useful brand personality is described through examples, prohibitions and quality criteria, then measured on concrete situations.

To turn these principles into a first decision, the free Junyr AI maturity audit is a 30-minute video call with no commitment that positions your business on the scale, identifies the main blocker and first sensible project, and provides a one-page follow-up.

• Bring one candidate process, its volume and two example cases.
• You will leave with an explicit priority, without a promise of certification or a detailed roadmap.

Disclaimer. This article is educational and informative; it is neither investment advice nor a recommendation to buy or sell, and the cited rankings and prices are dated observations that may change.

Sources

[1] Stanford HAI, AI Index 2025: State of AI in 10 Charts, 2025.

[2] OpenAI, Introducing SWE-bench Verified, 13 August 2024, updated 24 February 2025.

[3] OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities, 23 February 2026.

[4] OpenAI, Separating signal from noise in coding evaluations, 8 July 2026.

[5] Arena, Arena’s Ranking Method, 14 November 2025, updated 28 February 2026.

[6] Google AI for Developers, Gemini API release notes, accessed 6 September 2026.

[7] Epoch AI, Open models lag state-of-the-art closed models by 4 months, 29 May 2026.

[8] OpenAI Help Centre, Batch API FAQ, accessed 6 September 2026.

[9] Google AI for Developers, Context caching, updated 13 August 2026.