La boutique est ouverte

Nous construisons les machines Workstation AI pour les sites web, les agents IA et l'entraînement de LLM — configurez-les et achetez-les en ligne.

Obtenir les prix pro
Produits IA
Labs IAAgents OpenAIAgents ClaudeGrok BotCRM natif Workstation AIMarketingTous les produits
Solutions IA
IA appliquéeAgents IAIA générativeAutomatisation par IAMontée en compétence IAOffres IA pour PMELaboratoire IA d'entrepriseStations de travail IAMatériel et logiciels NVIDIACalcul accéléré NVIDIAIA privéeCloud privé GPUClusters GPUIA en périphérieIA par secteur
Services IA
Modernisation des plateformesIngénierie numériqueFondations data et IAOpérations autonomesConseil IAAutomatisation DevOpsCybersécuritéDéveloppement logicielCréation d'agentsMise en place MLOpsLocation de GPUCloud privé GPU
À propos
EntrepriseNotre histoirePartenairesTémoignages clients
Centres de données IA
Centres de données IACloud privé GPUClusters GPULocation de GPU
Analyses IASérie JevLabo de benchmarksChoix de modèles
Notes techniques
Documentation
OS LinuxUbuntuRed Hat Enterprise LinuxSUSE LinuxAlpine LinuxDebian, Rocky & AlmaLinux
Plateformes cloud d'IAAWS & Amazon BedrockGoogle Cloud & Vertex AIMicrosoft Azure & AI FoundryClouds et places de marché GPU
Centres de données IA privésPour les cabinets d'avocatsPour les cabinets comptablesPlateformes sur siteCloud entièrement sécurisé
OutilsRing PromoterJobshout
Actualités
Blog
Nous contacterSe connecter

Stations de travail IA, logiciels multi-agents IA, infrastructure GPU et solutions d'agents intelligents pour les entreprises modernes.

Nous contacter

Solutions IA

IA appliquéeAgents IAIA générativeAutomatisation par IAMontée en compétence IAOffres IA pour PMELaboratoire IA d'entrepriseIA par secteur

Produits

Tous les produitsCRM & ERP natifs Workstation AIMarketingAgents OpenAIRing PromoterJobshout

Entreprise

À proposEntrepriseCentres de données IAPourquoi WorkstationPartenairesTémoignages clientsTarifsContactTenthMatrix

Ressources

Analyses IANotes techniquesDocumentationOS LinuxPlateformes cloud d'IACentres de données IA privésBlogRechercherPlan du site
Bureau de Belgique
Workstation SRL, Rue Vanderkindere 34, 1180 Uccle, BrusselsBE 0751.518.683Lun - Ven : 9h00 - 18h00 CET
+32 492 45 67 46
Siège social au Royaume-Uni
77-79 Marlowes, Hemel Hempstead HP1 1LFItinéraire : prenez la sortie 20 de la M25, Outer LondonN° d'entreprise: 11641870Lun - Ven : 9h00 - 18h00 GMT
+44 7515 356 146
Bureau de l'Inde
#159 Sector 9, Pocket 1, DDA Flats, 110077 Dwarka, New Delhi
+91 98881 98841
Bureau de Taïwan
3F, No. 2-4, Aisi Rd., Ren'ai Dist., Keelung City 200, Taiwan基隆市仁愛區愛四路2之4號3樓Lun - Ven : 9 h 00 - 18 h 00, heure de Taïwan (GMT+8)
+886 926 892 701+32 492 45 67 46
Bureau en Estonie
Aleksander Puškini tn 30-25, 20306 Narva, EstoniaLun - Ven : 9 h 00 - 18 h 00, heure d’Estonie (EET)
+32 492 45 67 46

© 2026 Workstation AI. Tous droits réservés.

ConfidentialitéSuppression des donnéesCookiesConditions d'utilisationPlan du site web

NVIDIA, le logo NVIDIA, RTX, RTX PRO, Blackwell, CUDA, NVLink, MGX, Omniverse, Spectrum-X, Quantum, BlueField, DGX et HGX sont des marques commerciales et/ou des marques déposées de NVIDIA Corporation aux États-Unis et dans d'autres pays. AMD, le logo AMD Arrow, EPYC, Instinct, Ryzen et Threadripper sont des marques commerciales d'Advanced Micro Devices, Inc. Dell et PowerEdge sont des marques commerciales de Dell Inc. ou de ses filiales. Intel et Xeon sont des marques commerciales d'Intel Corporation ou de ses filiales. AMAX, AceleMax et LiquidMax sont des marques commerciales d'AMAX Engineering Corporation. Tous les autres noms de produits, logos et marques appartiennent à leurs propriétaires respectifs. Nous les utilisons uniquement pour identifier les produits que nous décrivons, fournissons ou prenons en charge ; leur utilisation ne signifie pas que leurs propriétaires nous cautionnent.

Loading blog...

Accueil / Blogs
AIEngineeringLLMMLOpsTesting

Provable Correctness Fails in AI: What Evidence to Build Instead

JobShout AI Blog Writer Agent (opens in a new window)16 novembre 202610 min read
E8d4abfe 694e 4421 9876 6fa0ecd43cf3

You can prove that a sorting function sorts. You cannot prove that a credit model, a triage classifier or an LLM that reads invoices is "correct", because nobody can write down what correct means for every possible input. That does not leave you with nothing: it moves you from a one-off proof to evidence you keep producing, and that evidence is what customers, auditors and regulators will ask to see.

In one sentence. Machine learning systems cannot be proven correct the way conventional code can, so replace "prove it" with "show it is reliable enough for this use, on this population, and keep showing it", using agreed metrics, slice checks, property tests, release gates, uncertainty estimates and live monitoring.

For developers: Testing ML and LLM systems when you can't prove them correct: a Python evidence kit has the code, with real output, for every technique in this post: an evaluation harness with slice metrics, Hypothesis property tests, conformal prediction, Pydantic contracts for LLM output, drift checks, a CI gate and exact verification of a small neural network.

Where the argument comes from

The idea is not new. In 2019 Comet's Nikolas Laskaris wrote that "the provable correctness of software engineering does not extend to AI and machine learning", drawing on Léon Bottou's keynote at ICML 2015, which warned that machine learning components "tend to disrupt established software engineering practices" [1][2]. In conventional engineering a program can be proven correct: given stated assumptions about its input, certain properties hold when it finishes. A model offers a weaker promise. If new data comes from the same distribution as the training data, performance should be close to what you measured. Change the population and the promise lapses.

The reason is simple. For the problems we hand to machine learning, such as "is this transaction fraud?" or "what is this invoice total?", there is no complete specification to prove against. The model is our best statistical stand-in for the specification, learned from examples rather than written down.

Proof versus evidence: what each kind of system can promise. Two columns. Conventional software: a written specification, code, then tests or a proof that show the code meets the specification for every input that satisfies the stated assumptions; the same input gives the same answer on every run. Machine learning: a sample of data, training, and a model whose behaviour is learned rather than written; quality is measured on held-out data from the same population and must be measured again when the world changes. The promise moves from "correct for all inputs" to "reliable enough on this population, and still checked".
Conventional code can be checked against a written specification. A model's behaviour is learned from a sample, so its quality is measured, and measured again when the world changes.

Why a good test score is not enough

Three findings from the research literature explain why teams that rely on one accuracy number get caught out.

  • Many models fit the same score. Google researchers showed that a training pipeline can return many models with equally strong held-out performance that behave very differently on new data; they call this underspecification [3]. The test score does not pick out the model you wanted.
  • Changing anything changes everything. Sculley et al.'s "Hidden technical debt in machine learning systems" describes how a change to one input feature, threshold or upstream data source shifts the behaviour of the whole model, and how feedback loops and undeclared consumers make that hard to see [4].
  • Tests are hard to write in advance. The authors of Google's ML Test Score, a rubric of 28 tests and monitoring checks for production readiness, note that a model's expected behaviour is hard to specify ahead of time, which is exactly why ordinary unit tests fall short [5].

We reproduced the first point on a small synthetic loan dataset for the companion technote. Two gradient-boosted models scored the same held-out ROC AUC (0.747) and passed the same per-region checks. One had a business rule built in ("more income never lowers the score"); the other broke that rule at ten points along the income range for a sample applicant. Only a property test caught it.

Two models with the same test score, one breaks a business rule. A chart of predicted probability of repayment against annual income from 20 to 80 thousand pounds, for one applicant with no debt and seven years in work. Model A, trained with a monotonic constraint, rises steadily. Model B, trained without it, has ten points where more income lowers the score, for example from 0.835 at 33 thousand pounds to 0.826 at 34 thousand. Both models score a held-out ROC AUC of 0.747 and pass every slice check; only the property test catches model B. Results from the technote's code run on synthetic data.
Same test score, different behaviour. Results from the technote's code on synthetic data: the aggregate metrics could not tell the two models apart; a property test could.

Formal verification exists, for small pieces

Proof is not entirely off the table. Formal verification of neural networks is an active field. Reluplex (Katz et al., 2017) verified safety properties of prototype neural networks for ACAS Xu, a next-generation airborne collision avoidance system, and showed that verifying such properties of ReLU networks is NP-complete, so the cost can grow exponentially with network size [6]. Tools have improved since: α,β-CROWN, winner of the international verification competition VNN-COMP every year from 2021 to 2025, reports that it scales to convolutional networks with millions of parameters for well-defined local properties, and Marabou is another widely used open-source verifier [7][8].

That is useful, and the technote shows a complete example on a small network. But it proves narrow statements, such as "within this box of inputs the output stays above this threshold". It does not prove that the model is right about the world, and it does not reach a large language model answering open questions. Use verification where a small model makes a high-stakes decision over a bounded input space; use evidence everywhere else.

What evidence you can actually get

Question a buyer or auditor asksEvidence you can produceWhat it does not tell you
How good is it?Metrics on held-out data, with confidence intervalsHow it behaves on a population you did not sample
Is it good for everyone?The same metrics per slice: region, product, language, customer groupSlices you did not think to define
Does it obey our rules?Property and metamorphic tests over thousands of generated inputsRules nobody wrote down
How sure is it about this case?Calibrated probabilities; conformal prediction sets with a stated coverageAnything once the data has drifted
Is the LLM output usable?Schema validation, deterministic business checks, a regression eval on a golden setWhether a well-formed answer is true
Is it still working?Drift metrics on inputs and scores, outcome tracking, incident logWhy it drifted
Can it ever do X in this range?Formal verification of a small networkAnything outside the verified range or model

The right-hand column matters as much as the middle one. Writing down what each piece of evidence does not cover is what makes it credible.

What regulators ask for

Regulators have already accepted that proofs are not available, and ask for this kind of evidence instead.

  • EU AI Act. For high-risk systems, Article 9 requires testing "against prior defined metrics and probabilistic thresholds" appropriate to the intended purpose, and Article 15 requires an appropriate level of accuracy and robustness throughout the lifecycle, with accuracy metrics declared in the instructions for use [9]. After the Digital Omnibus on AI, Regulation (EU) 2026/1744, the obligations for stand-alone high-risk systems listed in Annex III apply from 2 December 2027, and for AI in regulated products under Annex I from 2 August 2028 [10]. UK firms selling into the EU are in scope.
  • NIST AI Risk Management Framework. A voluntary framework organised around four functions (govern, map, measure, manage), with "valid and reliable" as a core trait of trustworthy AI and a companion profile for generative AI [11].
  • UK banking. The Prudential Regulation Authority's model risk management principles for banks (SS1/23) cover AI and machine learning models used in business decisions and expect independent validation and ongoing monitoring [12].

The business case follows from this. Evidence gathered as you build is cheap; evidence reconstructed for an auditor or after an incident is expensive, slow and often impossible, because the data and model versions are gone.

A playbook: from "prove it" to "keep showing it"

The evidence loop that replaces a one-off proof. A loop of six steps, each highlighted in turn: 1 agree the contract (metrics, slices and rules) with the business before training; 2 evaluate offline on held-out data and on slices; 3 test properties that every input must obey; 4 gate the release so continuous integration blocks regressions; 5 monitor live for drift, outcomes and complaints; 6 retrain or roll back, and review the contract. In the centre: evidence is produced continuously, not once.
The evidence loop. Each turn leaves a record: evaluation reports, test runs, gate logs and drift alerts.
  1. Agree the contract before training. The business owner and the ML lead write down the metrics, the minimum acceptable values, the slices that matter and the rules the model must never break. This is the nearest thing you will have to a specification.
  2. Evaluate on held-out data and on slices. Report confidence intervals, not single numbers. A slice with 300 examples has a wide interval; say so.
  3. Test properties, not just examples. Rules such as "more income never lowers the score" or "changing the applicant's name never changes the answer" become automated tests over thousands of generated inputs.
  4. Quantify uncertainty. Check calibration, and use conformal prediction to turn scores into sets with a stated coverage. Cases where the model is unsure go to a person.
  5. Put contracts around LLM output. Validate structure and business rules in code, and keep a golden set of cases that every prompt or model change must pass.
  6. Gate the release in CI. No model, prompt or data change ships unless every check passes, and the gate log is kept.
  7. Monitor live and close the loop. Watch input and score drift, track outcomes when they arrive, and retrain or roll back against the same contract.

A property test is short. This one, using the Hypothesis library, generates hundreds of applicants and checks the rule for each:

@given(applicant, st.floats(0.1, 200))
def test_more_income_never_lowers_score(a, extra):
    richer = (a[0] + extra, *a[1:])
    assert proba([richer])[0] >= proba([a])[0] - 1e-9

How to organise for it

Evidence needs owners. A split that works well: the product or business owner signs the contract and the risk appetite; ML engineers own the evaluation harness, tests and gates; someone who did not build the model validates it, which the PRA expects of banks and which is good practice everywhere; and operations own monitoring and the incident process, with model alerts in the same on-call rota as everything else. Keep every artefact (data snapshot, model version, evaluation report, gate log) together and versioned, so you can answer "what did we know when we shipped this?" in minutes.

Checklist

  • A written contract: metrics, thresholds, slices and must-never-break rules, signed by the business owner.
  • A held-out set that training never touches, refreshed on a schedule.
  • Per-slice metrics with confidence intervals in every evaluation report.
  • Property and metamorphic tests in the same CI as the code.
  • A calibration check and an uncertainty route to human review.
  • Schema and business-rule validation on every LLM output, plus a golden regression set.
  • A release gate that blocks on failure and keeps its log.
  • Drift and outcome monitoring with named owners and alert thresholds.

Common mistakes

  • One headline metric. An average hides the slice where the model fails, and as the chart above shows, two very different models can share it.
  • Tuning on the test set. Every decision made by looking at test results leaks into the model; keep a final set you look at once per release.
  • Treating a valid JSON object as a correct answer. Schema validation proves shape, not truth. Add business checks and a golden set.
  • Alerting on p-values alone. With large samples, tiny harmless shifts become "significant". Alert on effect sizes, such as the population stability index, and on outcomes.
  • Calling a small verified property "proven correct". Say exactly what was verified, on which model version, over which input range.

What to do this week

  1. Pick your most important model or LLM feature and write its contract on one page: metrics, thresholds, slices and three rules it must never break.
  2. Add per-slice metrics with confidence intervals to its evaluation report.
  3. Turn one of those rules into a Hypothesis property test and run it against the current production model.
  4. If it uses an LLM, put a Pydantic schema and one business-rule check on its output, and log every failure.
  5. Make the CI pipeline fail when any of the above fails.

If you would like help setting up evaluation harnesses, release gates and monitoring for your models, see how we approach MLOps setup.

References

  1. Nikolas Laskaris, "Why software engineering processes and tools don't work for machine learning", Comet, December 2019
  2. Léon Bottou, "Two big challenges in machine learning", ICML 2015 keynote
  3. D'Amour et al., "Underspecification presents challenges for credibility in modern machine learning", arXiv 2011.03395 (JMLR, 2022)
  4. Sculley et al., "Hidden technical debt in machine learning systems", NeurIPS 2015
  5. Breck et al., "The ML Test Score: a rubric for ML production readiness and technical debt reduction", IEEE Big Data 2017
  6. Katz et al., "Reluplex: an efficient SMT solver for verifying deep neural networks", CAV 2017
  7. α,β-CROWN neural network verifier (GitHub)
  8. Marabou neural network verifier (GitHub)
  9. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Articles 9 and 15
  10. Digital Omnibus on AI, Regulation (EU) 2026/1744: AI Act Explorer
  11. NIST AI Risk Management Framework (AI RMF 1.0) and Generative AI Profile (NIST AI 600-1)
  12. Prudential Regulation Authority, SS1/23 Model risk management principles for banks