Ask AI three probability questions, then add them up to spot a doubtful number

You paste a pile of customer notes into an AI assistant and ask: “What is the chance a customer cancels their plan this quarter?” It answers 15%, and it sounds very sure. You have no real answer to compare against, because the quarter is not over. A preprint on arXiv (a public archive of scientific papers that have not been peer reviewed) dated October 22, 2026 proposes a check that needs no real answer: ask about the same thing in three ways and see whether the numbers add up.

Try it now with your own notes

Mr. Hung manages a store that sells online learning plans; he is an example built for this article. He has three months of customer-care notes and wants the AI to estimate the chance that a customer cancels.

1. Before pasting, delete customers’ names, phone numbers, emails and addresses, and keep only the notes the question needs. Open three new conversations and paste the same notes into each. The preprint also asks each question in a separate call; if you ask them one after another in the same conversation, the model may remember the earlier number and make the new one fit.
2. In the first conversation, ask for the overall number: “What is the chance, in percent, that a customer cancels their plan this quarter?” Mr. Hung gets 15%.
3. In the second, ask for the share of each group. You choose how to split customers into two or three groups, based on what the notes can actually tell you, for example annual-plan and monthly-plan customers: “Of these customers, what percent belong to each group?” Each customer belongs to exactly one group; if the shares the AI gives do not add up to 100%, ask again until they do before you add anything. He gets 40% annual, 60% monthly.
4. In the third, ask for the chance of cancelling within each group: “If the customer is on an annual plan, what is the chance, in percent, that they cancel this quarter? And on a monthly plan?” He gets 10% and 30%.
5. Add it up yourself: 0.40 × 10% + 0.60 × 30% = 4% + 18% = 22%. The overall number from the first conversation was 15%, a gap of 7 percentage points.

You now have three numbers, one sum and one gap to compare with a threshold. Set the threshold before you ask, for example a few percentage points, rather than choosing it after you have seen the numbers. If the model answers with a range or in words instead of a single number, ask for one percentage per question and write down that you asked for it.

Read the gap, then decide

A 7-point gap, as in the example, is well beyond a threshold of a few points. Throw out both numbers instead of picking the one you like better, and do not keep asking until they match. You have two routes: count directly in a spreadsheet if the notes hold enough data, or treat the AI’s number as a guess not yet usable for a decision. A small gap only tells you the numbers agree with each other; it does not tell you they match reality.

This suits numbers you cannot count directly: the chance a case runs late, a customer buys again, or a complaint escalates, where the AI has to read free-text notes and estimate. If the notes already contain the figures, counting in a spreadsheet is faster and more accurate. The appeal of this check is that it is cheap: three questions and one sum.

Why the sum can flag error

The author, Peter Potash, asked seven large language models whether their probability judgments obey the law of total probability. The law says the chance of X equals the sum, over each group z, of (the chance a customer is in group z) times (the chance of X if the customer is in group z). Here the group z is something not written down anywhere; the model has to infer it from the records in its context. Every question in the test has a correct answer that can be computed exactly.

According to the preprint, GPT-6.1 Sol and Claude Opus 5.5 were the most consistent, with a mean absolute gap of 0.08 and 0.12 log-odds (a scale for differences in probability expressed as odds), about 2 percentage points. Claude Haiku 5.5, Gemini 3.8 Flash and DeepSeek V4-Pro were persistently less consistent, at 0.27–0.35 log-odds, and that gap varied mostly from question to question rather than being one shared offset. Within every model, the larger the gap, the further the answer was from the truth (Spearman rank correlation of 0.40 to 0.62), so the gap, which needs no real answer to compute, flags answers that are less reliable.

Two other details are worth keeping. The length of the model’s reasoning did not predict consistency, so do not treat a long answer as a sure one. And simple consistency checks nearly all pass; the failures appear when the model has to reason through a hidden variable, which is exactly the “which group cancels” question above. This is one author’s preprint, not yet peer reviewed, tested on problems whose answers were built in advance, and the preprint reports that large gaps go with large errors, not that small gaps mean the answer is right. The code and data are published with the paper on GitHub (a site for storing and sharing source code), so your technical team can rerun them before bringing the method into your workflow.

Sources

1. Peter Potash, “Latent Coherence: Do Frontier Language Models’ Probability Judgments Obey the Law of Total Probability?” (arXiv:2610.31833, October 22, 2026) — source for the law of total probability, the seven models, the 0.08–0.35 log-odds figures and the Spearman correlation of 0.40–0.62.
2. Code and data for the preprint on GitHub — linked from the arXiv page.