Two customers write to the same support inbox an hour apart, asking your AI assistant the identical question — can I return this after 40 days. One gets told no, the window is 30 days. The other gets told yes, exceptions are made for unopened items. Nobody edited a policy in between. Nobody trained a new model. The same system answered the same question twice and gave two different answers.
The instinct is to treat this as a bug — something broken that needs fixing before you trust the tool with anything that matters. That instinct is half right. The mechanism behind it is not a defect; it is how these systems generate text by design. What is fixable is whether you have put that mechanism somewhere it can do damage.
What's actually happening
- Modern AI text generation is sampling, not lookup. At each step the model computes a probability distribution over possible next words and draws from it — it does not retrieve one fixed answer from memory.
- A setting called "temperature" controls how much randomness that draw uses. Lower temperature narrows the draw toward the single most likely word each time; higher temperature spreads it across less likely alternatives, which is what makes output feel varied or creative.
- Even temperature zero — theoretically "always pick the most likely word" — is not fully deterministic in production. Server-side batching, GPU floating-point behaviour and how requests are grouped together can shift results slightly run to run, according to inference-engineering research from Thinking Machines Lab.
- This is not unique to conversational answers. A 2026 arXiv study found that roughly 30% of content-moderation decisions from AI classifiers changed when the only thing varied was the random seed used during training — same content, same model architecture, different verdict.
- The most cited real-world case is a single wrong answer, not two conflicting ones — and that is the more useful lesson. A Canadian tribunal held Air Canada liable in 2024 after its website chatbot told a grieving customer he could apply for a bereavement discount after buying his ticket, when the airline's actual policy required applying first. The company argued the bot was a separate legal entity responsible for its own words. The tribunal did not accept that.
01Sampling, not lookup
A database returns the same row every time you run the same query, because a database's job is retrieval — something was stored, and you are getting it back unchanged. An AI language model does not work that way. It generates each word of its response by calculating how likely every possible next word is, given everything so far, and then drawing one from that probability distribution.
Most of the time that draw lands on the highest-probability word, which is why AI output usually looks coherent and consistent on the surface. But "usually" is doing real work in that sentence. Where the top few candidates are close in probability — which happens constantly in ordinary language, because there are often several reasonable ways to phrase the next part of a sentence — the draw can land on a different one of them. That single different word changes what comes after it, and the divergence compounds from there. Two runs that start identically can end up making different claims by the third sentence.
This is not the model "not knowing" the answer, in the sense of lacking the relevant information. It is the generation process itself being probabilistic by construction, at every single step of every response, whether the underlying facts are simple or contested.
02The dial that controls how much this shows
The setting most directly responsible for how much variation you see is usually called temperature. It scales how sharply the model favours the single most probable next word versus spreading weight across plausible alternatives. Set low, temperature makes the output converge tightly on the most likely phrasing each time — more repeatable, more literal, closer to "boring." Set high, it makes the output range more widely across alternatives — more varied, more inventive, and more prone to producing something you did not expect.
Neither setting is the "correct" one in the abstract. A tool generating marketing headline options should probably run warmer, because variety is the entire point of asking for five options instead of one. A tool quoting your shipping policy to a customer should run as cold as the platform allows, because variety is exactly what you do not want from a policy statement.
The three-minute version
If your AI vendor exposes a temperature or "creativity" setting, find it. For anything customer-facing that states a fact — a price, a policy, an eligibility rule — set it as low as the interface allows, or use whatever preset is labelled "precise" or "factual" rather than "creative" or "balanced." Most consumer chat tools don't expose this at all; if yours doesn't, that is itself useful information about how much control you actually have over consistency, and it belongs in the vendor conversation covered in what to check in an AI vendor contract.
03Where it's harmless and where it's a real problem
The mechanism is identical whether the stakes are trivial or serious. What differs is whether inconsistency between two runs actually costs you anything.
| Use case | Does the variation matter? | Why |
|---|---|---|
| Drafting five subject-line options for a campaign | No — it's the feature | You wanted range. A second draft that differs from the first is the tool doing its job. |
| Brainstorming names, taglines, or blog angles | No | A human reviews and picks; nothing ships unread. |
| Summarising an internal meeting for your own notes | Rarely | Small wording differences between two summaries don't change what you decide to do. |
| A support bot stating your return, refund, or warranty policy | Yes — directly | Two customers who received different answers to the same question have a legitimate grievance, and one of the two answers is wrong. |
| A tool quoting price, availability, or delivery timing | Yes | Customers compare notes. A quoted number that moves between conversations looks like bad faith even when it's a sampling artifact. |
| An AI screening applications, claims, or compliance cases for similar facts | Yes — and hard to see | Two similar cases resolved differently is a fairness and, in regulated contexts, a legal problem — and unlike a customer complaint, nobody necessarily notices unless you go looking. See which decisions never to hand to an AI agent. |
The dividing line isn't "creative versus factual" as a vibe — it's whether a human downstream relies on the specific wording as if it were policy. A first draft nobody acts on is low stakes by definition; a sentence a customer or regulator can quote back to you is not.
The Air Canada case is worth sitting with here, because it's often told as a story about an AI "lying," which slightly misses the point. The chatbot did not know it was wrong. It generated a plausible-sounding answer to a question about a policy it had been given imperfect information about, and generated it with exactly the same confidence it would have used for a correct answer. Confidence is not evidence of accuracy in these systems, and it never reads as lower for a wrong answer than a right one — which is precisely why a second customer asking the same question could easily have received a third, differently wrong, answer.
04Reducing variability where it counts
- Lower the temperature, or use a "precise" preset, for anything stating a fact. This is the single highest-leverage lever if your platform exposes it. It won't make the tool infallible, but it narrows the range of plausible wrong answers.
- Replace open-ended prompting with structured templates for repeatable tasks. A support bot answering "what's your return policy" from a fixed, retrieved policy snippet is far more consistent than one generating the answer fresh from general reasoning each time — the model has less room to improvise when the actual policy text is right in front of it rather than something it has to recall.
- Put a human or a rules check between the AI and anything binding. A quoted price, a stated exception, or a compliance determination that a person confirms before it goes out closes the gap between "the model's best guess" and "what your business is actually committed to."
- Test the same question multiple times before you trust a deployment. Ask your support bot the identical question five separate times. If the answers vary in substance — not just phrasing — that is a finding, not noise, and it tells you where to add guardrails before customers find it for you.
- Don't fight it where it doesn't matter. Demanding identical output from a brainstorming tool, or reprompting repeatedly because two drafts weren't the same, spends effort solving a problem that was never actually a problem.
- Log what customer-facing tools actually said. If a customer disputes what your AI told them, you want the transcript, not a reconstruction. This is the same discipline covered in backing up what an AI agent changes, applied to words instead of files.
05The honest summary
An AI tool answering the same question two different ways isn't malfunctioning — it's sampling from a probability distribution the way it always does, and most of the time that's invisible because the top candidate wins. The two places it stops being invisible are exactly the two places you should already be paying attention: anything a customer can hold you to, and anything that decides a case. Everywhere else, the variation is the tool working as intended, and the fix is to stop treating it as a defect and start treating it as a setting — temperature down, structure up, human in the loop wherever the sentence has to be right the first time.
07Common questions
Why does my AI chatbot give different answers to the same question?
Because generating text with a modern AI model is a probabilistic process, not a database lookup. At each step the model computes how likely every possible next word is and draws one from that distribution rather than retrieving a single stored answer. Most of the time the most likely word wins, which is why output usually looks consistent, but where two or more words are close in probability the draw can go a different way on a second attempt — and that single difference can cascade into a materially different answer by the end of the response.
Is this the same thing as 'the AI is hallucinating'?
Related but not identical. Hallucination usually refers to a model stating something false with full confidence. Sampling variability is about the same model potentially giving two different responses — which may both be plausible-sounding, and either or both could be wrong — to the identical prompt. A model can be perfectly consistent and still hallucinate every time, or vary its wording while stating the same correct fact each time. They're separate failure modes that happen to share a root cause: the model generates its best guess rather than retrieving a verified fact.
What is temperature in an AI model, in plain terms?
A setting that controls how much randomness goes into picking each next word. Low temperature narrows the choice toward whatever the model considers most likely, producing tighter and more repeatable output. High temperature spreads the choice across less-likely alternatives, producing more varied and sometimes more surprising output. Most consumer chat apps don't expose this dial directly, but many business and developer-facing AI platforms do, often as a slider or as presets labelled something like 'precise' versus 'creative.'
Does setting temperature to zero make an AI's answers identical every time?
Not reliably, in practice. Temperature zero should in theory mean the model always picks the single most probable word, which sounds fully deterministic. But production AI systems batch multiple users' requests together for efficiency, and factors like GPU floating-point rounding and how a request happens to be grouped in that batch can still introduce small differences between runs, according to inference-engineering research. Zero temperature meaningfully reduces variation; it doesn't guarantee an identical answer every time.
Where does this inconsistency actually cause a business problem?
Wherever a customer, employee, or regulator can reasonably treat what the AI said as your business's actual position — a stated price, a quoted policy, a stated eligibility decision, or a compliance judgment on a specific case. Two customers who received different answers to the identical question about your return policy have a legitimate complaint, and by definition one of them was told something wrong. The 2024 Air Canada case, where a tribunal held the airline liable for its chatbot misstating a bereavement fare policy, is the clearest illustration: the company was on the hook for what the bot said, not for what its written policy actually was.
Can this bias matter even when a human reviews the AI's output?
It reduces the risk but doesn't remove it, particularly at volume. A 2026 study on AI content moderation found that roughly 30% of decisions changed when only the random seed used during training was varied — same content, same model design, different verdict. If a human is reviewing every single AI-assisted decision closely, inconsistency gets caught. If the AI is doing first-pass triage across hundreds of similar cases and only a sample gets close review, two nearly identical cases can land on different outcomes without anyone noticing the pattern.
How do I reduce inconsistency in a customer-facing AI tool without giving up its usefulness?
Lower the temperature setting (or select a 'precise' preset) for anything stating a fact, and replace open-ended prompting with structured templates that pull from your actual current policy or pricing data rather than asking the model to recall it from general training. Add a human or a rules check before anything binding goes out — a quoted price or a stated exception a person confirms is far safer than one generated fresh each time. None of this requires giving up AI for the task; it requires giving the task less room to improvise.
Should I worry about inconsistency in a brainstorming or drafting tool?
Generally no. If you're asking for five headline options or a first-draft summary and a human reviews the output before anything ships, variation between runs is the tool doing exactly what it's meant to do — producing range for you to choose from. Fighting for identical output from a creative tool spends effort on a problem that isn't actually costing you anything. Save the scrutiny for tools stating something a customer or regulator could hold you to.
Not sure where your AI tools actually sit
Tell us which of your customer-facing AI tools states a price, a policy, or an eligibility decision, and whether you've ever tested it by asking the same question twice. We'll tell you where sampling variability is a real exposure for your business and where it's a non-issue not worth touching. We don't sell AI platforms, so there's nothing riding on the answer either way.
Ask for a consistency checkSources, read 18 September 2026: Thinking Machines Lab's engineering research on nondeterminism in LLM inference and why temperature zero does not guarantee identical output; the 2026 arXiv paper "Algorithmic Arbitrariness in Content Moderation," which found roughly 30% of moderation decisions changed under different random seeds; and reporting and legal analysis on Moffatt v. Air Canada (2024), the small-claims tribunal ruling holding Air Canada liable for its chatbot misstating bereavement fare policy, including CBC News and the American Bar Association's Business Law Today. Related: Which Decisions Never to Hand to an AI Agent and Telling Customers They're Talking to AI.
Hero image from Unsplash, used under the Unsplash License.