Why AI Makes Things Up

This is the most important thing to understand about these tools, and it does not go away with better models or paid subscriptions. Understanding the mechanism tells you exactly where to be careful.

Why it happens

A language model is trained to produce likely continuations of text. Given a question, it generates the words that most plausibly follow — based on patterns across an enormous amount of writing.

It is not looking anything up. There is no internal database being consulted, no fact being retrieved, no confidence value attached to a claim. When it writes "the study was published in the Journal of Applied Psychology in 2019", it is producing a sequence that fits the shape of a real citation. Whether that specific study exists is not something the process checks.

This is why fabrications are so convincing. They are not errors in the sense of a typo or a miscalculation. They are the system working exactly as designed, applied to something it has no information about. A made-up citation looks right because looking right is precisely what the model optimises for.

The problem is well documented in the research literature — A Survey on Hallucination in Large Language Models catalogues the causes and the mitigation approaches, and its conclusion is the same one worth carrying into daily use: this is a known, structural property of how the models work, not a bug awaiting a patch.

Where it fails most

Risk	Why
Citations and references	Highest risk. Format is highly patterned, so plausible fakes are easy to generate.
Specific numbers and statistics	A plausible-looking figure is generated as readily as the true one.
Quotes	Often paraphrased, misattributed, or invented entirely.
Recent events	Training has a cutoff. Without live search it may confidently describe things that never happened.
Obscure or niche facts	Sparse training data means more gap-filling.
URLs	Frequently constructed to look right and lead nowhere.
Arithmetic	Predicts the text of an answer rather than computing it.

Two of those are worth expanding, because they cause the most real damage.

Citations. Fabricated references are the classic failure: right format, plausible journal, real-sounding authors, a title that fits the subject perfectly. Lawyers have been sanctioned for filing briefs citing cases that did not exist. Students have submitted bibliographies where nothing was findable. The reference looks so correct that people do not think to check.

Arithmetic. These models do not calculate; they predict what a calculation's answer tends to look like. For anything numerical, use something that actually computes — a spreadsheet, or a purpose-built tool like our percentage calculator or compound interest calculator.

Confidence tells you nothing

The hardest habit to build, because it runs against how we judge human answers.

With a person, hesitation is a signal — "I think it was around 2019, but check that" means something. With a language model, tone is generated the same way as content. It has no internal sense of certainty to express, so a fabricated citation arrives in precisely the same assured register as a well-established fact.

This has a specific consequence for how you use these tools: you cannot use the model's confidence to decide what to verify. You have to decide from the category of claim instead — is this a citation, a number, a quote, a recent event? — because that is the only signal available.

What actually reduces it

• Give it the source material. By far the most effective. Paste the document and ask questions about it. Working from text in front of it is much more reliable than working from memory.
• Use a tool with live search and citations. When it retrieves and links sources, you can check them. Still check them — the link may not say what the summary claims.
• Ask what it is unsure about. Imperfect, but asking it to flag uncertain claims does surface some.
• Ask for the reasoning. Explanations expose faulty steps that a bare answer hides.
• Keep questions specific. Broad questions invite gap-filling; narrow ones give it less room.

What does not fix it

• Paying for a subscription. Stronger models fabricate less often, but they still fabricate — and more fluently, which can make it harder to spot. Whether paying for AI is worth it is a real question, but reliability is not the reason.
• Asking "are you sure?" This often produces an apology and a different answer, which may be equally invented. Agreeableness is not verification.
• Asking it to cite sources. Without live retrieval, this frequently generates more fabricated citations rather than fewer — it produces what a cited answer looks like.
• Telling it not to make things up. The instruction is understood; the mechanism is unchanged.

Verifying without redoing the work

Checking everything defeats the purpose. Check by category instead:

1. Every citation, every time. Search the exact title. If nothing comes back from any database, it does not exist. This takes seconds and catches the most damaging failure.
2. Every number that matters. Anything load-bearing for a decision or a claim you are publishing.
3. Every URL. Click it.
4. Anything about recent events, unless the tool retrieved and linked a source.
5. Anything you would be embarrassed to be wrong about. The practical test.

What generally needs less checking: explanations of well-established concepts, rephrasing your own text, structural suggestions, and code you are about to run and test anyway.

The one that catches people out

Fabrications are hardest to spot in areas where you know just enough to follow along but not enough to catch an error. Complete beginners check everything; genuine experts spot problems instantly. The dangerous zone is in between.

If you are using AI to work slightly beyond your competence — which is one of its most valuable uses — that is exactly where verification matters most, and exactly where it feels least necessary.

Relatedly, fabricated citations are the strongest practical signal that a text was AI-generated, which is discussed in whether AI content detectors actually work — a reference that cannot be found is far better evidence than any detector score.

Frequently asked questions

Why does AI make things up?

Because it predicts plausible text rather than retrieving verified facts. A fabricated answer is generated by the same process as a correct one, so nothing internally distinguishes them. It is the system working as designed, applied to something it lacks information about.

What is an AI hallucination?

Content that is fluent, confident and false — an invented citation, a wrong statistic stated firmly, a quote attributed to someone who never said it. The term is slightly misleading, since it suggests a malfunction rather than a normal consequence of how these systems work.

Can AI hallucinations be fixed?

Reduced, not eliminated. Giving the model source material to work from, or using a tool that retrieves and cites live sources, helps substantially. But the underlying mechanism — predicting likely text — is what makes the tools useful, so the risk comes with the capability.

Does paying for AI reduce hallucinations?

Somewhat, but not enough to rely on. Stronger models fabricate less often and more fluently, which can make errors harder to notice. Verification habits matter the same regardless of what you pay.

How do I know if AI is making something up?

You cannot tell from the writing — confidence carries no information about accuracy. Check by category instead: verify every citation, number, quote, URL and claim about recent events, regardless of how certain the answer sounds.

Why does AI invent fake citations?

Because citation format is highly patterned. The model generates something with the right shape — plausible journal, real-sounding authors, a fitting title — without any step that confirms the work exists. Always search the exact title before relying on one.

Is it safe to use AI for research?

As a starting point, yes; as a source, no. Use it to find directions, explain concepts and summarise material you supply. Then verify anything factual against a real source, particularly citations and numbers.

Conclusion

These tools predict plausible text. Most of the time plausible and true coincide, which is what makes them useful — and what makes the exceptions so easy to miss.

You do not need to distrust everything. You need to know which categories fail: citations, numbers, quotes, recent events, obscure facts. Check those every time, ignore how confident the answer sounds, and the tools become genuinely reliable for the things they are actually good at.

Related: writing better AI prompts covers getting more useful output, and Claude vs ChatGPT vs Gemini compares the assistants themselves.