What does it mean when AI 'hallucinates' - and how to stop it costing you
AI sometimes states confidently incorrect information. This is "hallucination". Knowing when it matters and how to avoid it is the difference between AI helping and AI embarrassing you.
If you have used ChatGPT or Claude for any real work, you will have hit it eventually. The AI states something with absolute confidence. You check it. It is wrong. Sometimes spectacularly wrong - a quoted price, a date, a court case, a person who does not exist.
This phenomenon has a name: "hallucination". It is one of the most important things to understand about AI before you trust it with real business work.
What hallucination actually is
Modern AI models do not look things up. They predict what comes next, word by word, based on patterns from their training data. Most of the time this produces correct, useful output, because correct patterns are common in the training data.
Sometimes, though, the model produces a plausible-sounding answer that is not based on anything real. It does this with the same confidence as when it is right. It does not know that it is making something up. From inside the model there is no difference between the two.
The classic example is asking AI to cite a paper or court case. It will sometimes invent a citation that looks completely real - author, journal, year, page numbers - but does not exist. This caused a US court case in 2023 where a lawyer submitted AI-generated case law that turned out to be fictional.
Why it happens
Three main reasons.
The training data is incomplete. No model has seen every fact. When asked about something outside its training, it falls back on plausible-sounding generation rather than admitting it does not know.
The training data is out of date. Most models have a knowledge cutoff. Anything that happened after that date is unknown territory. Asking about a recent event invites a hallucination.
The prompt invites confident output. "Tell me three case studies of AI in North Wales manufacturing" is a question that demands three answers. If the model has not seen three real ones, the path of least resistance is to invent some.
Where it actually matters
Not everywhere. The risk is in proportion to the cost of being wrong.
Low risk: drafting. "Help me write an email to a client" - if the AI gets a phrase wrong, you spot it on review. The cost is a 30-second edit.
Medium risk: summarising. "Summarise this contract" - if the AI mis-states a clause, the cost is real but bounded. Always cross-check critical clauses against the source document.
High risk: facts presented as authoritative. "What does HMRC say about X?", "Cite three cases that support Y", "What is the threshold for Z?" - getting these wrong can damage your reputation, your client's case, or your business. Never accept these answers without verification.
The five rules
These reduce hallucination risk to manageable levels for most business work.
1. Provide the source material. If you want AI to summarise something, paste in the something. Do not let it work from memory. AI summarising a document you provide is far more reliable than AI summarising "what the document probably says".
2. Tell it what to do when it does not know. Add to your prompt: "If you are not sure, say you are not sure rather than guessing." This is surprisingly effective.
3. Ask for citations and check them. When AI cites a source, copy the citation into a search engine. If it is not findable, it is fabricated.
4. Use AI for first drafts, not final answers. Anything going to a client, regulator or court should be reviewed by the human responsible for the work. See the AI policy template for the discipline.
5. Use a second AI as a checker. Paste the first AI's output into a different model and ask "what is wrong with this?" or "what would you check before relying on this?". Different models have different blind spots; the disagreement is informative.
Where this is heading
Hallucination is getting better. The current generation of models hallucinates less than the one before, and tools that pull from live sources (Bing search, Perplexity, ChatGPT search, Gemini search grounding) reduce the problem further. Retrieval-augmented generation is one of the main reasons.
But "less" is not "never". The skill of treating AI output with appropriate scepticism is going to be useful for years. Building it into your habits now is cheaper than learning it after a public mistake.
The honest summary
AI is excellent at language, weak on facts. Use it for drafting, summarising and explaining. Verify when it claims something specific and authoritative. Always have a human review work that goes external.
If you would like to walk through your specific use cases and where the risks sit, that is what a discovery call is for. The 30-day plan post has the broader structure for adoption.