Why AI “Hallucinates”

Technical system diagram showing how context, instructions, knowledge base, and feedback narrow possible AI paths toward the right answer.

I want to share a thought on why AI “hallucinates.”

I am putting the word “hallucinates” in quotation marks deliberately, because very often this word becomes a way for people to shift responsibility onto AI itself.

When we give a task to a model, we need to remember that we are still working with a neural network. Every word we add changes the probabilities of how the model will move through its knowledge graph. And when the context is weak, those probabilities can lead it down the wrong path.

So I would suggest that what we call AI hallucinations are often situations where the model does not have enough context or enough input data to choose the “right route” through its knowledge graph.

In other words, imagine that at the point where you are currently interacting with the model, there are a thousand possible paths the LLM could take while generating an answer. In a simplified sense, the chance of choosing the right answer may be one in a thousand. Only by adding more input data can you increase that probability, reducing those thousand possible options to a hundred, or even fifty.

This is where the value of a knowledge base and clear instructions for your AI agent becomes obvious. Every additional piece of relevant information you provide reduces the number of possible wrong paths.

The next question I often hear from people I teach to use AI is: how detailed should the instructions be?

I do not think there is one universal answer. A better answer is this: instructions should be as detailed as necessary to significantly reduce the number of wrong decisions.

Let’s take a simple example.

You send your 10-year-old child to the store to buy milk. What do you need to explain?

  1. You need to tell them which store to go to.
  2. You need to make sure they actually understood which store you mean, so you ask for feedback.
  3. You do not just tell them to buy milk. You explain which brand, or what fat percentage, they should buy.
  4. You give them money and explain how much of it can be spent on the milk, and how much change they should bring back.
  5. You probably also tell them to go straight home after the store, instead of staying outside with friends.

I believe the same level of detail is often useful when working with AI.

If you want AI agents to work properly, you need to give them context, constraints, feedback, and instructions that are clear enough to reduce the number of wrong routes they can take.