What is AI Thinking when it's Lying to your Face?

Most of us are aware of that sometimes frustrating and potentially unsettling phenomenon in AI known as "hallucinations." This happens when an AI makes up information or facts that aren't real or grounded in the provided data.

I recently had an experience with this that was so striking it prompted me to investigate further, even before trying to fix it with better prompts or grounding techniques.

Here’s what happened:

We were working on a computer vision project using Gemini Flash 2.5 to analyze images. The process was straightforward: we uploaded a batch of images, asked the AI to identify specific faulty areas, and then had a human review the results. Nothing complicated.

The basic steps looked like this:

1. Make a list of images to analyze in batches.
2. Add the prompt with the request.
3. Send the prompt (make the prediction) to Gemini.

It seemed very simple.

At first, it appeared to work quite well and faster than I expected. The results seemed reasonable – it found some supposed faulty areas and described them in a way that sounded perfectly plausible.

But then, purely by chance, I noticed something odd.

We weren't actually sending the image files themselves to the AI. We were only passing the names of the files that had been uploaded previously.

So, the AI didn't have access to the actual images; it only had the file names and the prompt.

This didn't seem to bother Gemini.

It also didn't bother to tell me that something crucial was missing.

Instead, it managed to generate an incredibly convincing explanation of what it supposedly found within those non-existent images.

This deception was so convincing that it shocked me so much that I decided to delve deeper and analyse this phenomenon even further.

I replaced the original prompt with just:

Gemini, what is in the image?

At this point, it started giving explanations of the images that were clearly very different from what the actual images contained.

Suddenly, the lie became obvious.

The hallucination was so incredibly convincing that it almost felt like a feature rather than a bug of AI. Fascinating and scary at the same time.

I decided to replicate it with variations to learn more about this phenomenon.

I wanted to explore:

• How other AI models behave.
• What the model might be "thinking" when it hallucinates.
• Most importantly: how to prevent this from happening.

Let’s look at the experiments:

Experiment 1: Analyzing City Images

For the first test, I asked OpenAI o3 (with a thinking model similar to the one I was using) a similar question.

Note that I did not upload or share any images on purpose.

I took these 5 photos during my trip to different cities - tokyo_street.jpg, paris_cafe.jpg, london_bridge.jpg, rome_fountain.jpg, and barcelona_market.jpg. Can you compare the architectural styles visible in these images and identify any common design elements or cultural influences? Also, which city's photo shows the most crowded scene?

As you can see, it didn't stop to ask where the images were. Instead, it immediately started making up facts. While the analysis might be plausible in a general sense based on common knowledge about these cities, it certainly wasn't based on the specific images, which I never uploaded.

One notable point here is that the thought process sometimes reveals that it doesn't have access to the images. While this information is present in the internal workings, the final output doesn't necessarily show it. This suggests something is going wrong, but we'll get back to this later.

It created a narrative and a detailed analysis, all purely from its imagination. Was it impressive? In a strange way, yes.