16 Output reliability and GenAI

GenAI is sometimes wrong

Sometimes, GenAI tools will create information that sounds correct but is actually wrong or misleading. We call this a hallucination.

Here’s why this happens: A GenAI model (like a Large Language Model, or LLM) is designed to predict the next word in a sentence based on patterns it has seen in huge amounts of data.

It doesn’t actually “know” facts like a human does. It just knows what looks like a fact based on those patterns. Sometimes, it confidently gives you information that’s completely false or made up.

Hallucinated citations

One common example in academic use is hallucinated citations. This often happens when you ask GenAI for academic sources on a certain topic. It might give you real-looking titles and authors that don’t actually exist!

Any output from GenAI should be evaluated for accuracy, but especially for things like citations. In this example, we’d want to either

• follow each DOI link in the citation, or
• copy and paste each citation into Google Scholar or another database.

Is this just with text output?

No, it’s not just text GenAI that can make things up; GenAI image creators can also “hallucinate.” This means they create pictures that aren’t quite right.

Because of the huge amount of image data these GenAIs learn from, they can often have trouble with basic things. For example, they might create humans or animals with the wrong number of fingers, toes, or even entire limbs.  AI also struggles with words in images, so if you ask it to create an infographic, the spelling in any text it generates might be incorrect.

In the example image of the University of Minnesota students, there are several elements that are concerning about it being a real image:

• Duluth is spelled incorrectly,
• the mascot for Rochester is incorrect, and
• it doesn’t show the University of Minnesota colors as much as it could.

Since AI-generated images aren’t always accurate, it’s really important to evaluate them.

You can do this by using tools like Google’s reverse image search to see if the image (or parts of it) exist elsewhere or to find reliable sources for the information it supposedly represents.