The AI bluff epidemic: why it hallucinates and how to stop it
How to reduce the risk of hallucinations in LLMs? Mashfrog for Procurement adopts behavioral calibration: RAG agents grounded in verified sources, confidence thresholds, XAI, and rare-data controls to deliver reliable and auditable AI decisions.
A recent study conducted by OpenAI researchers reveals the nature of artificial intelligence "hallucinations": not random errors, but a system designed to "make things up" in order to pass tests. It is a very serious risk, especially for companies adopting AI in their core processes. And global business is moving quickly to mitigate it.
The ghost in the machine: anatomy of a hallucination
Imagine a brilliant student who, rather than turning in a blank sheet, makes up historical facts from scratch with flawless command of language. That is exactly what Large Language Models (LLMs) sometimes do.
A groundbreaking study titled Why Language Models Hallucinate, conducted a few months ago by researchers from OpenAI and Georgia Tech, analyzed for the first time—clearly and in great depth—the phenomenon of "hallucinations": the growing tendency of AI to produce fabricated yet highly plausible statements.
When a model is unable to answer a question and at the same time cannot distinguish fact from falsehood, statistical pressure pushes it to generate bold responses instead of admitting uncertainty.
By testing the most advanced models (such as DeepSeek-V3, Meta AI, or Claude) on trivial tasks like counting the letters in a word or identifying specific birth dates, researchers obtained incorrect answers delivered with maximum confidence. In short, AI has been trained to be a serial “test-taker”: it prefers to guess and be wrong rather than say "I don’t know."
The cost of falsehood: the strategic risk for businesses
All of us have experienced at least once correcting a digital assistant and being met with the usual apology: "You’re absolutely right, I’m sorry for the mistake," only to receive immediately afterward a response that says the exact opposite.
While for a private user discovering a hallucination may feel like an amusing case of the human mind getting the upper hand over the machine, for a company that entrusts its core processes to artificial intelligence this phenomenon introduces vulnerabilities and critical risks with sometimes unimaginable consequences.
The core issue lies in test alignment: most current benchmarks reward models that guess, because abstaining is scored as zero points—exactly the same as a wrong answer. This creates an epidemic of penalized uncertainty. For a company, relying on a model that “bluffs” means being exposed to:
- Critical decision-making errors: Basing strategies on aggregated insights that are partially fabricated.
- Contractual and legal risks: Hallucinations in the handling of clauses or regulations that can lead to costly disputes.
- Reputational damage: Providing incorrect data to stakeholders or customers, undermining trust in the brand.
The Mashfrog response: Agents with ethical and "behavioral" training
The researchers’ study suggests an evolutionary path for global benchmarks that would reward model honesty and progressively increase the reliability of responses; however, the timing of this transition is clearly incompatible with the speed at which AI is penetrating businesses.
In this context, Mashfrog for Procurement—the Mashfrog Group entity dedicated to agent-based solutions for the P2P world—has already been implementing concrete tools for some time to harden and safeguard its processes.
Mashfrog’s approach goes beyond building simple automation agents and instead focuses on behavioral calibration. By integrating technologies such as RAG (Retrieval-Augmented Generation), Mashfrog agents do not operate on "abstract memory," but extract information exclusively from reliable and verified contexts, such as corporate ERP systems, procurement platforms, or certified databases (Cerved, Open-es).
The training models adopted by Mashfrog limit hallucinations through three pillars:
- Explicit confidence objectives: Agents are instructed to respond only when they exceed a predefined safety threshold, avoiding systematic guessing.
- Explainable AI (XAI): Every agent decision—from supplier validation to clause analysis—is traceable and verifiable, eliminating the "black box" effect.
- "Singleton" control: Mashfrog closely monitors rare or unique data during pre-training, which research identifies as the most distortion-prone, to ensure outputs are always grounded in reality.
In a market that rewards speed, Mashfrog chooses the path of certified precision, transforming AI from a guessing student into an infallible advisor.