How to Use AI at Work When It Sometimes Makes Things Up

In a Harvard Business School and BCG field experiment involving 758 consultants, participants using GPT-4 on a difficult brand-strategy task that required reconciling spreadsheet data with subtle interview notes performed worse than those without AI access. Accuracy dropped to 70.6% among GPT-4 users. Among those who also received a prompt-engineering overview, it dropped further, to 60%.

Read that again. More prompting skill, worse results.

This is the hallucination problem as it actually shows up in business. Not the dramatic, obviously wrong answer. The confident, fluent, plausible answer that is quietly wrong in a way that only someone with domain knowledge would catch. That is the failure mode worth building your workflow around.

One position here is simple: AI hallucinations are a workflow problem, not a model problem. You cannot wait for the models to get better. You manage this now, by designing how work flows through the system.

Why the Model Sounds Confident When It Shouldn't

Understanding the mechanism makes the solution obvious.

A large language model generates text by estimating which word is most likely to follow the previous words, then the next, then the next. It is not checking whether claims are true. It is not aware that a source does not exist. It is producing text that matches the pattern of a correct answer.

Patrick Gilbert covers this distinction in Never Always, Never Never, drawing on Sam Tomlinson's framing: an LLM gives you a response that matches the pattern of a correct response. Whether that response is actually correct is a separate question entirely. Models are probabilistic systems, not deterministic ones. They do not know when they are wrong. They just know what comes next.

This explains three failure modes that show up repeatedly in business use:

• Fluency masks errors. A grammatically polished, well-structured paragraph feels authoritative. That feeling has no relationship to accuracy.
• Gaps get filled. If your prompt implies that a report, person, or regulation exists, the model will often construct a plausible completion rather than stop and say it cannot find one.
• Confidence is not calibrated. Models use the same assured tone whether drawing on strong evidence or inventing a likely-sounding answer.

None of this is a bug in a fixable sense. It is what probabilistic text generation does. The practical question is not "how do we stop the model from hallucinating" but "how do we design work so that hallucinations get caught before they matter?"

If you want to go deeper on how AI works under the hood, the core mechanics are worth understanding before you build any workflow around these tools.

The Jagged Frontier: Where AI Helps and Where It Hurts

BCG's experiment did not show that AI is unreliable. It showed that AI is unreliable in specific conditions. On innovation and market-oriented tasks, the same consultants completed 12.2% more work, finished roughly 25% faster, and produced output rated approximately 32% higher in quality.

Same tool. Radically different outcomes. The difference was the nature of the task.

Researchers have started calling this the "jagged frontier." Some tasks sit clearly inside the frontier, where AI performs well. Others sit just outside it, where it fails in ways that are hard to predict from the outside. The difficult brand-strategy task in the BCG experiment looked like a research task. It was actually a contradiction-detection task. Models cannot reliably notice when two pieces of information conflict, especially when one is explicit and one is implied.

A separate preregistered field experiment with 776 Procter & Gamble professionals found that individuals working with AI matched the performance of teams working without AI on product-innovation challenges. That is a real productivity finding. It also does not tell you anything about whether AI is safe for compliance decisions, financial reporting, or legal interpretation.

An operating principle follows from this: use AI freely where mistakes are cheap and visible; require source verification where facts matter; require accountable expert review where errors create legal, financial, safety, or reputational consequences.

That principle needs to be a written policy, not a mental model that lives in one person's head. See writing an AI policy people will actually follow for the structure.

The Answer Stack Problem

Another layer of this: the same AI tool can produce its answer from very different sources depending on how the question is framed, and you often cannot tell which source it used.

Six layers make up the AI Answer Stack: base model knowledge, prompt context, reasoning, retrieval from your documents, live web search, and deep research. Each has different reliability characteristics.

Base model knowledge is fast but frozen at a training cutoff and potentially wrong on specifics. Live web search is current but noisy. Retrieval from your own documents is accurate within scope but fails silently if the right document was not retrieved. Models themselves do not announce which layer they used. They just answer.

Two people asking the same question on different days, or with slightly different phrasing, may get answers from completely different layers of the stack. One answer might be grounded in a document you uploaded. Another might be a confident confabulation from base model knowledge. Both will look identical on the surface.

Fix this not by trusting the answer less, but by forcing the system to show its work.

The Scale of the Problem

Adoption data makes this urgent. McKinsey's 2025 global survey of 1,993 respondents found that roughly one-third of organizations had started scaling AI across the enterprise. Deloitte found that only 25% of organizations had moved at least 40% of their AI pilots into production. A MIT-related analysis put the share of S&P 500 companies with AI deeply embedded in their businesses at 11% by the end of 2025, with 45% still running pilots.

Closing the pilot-to-production gap is not mainly a technology problem. It is a workflow and governance problem. Organizations can get a model to produce impressive output in a demo. The hard part is knowing which outputs to trust, who is accountable when something is wrong, and how to build checking into the process without erasing the time savings.

A widely circulated 2025 analysis attributed to MIT's NANDA initiative found that 95% of organizations piloting generative AI reported no measurable profit-and-loss return. That figure comes through a secondary summary rather than the original publication, so treat it as directional rather than precise. But the direction it points is consistent with everything else: access to a model is easy. Redesigning work around it is hard.

Most organizations are measuring AI adoption by counting licenses and prompts written. That is measuring input. The AI ROI measurement post covers what actually constitutes a business result.

Task Classification: The First Decision

Before anyone on your team runs a task through AI, they need a mental model for which category of work it belongs to. Three tiers:

Tier 1: Use AI with light review.

Brainstorming, rewriting for clarity, creating outlines, summarizing documents you have supplied to the model, converting notes into action items, drafting internal communications, producing first-pass code or formulas that will be tested, classifying material where mistakes are reversible. Errors here are low-cost and visible. Run fast.

Tier 2: Require source verification and human review.

Market, competitor, or regulatory research. Financial analysis. Customer communications. Policy interpretation. Executive briefings. Any output containing dates, numbers, quotations, names, or legal claims. AI should work from retrieved sources here, with citations. A human who understands the domain checks the output.

Tier 3: Require expert approval before the output goes anywhere.

Legal advice. Medical or safety decisions. Credit, employment, or eligibility decisions. Financial reporting. Public statements about material events. Anything that changes a contract, payment, production system, or customer record. AI can assist in drafting. It cannot be the final authority.

These tiers are not about the tool. They are about the consequence of an error. Depending on context, the same AI tool might be Tier 1 for one task and Tier 3 for another.

Ethan Mollick at Wharton has written extensively on this point: the gains from AI are real, but they depend on workers understanding where the model is reliable and where it is not. That is not something you can delegate to the model itself to determine.

The Verification Failure Most Teams Skip

Most teams do not verify AI output systematically. They read it. Reading is not verification.

Reading an AI-generated document and finding it plausible tells you that the model produced something that sounds like a correct answer. It does not tell you whether the claims inside it are true.

BCG's experiment is a useful warning here. Consultants who performed worst on the difficult task were not careless. They were using the tool as instructed and the tool produced confident, well-written output. Errors were invisible to anyone who did not independently know the right answer.

For Tier 2 and Tier 3 work, the checking process needs to be structural, not incidental. That means a two-pass approach:

1. Generate the draft.

2. Run a separate verification pass that checks every material factual claim against the source it supposedly came from.

Verifiers should not ask whether the draft "sounds right." They should produce an audit: what is the specific claim, where is the source, is the claim actually supported by that source, or was it inferred or invented.

Configuring the model itself to show its work reduces the burden. Instructions like "cite the source title and location for every factual claim," "distinguish between what the source states and what you are inferring," and "say 'not found in the provided sources' when evidence is missing" do not eliminate hallucinations. They make them visible.

The Artifact: AI Task Verification System

Below is a complete, copyable system for managing AI accuracy at work. It has three components: a prompt template for forcing evidence into outputs, a claim audit table for verification, and a task classification card for your team.

---

Component 1: The Evidence-First Prompt

Paste this at the start of any AI task where accuracy matters (Tier 2 or Tier 3 work):

```

You are working from the sources I supply. Do not use general knowledge to answer questions about facts, figures, names, dates, or events. For every factual claim in your response, provide the source title and the specific location (page, section, or paragraph) where that claim appears.

Separate three types of content clearly:

• STATED: information directly quoted or closely paraphrased from a source
• CALCULATED: a number or conclusion derived from source data (show the calculation)
• INFERRED: a conclusion you are drawing that is not directly stated in the sources

If the sources do not contain enough information to answer the question, say so plainly. Do not fill the gap with general knowledge unless I explicitly ask you to. Do not invent names, citations, dates, statistics, or case studies.

If you are uncertain whether a claim is supported, flag it rather than state it confidently.

```

---

Component 2: The Claim Audit Table

After generating any Tier 2 or Tier 3 output, run this audit before the document leaves the team. One row per material claim:

| Claim | Source cited | Location verified? | Actually supported? | Action |

|---|---|---|---|---|

| [specific factual claim] | [source name] | Yes / No | Yes / Partial / No | Keep / Revise / Remove |

| [numerical claim] | [source name] | Yes / No | Yes / Partial / No | Keep / Revise / Remove |

| [interpretation or conclusion] | [evidence] | Yes / No | Inferred / No basis | Label as inference / Remove |

If "Actually supported?" is No for any claim, that claim must be removed or corrected before the document is used. Do not mark it for later follow-up and move on.

---

Component 3: Task Classification Card

Print this or add it to your team's AI guidelines doc. Have people refer to it before they start any AI-assisted task.

Before you use AI, ask two questions:

1. What is the worst realistic outcome if this output contains an error?

2. Would I know if the answer were wrong?

Then classify:

• Mistake is cheap and visible → Tier 1. Proceed with light review.
• Mistake is costly or hard to detect → Tier 2. Use the Evidence-First Prompt. Run the Claim Audit before the output leaves your hands.
• Mistake creates legal, financial, safety, or reputational risk → Tier 3. AI can draft. A qualified human must review and sign off. Document who approved it and when.

Red flags that bump a task up a tier:

• The output contains dates, dollar amounts, percentages, names, or citations
• The output will be sent to someone outside your team
• The output will inform a decision with real consequences
• You would not know independently if the answer were wrong

---

What Leaders Need to Change

Evidence points to operating changes, not tool changes.

Assign a named business owner to each AI use case. That person is accountable for the outputs, not for the prompts. Define what employees remain responsible for. A policy that says "use AI responsibly" is not a policy. It is a liability.

Create a small AI enablement function, even if it is one part-time role. That function's job is to maintain approved tools, provide prompt templates, run training, and evaluate which use cases are delivering real results. At AdVenture Media, working through this problem practically has shown that the difference between AI adoption that produces measurable output and AI adoption that produces busy work usually comes down to whether someone owns the workflow, not just the tool.

Train employees on verification, not just prompting. A short prompting course will not protect your team from failure on tasks outside the model's reliable capability frontier. BCG's experiment showed that explicitly. Train people to detect unsupported claims, missing context, false precision, and citation mismatch. Train them on when not to use AI.

Measure the whole process. Track error and rework rates, human review time, the percentage of outputs accepted without major correction, and the business outcome the workflow was supposed to improve. Prompts written and tokens consumed are not business metrics.

For teams earlier in this process, the AI Maturity Ladder is worth reading alongside this. Where your organization sits on that ladder determines which of these changes are realistic in the next quarter.

The Single Thing to Do First

Take one workflow your team already uses AI for. Apply the three-tier classification to it right now. If it is Tier 2 or Tier 3 work, copy the Evidence-First Prompt above and add it to the standard process for that task starting this week. Run the Claim Audit on the next piece of output it produces.

That is not a pilot. That is a workflow change. The difference matters.