Updated September 2026: rewritten with new research and verified botshit cases.

AI is now part of everyday business work. It drafts proposals, answers customers, summarizes contracts, builds financial models, and writes investor updates. It is fast, fluent, and confident. It is also, sometimes, completely wrong, and it sounds exactly as sure of itself as it is.

That gap between confidence and accuracy has a name: botshit. And it has moved well beyond embarrassing chatbot answers. It now shows up in court filings, customer promises, and paid consulting reports, with real financial and reputational consequences.

The good news is that the fix isn’t avoiding AI. It’s knowing when to trust it, when to check it, and how to build that checking into the way your business works.

What botshit is, and why AI makes things up

The term comes from management researchers Timothy Hannigan, Ian McCarthy and André Spicer, writing in Business Horizons in 2024, and in a companion Harvard Business Review article the same year. They draw a useful line. A hallucination is what the AI produces: coherent-sounding content that is inaccurate or made up. It becomes botshit when a person takes that content and uses it in real work.

That distinction matters because it puts responsibility where it belongs. The AI generates the error, but a human decides whether it goes into the contract, the pitch deck or the court filing.

Why does AI make things up at all? Chatbots don’t look facts up the way a database does. They generate the most plausible next words based on patterns in their training. Most of the time, plausible and true line up. Sometimes they don’t.

OpenAI researchers added another piece of the puzzle in a September 2025 paper. They found that the way models are trained and tested rewards guessing over admitting uncertainty, much like a student who fills in every multiple-choice answer because a blank earns zero. The result is a system that would rather sound sure than say “I don’t know.”

Newer models hallucinate less than earlier ones, especially when they can search the web or work from your own documents. But less is not never, and the confident tone doesn’t change when the facts do.

The cost of getting it wrong

These are real cases, and each one landed on a different part of the business.

In court: the case that started it all. In 2023, two New York lawyers in Mata v. Avianca used ChatGPT for legal research and filed a brief citing court decisions that did not exist. When the judge asked for copies, they went back to ChatGPT, which assured them the cases were real. The court fined the lawyers $5,000 and the story went global.

It was not a one-off. A public database maintained by legal researcher Damien Charlotin had logged more than 2,000 court cases worldwide involving AI-fabricated material by September 2026, up from roughly 200 a year earlier. Penalties have grown from small fines to bar suspensions.

With customers: the chatbot that made up a policy. In 2022, Jake Moffatt asked Air Canada’s website chatbot about bereavement fares after his grandmother died. The chatbot told him he could buy a full-price ticket and claim the discount afterward. That wasn’t the airline’s policy. When Air Canada refused the refund, it argued the chatbot was responsible for its own statements. In February 2024, a British Columbia tribunal rejected that argument and ordered the airline to pay. The amount was small; the precedent was not. Your business owns what your AI says.

In professional work: the consulting report with phantom sources. In 2025, Deloitte Australia delivered a 237-page report to the Australian government, under a contract worth A$440,000. A university researcher found it contained references to academic papers that did not exist and a made-up quote from a federal court judgment. Deloitte acknowledged generative AI had been used in preparing it, issued a corrected version and refunded part of its fee. The reputational cost was far larger than the refund.

The pattern is the same in all three: the AI produced something plausible, a person trusted it, and nobody checked before it reached a judge, a customer or a client.

Match the safeguard to the stakes

Not every AI task needs the same scrutiny. Hannigan, McCarthy and Spicer suggest sorting AI work by two questions: how much does accuracy matter, and how easy is it to check the answer? That gives four modes, each with its own risk.

Framework for managing AI hallucinations

Framework for managing AI hallucinations

Authenticated work is where most of the damage happens: high stakes and hard to check, like legal research, financial analysis and investor materials. The danger is trusting the AI more than it deserves, so every claim needs expert verification.

Automated work is high-stakes but easy to check. The risk is that checking becomes routine and sloppy over time.

Augmented work, like brainstorming, uses AI to spark thinking. Accuracy matters less, but you need to know you’re working with raw material, not facts.

Autonomous work can be handed to AI with light oversight. The risk is losing sight of how the work gets done, so review it now and then.

The lesson for any business: before you hand a task to AI, decide which box it belongs in.

Practical safeguards for your business

You don’t need a large compliance team to keep botshit out of your work. You need a few habits, applied consistently.

  1. Decide where AI is allowed, and where a human signs off. Write a one-page policy. List the tasks where AI can work freely (first drafts, brainstorming), the tasks that need review (anything customer-facing), and the tasks that need a named person’s sign-off (legal, financial and investor documents). See our article on lead-to-sale automation here.
  2. Check every fact, figure, quote and citation before it leaves the building. If AI gives you a statistic or a source, open the source. If you can’t find it, it doesn’t go in. This single habit would have prevented all three cases above.
  3. Ground AI in your own information. Tools that answer from your documents, policies or approved knowledge base hallucinate far less than a general chatbot answering from memory. For customer-facing bots, this is essential.
  4. Make it easy for AI to say “I don’t know.” Tell it, in your instructions, to say when it is unsure or can’t find a source rather than guess. Ask it to show where each claim came from.
  5. Watch for the confidence trap. Polished, fluent answers feel trustworthy. Be most skeptical of neat numbers, perfect quotes and conveniently specific examples, which are exactly what AI invents most convincingly.
  6. Keep humans accountable, not just involved. “A person looked at it” is not enough. Someone’s name should be on anything that matters, and they should know they own it.
  7. Test and monitor customer-facing AI. Before launch, try to trip it up with tricky questions about pricing, refunds and policies. After launch, review conversations regularly and fix what it gets wrong.
  8. Be transparent. If AI helped create something, say so where it matters, and never let a disclaimer stand in for actual checking.

The bottom line

AI is one of the most useful tools a growing business can have, and avoiding it isn’t the answer. But it doesn’t know when it is wrong, so your business has to.

The companies that get burned aren’t the ones using AI. They’re the ones that let fluent answers skip the checks they would never skip for a new hire. Match your safeguards to the stakes, verify what matters, and keep a person accountable. Do that, and AI speeds you up without putting your reputation on the line.

Sources