When AI Gets It Wrong: What Business Owners Must Know About AI Hallucinations
AI tools have become genuinely useful for a lot of business tasks. Drafting emails, summarising documents, generating marketing copy, answering routine questions from customers: these are areas where AI saves real time. But there's a problem built into the technology that doesn't get enough attention, and it's one that can cause serious harm to a business if you're not aware of it.
The problem is called hallucination. It's not a bug that can be patched. It's a fundamental characteristic of how these systems work, and understanding it is the difference between using AI productively and trusting it in ways you shouldn't.
What Hallucination Actually Means
When an AI model hallucinates, it generates information that sounds completely credible but is factually wrong. The model isn't lying. It doesn't know it's wrong. It's doing what it was designed to do, which is produce plausible-sounding text based on statistical patterns in the data it was trained on. It has no mechanism for checking whether a fact is true before stating it.
The outputs look the same whether they're accurate or invented. The confident, fluent tone doesn't change based on correctness. A model might accurately summarise a real regulation and then, in the next sentence, cite a subsection that doesn't exist, and both will read with equal authority.
Microsoft has publicly acknowledged that its AI agents, including Copilot embedded in Windows 11 and Microsoft 365, can produce hallucinated content. Despite this, it has proceeded with broad rollout of these features. That's not necessarily the wrong call, but it does mean the responsibility for verifying AI output sits with you, not the software.
What Goes Wrong in Practice
The consequences of hallucination range from mildly embarrassing to genuinely costly, depending on where the AI output ends up.
A business owner asks an AI assistant to explain their obligations under a particular Australian tax rule. The AI produces a clear, confident explanation that cites a specific ATO provision. The provision either doesn't exist or applies to a different scenario. If the business owner acts on that explanation without checking the ATO website or speaking to an accountant, they may underpay, overpay, or miss a compliance requirement entirely.
A tradie using an AI tool to generate a quote for materials asks it to look up a supplier's current pricing. The AI generates numbers that look like real prices but are either outdated or fabricated. The quote goes to a client, is accepted, and the actual material costs come in significantly higher.
A small law firm uses an AI assistant to draft contract clauses. One clause includes a reference to a statutory provision that, on closer reading, doesn't exist. The clause makes it through into a signed contract. The implications depend on what the clause was meant to do, but discovering it later is an uncomfortable conversation with a client.
A marketing professional asks an AI to research competitors and draft comparative claims. The AI confidently attributes capabilities or pricing to competitors based on information that's either out of date or made up entirely. Those claims go into published marketing material.
Why This Happens
These systems don't reason the way humans do. They're trained on enormous volumes of text and learn to predict what word, phrase, or sentence is likely to follow what came before. They're extraordinarily good at generating fluent, contextually appropriate text. They're not good at distinguishing what they know from what they've inferred, or what's accurate from what just sounds plausible.
Think of it as a very confident student who has read widely but has no way to check their own memory. They'll give you an answer either way, and they'll give it with the same tone regardless of whether they actually know or are pattern-matching from something adjacent that they half-remember.
This is also why hallucinations are hard to predict. An AI might handle a complex question accurately and then hallucinate on something simple. There's no reliable relationship between the difficulty of a question and the likelihood of a wrong answer.
A Second Risk: Prompt Injection
Beyond hallucination, there's a related threat that businesses using AI agents should understand: prompt injection.
If an AI agent can browse the web, read emails, or process documents from external sources, it can be manipulated by malicious content embedded in those sources. An attacker might send your business an email that contains hidden instructions telling the AI to forward sensitive information, take an action on your behalf, or change how it behaves.
This isn't theoretical. Researchers have demonstrated prompt injection attacks against AI assistants integrated into email clients, browsers, and document tools. Microsoft's own documentation acknowledges the risk for its Copilot products. As AI becomes more autonomous, and as businesses begin using AI agents that can take actions rather than just answer questions, this attack surface grows.
A Safety Checklist for Using AI in Your Business
These risks don't mean you should avoid AI tools. They mean you should use them with clear rules about where human review is required.
Verify any factual claim before acting on it. If AI tells you about a regulation, a price, a competitor's feature, or a legal requirement, check it against a primary source. The ATO website, the supplier's own pricing page, or your solicitor's advice are the references, not the AI summary.
Never let AI send communications directly to clients without review. AI-generated emails, quotes, or proposals need a human to read them before they leave your business. Errors in client-facing communications damage trust and can create legal exposure.
Do not use AI as a substitute for legal or financial advice. AI can help you understand concepts, draft questions, or prepare for a conversation with a professional. It cannot replace the professional. This applies to tax, employment law, contracts, and any area where the consequence of a mistake is significant.
Be cautious with AI agents that can take actions. If you're using a tool where the AI can browse, send emails, book things, or access systems on your behalf, understand exactly what access it has and limit it to what's necessary. Review what it has done, not just what it has said.
Treat AI output from external documents with scepticism. If your AI assistant processes a document sent by a third party, be aware that the document could contain embedded instructions designed to manipulate the AI. This is a good reason to keep AI agents away from processing untrusted external files with elevated permissions.
The Right Relationship With AI
None of this should put you off using AI tools. The time savings are real, and for tasks where errors are low-stakes and easy to spot, AI can significantly reduce the grind of routine work. The key is matching the level of trust to the stakes involved.
Use AI to draft the email, then read it before you send it. Use AI to explain a concept, then verify the specific figures with the source. Use AI to generate options and ideas, then apply your own judgement. That combination of AI speed and human verification gets you most of the benefit while managing most of the risk.
The businesses that will get the most from AI over the next few years won't be the ones who trust it blindly. They'll be the ones who understand what it's good at, know its limits, and build sensible habits around both.
