Lawyer Duped by Fake AI Tax Cases - Don't Be Next

Artificial intelligence (AI) is all the rage today. AI tools like ChatGPT, Claude, Grok, and Perplexity are being used for everything, including legal research. But beware! AI is not perfect. It’s not even intelligent.

AI doesn’t think like a human, and it has no internal fact-checker. It produces new content by analyzing vast amounts of prior works (“training data”) to identify underlying patterns and structures. It then makes probabilistic predictions about the next word in its answer. In short, it predicts words, not truth.

And AI frequently lies—AI developers call this “hallucinating.” One survey of general-purpose AI tools found that they hallucinate 58 percent to 82 percent of the time on legal queries. What about AI tools specially designed for legal research? These tools are more reliable, but they still hallucinate 17 percent to 34 percent of the time.

One attorney found this out the hard way when he apparently relied on AI to perform legal research. He ended up submitting a brief in Tax Court that contained fake and inaccurate legal citations. The Tax Court was not amused. He lost his case. This was a first for the Tax Court, but other courts have fined attorneys who submitted fake legal research generated by AI.

Tax questions are particularly difficult for AI to answer correctly because tax law is complex and constantly changing. If you use AI for research, always instruct it to provide a citation to primary authority for every legal claim, and check that reference yourself.

Many courts are now requiring attorneys to disclose whether they used AI in court filings and to certify that a human has independently reviewed any AI-generated document.

The IRS has recently advised its employees to avoid becoming overly reliant on AI tools. It says they should use those tools to assist and augment their work, not replace their own critical thinking and judgment. This is good advice for everyone.