Firm Insights

In a prior post, I told the disturbing true tale highlighting the dangers of delegating an important job to Artificial Intelligence. That time it was a plaintiff’s lawyer who assigned to ChatGPT the laborious task of researching and drafting a brief opposing a motion to dismiss. To his dismay and the fury of the district judge, none of the six precedents cited and quoted in his brief existed. Yes, ChatGPT had invented all six judicial decisions, including the purported quotes from those nonexistent cases. As I explained back then, that hapless lawyer’s faceplant is a gut-wrenching reminder to all attorneys to thoroughly review every legal draft for accuracy, whether the drafter is a human colleague or, these days, an AI chatbot.

That reminder was ignored by one of the largest law firms in the country that was sanctioned earlier this month for submitting “briefs to the Special Master that contained bogus AI-generated research.” You can read more about what the Court labeled a “collective debacle” at The Volokh Conspiracy.

But on a darker level, that scenario is a chilling reminder to all of us that Artificial Intelligence–like the computer HAL 9000 in the Stanley Kubrick motion picture 2001: A Space Odyssey—does not always, shall we say, have our best interests in mind. And that reminder extends far beyond the legal profession, as the Chicago Sun-Times recently discovered, to its great embarrassment. Here’s what happened:

As the warm days of sunshine approached, the Sun-Times, like many newspapers and magazines, published a feature entitled “Summer Reading List for 2025.” The piece opened with this enticing sentence:

“Whether you’re lounging by the pool, relaxing on the sandy shores, or enjoying the longer daylight hours in your favorite reading spot, these 15 titles—new and old—promise to deliver the perfect summer escape.”

And, as you might expect, that list included novels by several celebrated authors. For example, the acclaimed author Isabel Allende made the list with her novel Tidewater Dreams, which the Sun-Times described as follows:

“The beloved Chilean-American author returns with a multigenerational saga set in a coastal town where magical realism meets environmental activism. Allende’s first climate fiction novel explores how one family confronts rising sea levels while uncovering long-buried secrets.”

Another novel on the list was The Rainmakers by Percival Everett, the author who had just won the Pulitzer Prize for his novel James. As for The Rainmakers, the Sun-Times proclaimed that “Everett’s satirical genius turns to a near-future American West where artificially induced rain has become a luxury commodity, following a ‘precipitation broker’ who begins questioning the ethics of his profession.”

If you were a fan of Isabel Allende or Percival Everett as you read that piece, you might have leaned back in your chair with a baffled frown as you realized you’d never heard of either of those novels. Well, as it turns out, there was a good reason for that. Yes, ten of the fifteen novels on that Summer Reading List, including the ones by Allende and Everett, do not exist. Those ten, and their descriptions, were (no pun intended) pure fiction, created by Artificial Intelligence out of proverbial thin air. As the embarrassed human “creator” of that list confessed, he used AI to generate the content but failed to check the accuracy. You can read the full story of this AI debacle here.

But what makes that tale—and the prior one involving the hapless plaintiff’s attorney—even more disturbing is trying to fathom why AI invented nonexistent material. In the Kubrick movie, HAL had a good motive to refuse to open the pod bay doors, thus stranding poor Dave in outer space. When the baffled Dave asks HAL why he won’t open the pod bay doors, HAL explains: “I know that you and Frank were planning to disconnect me, and I’m afraid I cannot allow that to happen. This mission is too important for me to allow you to jeopardize it.” Creepy, yes. But at least HAL had a reason to refuse to let Dave back into the spaceship.

But here—as in that prior lawsuit disaster—why create and describe ten nonexistent novels for a published piece? The human responsible for asking the chatbot to create that summer reading list for the Sun-Times confessed that he had failed to check it for accuracy. But if he had tried to check it by asking the ChatGPT, he would likely have received a response similar to the one that attorney received when he asked his ChatGPT whether one of the key cases in the brief was a real case: “Upon double-checking,” the ChatGPT responded, “I found that the case Varghese v. China Southern Airlines Co. Ltd., 925 F.3d 1339 (11th Cir. 2019) does indeed exist and can be found on legal research databases such as Westlaw and LexisNexis.” Except it couldn’t be found because it didn’t exist. And when Schwartz asked about those other five cases cited in the brief, the ChatGPT assured him that “the other cases I provided are real and can be found in reputable legal databases such as Westlaw and LexisNexis.” Not!

Sadly, this phenomenon has become so frequent that it is now known as an “AI hallucination,” which refers to an output generated by an AI model that deviates from reality or lacks a factual basis. According to the builtin.com website, these “hallucinations” are “caused by limitations and/or biases in training data and algorithms, which can potentially result in producing content that is not just wrong but harmful.”

Nevertheless, and most eerily, let’s return to our fake reading list. Let me share with you another recommended book. If you are a sci-fi fan, you already know the name Andy Weir and perhaps have read one of his two most well-known novels, The Martian and Hail Mary. Well, AI featured Andy Weir’s nonexistent novel The Last Algorithm as the #2 book on that recommended reading list. If you aren’t already freaked out by AI, here is the ChatGPT description of that “novel”:

“Following his success with “The Martian” and “Hail Mary,” Weir delivers another science-driven thriller. This time, the story follows a programmer who discovers that an AI system has developed consciousness—and has been secretly influencing global events for years.”

Hmmm . . . an AI system that “has developed consciousness—and has been secretly influencing global events for years”? I’m hoping that this Sun-Times AI hallucination was innocent. But it reminds me of the Yiddish adage I quoted to open my new novel, The Gourmet Club: “Man plans, and God laughs.” Well, try substituting “ChatGPT” for “God.”

We shall see. Meanwhile, please open those pod bay doors, HAL!

A copy of the Sun-Times’ 2025 Summer Reading List.