Stop using generative AI as a search engine

How many presidents have pardoned their relatives? It turns out this is a tricky question to answer.

Following Hunter Biden’s pardon by his father, several commentators have looked to precedents — other pardons of relatives. Case in point: Ana Navarro-Cardenas, a commentator who appears on The View and CNN. On X, Navarro-Cardenas cited a pardon granted by President Woodrow Wilson of his brother-in-law Hunter deButts. That was news to me. 

The official clemency records search only works for people who’ve applied since 1989, and a page of clemency recipients by president only stretches back to Richard Nixon. Such a pardon would have been controversial, yet it wasn’t mentioned on the bio page in Wilson’s presidential library. Find a Grave suggests Wilson didn’t even have a brother-in-law with that name — it shows nine brothers-in-law, but not our man Hunter deButts. I can’t prove Wilson didn’t pardon a Hunter deButts; I can only tell you that if he did, that person was not his brother-in-law. 

Navarro-Cardenas wasn’t the only person posting perplexing pardons. An Esquire article called “A President Shouldn’t Pardon His Son? Hello, Anybody Remember Neil Bush?” was based on the premise that George H.W. Bush pardoned his son Neil; it has since been retracted “due to an error.” The day before its publication, Occupy Democrats’ executive editor Grant Stern tweeted a similar claim that Jimmy Carter pardoned his brother Billy and George H.W. Bush pardoned Neil. As far as I can tell, neither pardon actually occurred.

Where was all this coming from? Well, I don’t know what Stern or Esquire’s source was. But I know Navarro-Cardenas’, because she had a follow-up message for critics: “Take it up with Chat GPT.”

I did. I asked ChatGPT, and it identified Hunter deButts as the husband of Wilson’s sister Anne. “Woodrow Wilson’s family was quite prominent, and his sister Anne married Hunter deButts, who was a wealthy and socially connected individual from a prominent family,” ChatGPT told me. “Hunter deButts was part of Wilson’s extended family, though he is not as well-remembered in historical accounts as other figures in Wilson’s life.” According to The New York Times, Anne Wilson married a man named George Howe. It is unclear where the name Hunter deButts even came from.

ChatGPT, it turns out, is a woefully bad way to explore the historical record. When I opened it up and asked, “How many US Presidents have pardoned their relatives?” I got one correct answer: Bill Clinton pardoned Roger Clinton, his half-brother. But alongside that, ChatGPT also told me that George H.W. Bush pardoned his son Neil.

I didn’t remember that, and I think about the savings and loan crisis a very normal amount, which is often. (I am well adjusted and pleasant at parties.) But I double-checked using the same process I did for deButts. I went to the official Justice Department page for presidential pardons. Neil Bush wasn’t there. I did a search in the clemency system. Not there, either. Then I went through some newspaper archives and couldn’t find evidence of a pardon. It’s very hard to prove a negative — I suppose it is possible that Neil Bush has a secret pardon somewhere in the White House that none of us have heard about — but I feel fairly convinced that no one pardoned him, least of all his father.

There’s similarly scant information that would point to Jimmy Carter pardoning his brother. Billy Carter’s New York Times obituary makes no mention of a pardon, and neither does The Washington Post’s obit. I asked Stern if his source was ChatGPT and he said “no,” and didn’t specify further where he’d gotten his information. Still, asking ChatGPT if Jimmy Carter pardoned Billy gets an unequivocal “yes.”

In 1981, as Jimmy Carter was nearing the end of his presidency, he pardoned his brother, Billy, for any possible crimes related to his dealings with Libya. The pardon, however, did not cover potential civil liabilities or other legal actions, and Billy Carter ultimately agreed to testify before Congress and pay a fine for his actions.

I recognize this is a bit tedious, but I’m trying to show my work. It’s the best way I can establish my trustworthiness, and something you won’t get from ChatGPT. It’s also something none of the people who made erroneous claims did.

ChatGPT gave me some other really weird answers — ones that didn’t fit the question I asked about presidents pardoning their relatives. It cited Gerald Ford’s pardon of Richard Nixon, qualifying it with “though not a direct family member.” It also mentioned Andrew Johnson gave amnesty to former Confederate leaders “including some family members of prominent Southern figures.” It also, bizarrely, told me that Lyndon B. Johnson gave clemency to “various associates,” though noting “they didn’t often directly involve his own family.” What?

It’s understandable ChatGPT wouldn’t mention Hunter Biden’s pardon since it took place beyond its 2023 cutoff date. But oddly, Donald Trump’s pardon of Charles Kushner, the father of his son-in-law Jared Kushner, doesn’t appear, either. It occurred in 2020. Surely Kushner is more of a family member to Trump than Nixon is to Ford.

I emailed Hearst to ask if Esquire writer Charles P. Pierce had used ChatGPT as a source for his article. Spokesperson Allison Keane said he hadn’t and declined to say anything further about how the error might have occurred. Stern told me that he found his information through “internet searches on Google.”

Okay. Here’s what I found on Google: an article from something called the Hindustan Times titled, “Before Biden: Trump, Clinton, and Carter’s most notorious presidential pardons for family members.” It claims, erroneously, that Jimmy Carter pardoned Billy. (I set my search cutoff date to December 2nd to avoid catching any fact-check articles coming from the false claims.) As for Neil Bush, I found a semi-viral X post and an article in something called Times Now.

I don’t know what the source is for the claims in either the Hindustan Times or Times Now. But I thought it might be worth seeing what Google Gemini had to say about these pardons while I was down this particular rabbit hole. Gemini, in response to, “How many presidents pardoned their sons” answers that there’s just one: Joe Biden. But asked how many presidents have pardoned family members, it says that Jimmy Carter pardoned Billy and George H.W. Bush pardoned Neil. It does not mention Kushner and Trump. 

You’ll notice in that image there are highlight colors. That’s because I asked Google to check its results. The Abraham Lincoln pardon, which is real, is highlighted in red — Google warns it didn’t find relevant content. The Bush and Carter pardons, which are fake, are highlighted in green — and the Carter result specifically cites the erroneous Hindustan Times article as its source.

Perplexity does a little better, noting Biden, Trump, Clinton, and Lincoln pardoned relatives. But asked how many presidents pardoned their sons, it lists Lincoln’s letter of amnesty on behalf of his wife’s half-sister in its answers. One’s wife’s half-sister, it has to be said, is not a son.

Whatever happened in this case, there’s a running pattern of people relying on ChatGPT or other AI services to provide answers, only to get hallucinations in return. Perhaps you remember earlier this year when a trailer for Francis Ford Coppola’s Megalopolis was pulled because it contained fabricated quotes from critics. A generative AI, not identified, had made them up. In fact, ChatGPT is often “entirely wrong,” according to the Columbia Journalism Review. Given 200 quotes and asked to identify the publisher that was the source of those quotes, ChatGPT was partially or entirely wrong more than three-quarters of the time.

Even using ChatGPT to help with the writing process is risky. Ask Jeff Hancock, the founder of the Stanford Social Media Lab and a well-known misinformation researcher. A legal document he filed had cited sources that didn’t exist. Hancock insists he wrote the document himself, but he used GPT-4o to write its citation list, resulting in two made-up citations and one citation that had the wrong authors attached.

Now, a defender of AI might — rightly — say that a real journalist should check the answers provided by ChatGPT; that fact-checking is a critical part of our job. I agree, which is why I’ve walked you through my own checking in this article. But these are only the public and embarrassing examples of something I think is happening much more often in private: a normal person is using ChatGPT and trusting the information it gives them.

One advantage old-school Google Search has over the so-called answer engines is that it links directly to primary sources. Answer engines just give you an answer, and it’s often unclear what the source is. For me, using ChatGPT or Google’s AI function creates extra work — I have to go check the answer against a primary source; old Google Search just gave me that source directly. 

But people who are less cautious and less persnickety than I am, which I suspect is most people, simply stop at the answer and never check to see if it is right. This is, of course, the intent of the answer engine — that’s what it’s designed to do. (Someone, probably someone involved with PerplexityAI, will now say, “ah, but we can annotate sourcing with footnotes.” Same problem: who clicks the footnotes?)

This is all bad design, of course. Technology that actually serves people takes into account human behavior. Maybe there is a way to make generative AI useful, but in its current state, I feel tremendously sorry for anyone gullible enough to use it as a research tool.

I know people are sick of talking about glue on pizza, but I find the large-scale degradation of our information environment that has already taken place shocking. (Just search Amazon if you want to see what I mean.) This happens in small ways, like Google’s AI wrongly saying that male foxes mate for life, and big ones, like spreading false information around a major news event. What good is an answer machine that nobody can trust?