I search my name on a regular basis, not only because I am an ego monster (although I try not to pretend that I’m not) but because it’s a good way for me to find reviews, end-of-the-year “best of” lists my book might be on, foreign publication release dates, and other information about my work that I might not otherwise see, and which is useful for me to keep tabs on. In one of those searches I found that Grok (the “AI” of X) attributed to one of my books (The Consuming Fire) a dedication I did not write; not only have I definitively never dedicated a book to the characters of Frozen, I also do not have multiple children, just the one.

Why did Grok misattribute the quote? Well, because nearly all consumer-facing “AI” are essentially “fancy autocomplete,” designed to find the next likely word rather than offer factual accuracy. “AI” is not actually either intelligent or conscious, and doesn’t know when it’s offering bad information, it just runs its processes and gives a statistically likely answer, which is very likely to be factually wrong. “Statistically likely” does not equal “correct.”

Still, I was curious who other “AI” would tell me I had dedicated The Consuming Fire to. So I asked. Here’s the answer Google gave me in its search page “AI Overview”:

I do have a daughter, but she would be very surprised to learn that after nearly 27 years of being called “Athena,” that her name was “Corbin.” I mean, Krissy and I enjoy The Fifth Element, but not that much. Also I did not dedicate the book to my daughter, under any name.

Here’s Copilot, Microsoft’s “AI”:

I have indeed dedicated (or co-dedicated) several books to Krissy, and I’m glad that Copilot did not believe that my spouse’s name was “Leloo.” But in fact I did not dedicate The Consuming Fire to Krissy.

How did ChatGPT fare? Poorly:

I know at least a couple of people named Corey, and a couple named Cory, but I didn’t dedicate The Consuming Fire to any of them. Also, note that ChatGPT not only misattributed to whom I dedicated the book, it also entirely fabricated the dedication itself. I didn’t ask for the text of the dedication, so ChatGPT voluntarily went out of its way to add extra erroneous information to the mix. Which is… a choice!

I also asked Claude, the “AI” of Anthropic, and to its (and/or Anthropic’s) credit, it was the only “AI” of the batch which did not confidently squirt out an incorrect answer. It admitted it did not have reliable search information on the answer and undertook a few web searches to try to find the information, and eventually told me it could not find it, offering advice instead on how I could find the information myself (for the record, you can find the information online; I did by going to Amazon and searching the excerpt there). So good on Claude for knowing what it doesn’t know and admitting it.

Interestingly, when I went to Grok directly and asked to whom the book was dedicated, it also said it couldn’t find that information. When I asked it why a different instance of itself incorrectly attributed a different dedication to the book, it more or less shrugged and said what I found to be the equivalent of “dude, it happens.” I also checked Gemini directly (which as I understand it powers Google’s Search “AI” Overview) to see if it would also say “I can’t find that information.” Nope:

I’m sure this comes as a surprise to both Ms. Rusch and Mr. Smith, who are (at least on my side) collegial acquaintances but not people I would dedicate a book to. And indeed I did not. When I informed Gemini it had gotten it wrong, it apologized, misattributed The Consuming Fire to another author (C. Robert Cargill, who writes great stuff, just not this), and suggested that he dedicated the book to his wife (he did not) and that her name was “Carly” (it is not).

(I also informed Copilot that it had gotten the dedication wrong, and it also tried again, asserting I dedicated it to Athena. I’m glad Copilot got the name of my kid right, but as previously stated, The Consuming Fire is not dedicated to her.)

So: Five different “AI” and two iterations of two of them, and only Claude would not, at any point, offer up incorrect information about the dedication in The Consuming Fire. Which I will note does not get Claude off the hook for hallucinating information. It has done so before when I’ve queried it about things relating to me, and I’m pretty confident I can get it to do it again. But in this one instance, it did not.

None of them, not even Claude, got the information correct (which is different from “offered up incorrect information”). Two of them, when informed they were incorrect, “corrected” by offering even more incorrect information.

I’ve said this before and I will say it again: I ask “AI” things about me all the time, because I know what the actual answer is, and “AI” will consistently and confidently get those things wrong. If I can’t trust it to get right the things I know, I cannot trust it to get right the things I do not know.

Just to make sure this confident misstating of dedication facts was not personal, I picked a random book not by me off my shelf and asked Gemini (which was still open in my browser) to name to whom the book was dedicated.

It certainly feels like Richard Kadrey might dedicate a book in the Sandman Slim series to the lead singer of The Cramps, but in fact Aloha From Hell is not dedicated to him.

Let’s try another:

Daniel H. Wilson’s Robopocalypse may be dedicated to his wife, but if it is, her name is not “Kellie,” as that is not the name in the dedication.

Let’s see if the third time’s the charm:

It’s more accurate to say this was a third strike for Gemini, as G. Willow Wilson did not dedicate Alif the Unseen to a Hasan, choosing instead her daughter, whose name that is not.

So it’s not just me, “AI” gets other book dedications wrong, and (at least here) consistently so. These book dedications are actual known facts anyone can ascertain — you can literally just crack open a book to see to whom a book is dedicated — and these facts are being gotten wrong, consistently and repeatedly, by “AI.” Again, think about all the things “AI” could be getting wrong that you won’t have such wherewithal to check.

What do we learn from this?

One: Don’t use “AI” as a search engine. You’ll get bad information and you might not even know.

Two: Don’t trust “AI” to offer you facts. When it doesn’t know something, it will frequently offer you confidently-stated incorrect information, because it’s a statistical engine, not a fact-checker.

Three: Inasmuch as you are going to have to double-check every “fact” that “AI”” provides to you, why not eliminate the middleman and just not use “AI”? It’s not decreasing your workload here, it’s adding to it.

Does “AI” have uses? Possibly, just not this. I don’t blame “AI” for any of this, it’s not those programs’ fault that the people who own and market them and know they are statistical matching engines willfully and, bluntly, deceitfully position them to be other things. You don’t blame an electric bread maker when some fool declares that it’s an excellent air filter. But you shouldn’t use it as an air filter, no matter how many billions of dollars are being spent to convince you of its air-filtering acumen. Use an actual air filter, damn it.

I dedicate this essay to everyone out there who will take these lessons to heart and not trust “AI” to tell you things. You are the real ones. And that’s a fact.

— JS

popular shared this story from Schneier on Security.

In the span of just weeks, the US government has experienced what may be the most consequential security breach in its history—not through a sophisticated cyberattack or an act of foreign espionage, but through official orders by a billionaire with a poorly defined government role. And the implications for national security are profound.

First, it was reported that people associated with the newly created Department of Government Efficiency (DOGE) had accessed the US Treasury computer system, giving them the ability to collect data on and potentially control the department’s roughly $5.45 trillion in annual federal payments.

Then, we learned that uncleared DOGE personnel had gained access to classified data from the US Agency for International Development, possibly copying it onto their own systems. Next, the Office of Personnel Management—which holds detailed personal data on millions of federal employees, including those with security clearances—was compromised. After that, Medicaid and Medicare records were compromised.

Meanwhile, only partially redacted names of CIA employees were sent over an unclassified email account. DOGE personnel are also reported to be feeding Education Department data into artificial intelligence software, and they have also started working at the Department of Energy.

This story is moving very fast. On Feb. 8, a federal judge blocked the DOGE team from accessing the Treasury Department systems any further. But given that DOGE workers have already copied data and possibly installed and modified software, it’s unclear how this fixes anything.

In any case, breaches of other critical government systems are likely to follow unless federal employees stand firm on the protocols protecting national security.

The systems that DOGE is accessing are not esoteric pieces of our nation’s infrastructure—they are the sinews of government.

For example, the Treasury Department systems contain the technical blueprints for how the federal government moves money, while the Office of Personnel Management (OPM) network contains information on who and what organizations the government employs and contracts with.

What makes this situation unprecedented isn’t just the scope, but also the method of attack. Foreign adversaries typically spend years attempting to penetrate government systems such as these, using stealth to avoid being seen and carefully hiding any tells or tracks. The Chinese government’s 2015 breach of OPM was a significant US security failure, and it illustrated how personnel data could be used to identify intelligence officers and compromise national security.

In this case, external operators with limited experience and minimal oversight are doing their work in plain sight and under massive public scrutiny: gaining the highest levels of administrative access and making changes to the United States’ most sensitive networks, potentially introducing new security vulnerabilities in the process.

But the most alarming aspect isn’t just the access being granted. It’s the systematic dismantling of security measures that would detect and prevent misuse—including standard incident response protocols, auditing, and change-tracking mechanisms—by removing the career officials in charge of those security measures and replacing them with inexperienced operators.

The Treasury’s computer systems have such an impact on national security that they were designed with the same principle that guides nuclear launch protocols: No single person should have unlimited power. Just as launching a nuclear missile requires two separate officers turning their keys simultaneously, making changes to critical financial systems traditionally requires multiple authorized personnel working in concert.

This approach, known as “separation of duties,” isn’t just bureaucratic red tape; it’s a fundamental security principle as old as banking itself. When your local bank processes a large transfer, it requires two different employees to verify the transaction. When a company issues a major financial report, separate teams must review and approve it. These aren’t just formalities—they’re essential safeguards against corruption and error. These measures have been bypassed or ignored. It’s as if someone found a way to rob Fort Knox by simply declaring that the new official policy is to fire all the guards and allow unescorted visits to the vault.

The implications for national security are staggering. Sen. Ron Wyden said his office had learned that the attackers gained privileges that allow them to modify core programs in Treasury Department computers that verify federal payments, access encrypted keys that secure financial transactions, and alter audit logs that record system changes. Over at OPM, reports indicate that individuals associated with DOGE connected an unauthorized server into the network. They are also reportedly training AI software on all of this sensitive data.

This is much more critical than the initial unauthorized access. These new servers have unknown capabilities and configurations, and there’s no evidence that this new code has gone through any rigorous security testing protocols. The AIs being trained are certainly not secure enough for this kind of data. All are ideal targets for any adversary, foreign or domestic, also seeking access to federal data.

There’s a reason why every modification—hardware or software—to these systems goes through a complex planning process and includes sophisticated access-control mechanisms. The national security crisis is that these systems are now much more vulnerable to dangerous attacks at the same time that the legitimate system administrators trained to protect them have been locked out.

By modifying core systems, the attackers have not only compromised current operations, but have also left behind vulnerabilities that could be exploited in future attacks—giving adversaries such as Russia and China an unprecedented opportunity. These countries have long targeted these systems. And they don’t just want to gather intelligence—they also want to understand how to disrupt these systems in a crisis.

Now, the technical details of how these systems operate, their security protocols, and their vulnerabilities are now potentially exposed to unknown parties without any of the usual safeguards. Instead of having to breach heavily fortified digital walls, these parties  can simply walk through doors that are being propped open—and then erase evidence of their actions.

The security implications span three critical areas.

First, system manipulation: External operators can now modify operations while also altering audit trails that would track their changes. Second, data exposure: Beyond accessing personal information and transaction records, these operators can copy entire system architectures and security configurations—in one case, the technical blueprint of the country’s federal payment infrastructure. Third, and most critically, is the issue of system control: These operators can alter core systems and authentication mechanisms while disabling the very tools designed to detect such changes. This is more than modifying operations; it is modifying the infrastructure that those operations use.

To address these vulnerabilities, three immediate steps are essential. First, unauthorized access must be revoked and proper authentication protocols restored. Next, comprehensive system monitoring and change management must be reinstated—which, given the difficulty of cleaning a compromised system, will likely require a complete system reset. Finally, thorough audits must be conducted of all system changes made during this period.

This is beyond politics—this is a matter of national security. Foreign national intelligence organizations will be quick to take advantage of both the chaos and the new insecurities to steal US data and install backdoors to allow for future access.

Each day of continued unrestricted access makes the eventual recovery more difficult and increases the risk of irreversible damage to these critical systems. While the full impact may take time to assess, these steps represent the minimum necessary actions to begin restoring system integrity and security protocols.

Assuming that anyone in the government still cares.

This essay was written with Davi Ottenheimer, and originally appeared in Foreign Policy.

As always, thanks for using my Amazon Affiliate links (US, UK, Canada), and for considering joining my Patreon

As an Amazon Associate, I earn from qualifying purchases.

Click here to go see the bonus panel!

Hovertext:
You ever imagine how you'd feel if there were constant fights between anonymous well-armed vigilante factions. Eventually everyone would just move to the suburbs.