Comments - Breaking: Google liable for hallucinations

Liked by Gary Marcus

You can't have it both ways. You can't say, oh wow, look at our superintelligent machines, way beyond a sackful of PhDs, you can't believe just how clever they are. And then say -- oh, they make the most horrendous mistakes, don't you know, they just make shit up, they do it all the time, it's up to you to check and verify. Don't blame us, it's *your* fault, the user, for using our product and believing what it said.

For me, this is a defining moment. I keep saying, when are they going to call these LLMs out as deceptive technology, as an outright fraud. You can't have it both ways. You can't create a universal encyclopedia that you claim can answer all questions and then say, oh, it makes up stuff all the time.

This has all been deception on a grand scale and it has to end, pronto. This is exactly the way to do it -- you simply make the AI companies responsible for their products. That's all. We have been bewitched by robots and this is how we break the spell. What's so hard about this? Why do AI companies get a pass on being responsible for the dangerous malfunctioning of their products when operated perfectly normally?

I have a screenshot from May 2 2026 of Google AI Overview telling me that 2027 is not next year, it's the year after next. Google AI Overview cannot add 6 + 1. If you bought a calculator and it told you very confidently that 6 + 1 = 6, you would take it back to the shop and get your money back.

Like (49)

Reply (4)

Jun 10 Edited

Fred Malherbe: In the age of specialization, we all often put our trust in the professional aura and (hopefully) that they are about guarding of their own good name.

It looks to me like Google is messing around with whatever basic trust they have built up over time and probably now take for granted.

They might not know it, but in business, and even though CEOs may not personally give a hoot about name or much else, and though tacit, a good name is already monetized, and its flow cycle occurs in the institutions of a CULTURE. Once the horse is out of that barn, especially with social media today, it's gone.

Like (5)

Reply (2)

A failed pocket calculator that underlies a wrong calculation that results in a bridge collapse doesn't probably place much of the liability burden on the calculator maker. A self driving automobile, employing many simultaneous calculations using similar silicon substrate, is not seen in the same way. Essentially, the closer software gets to providing an entire solution to anything it undertakes, the more exposed it is to a determination of culpability and liability.

My experience with AI as a subject matter expert (electrical engineer) is that the answers it yields are general and pretty Captain Obvious in nature. They might seem authoritative for someone without much knowledge otherwise but usually contribute little or nothing to actual thorny problems. But the pressure is towards replicating human thought.

Wait until they have to offer warrantees on their output or otherwise take responsibility for the accuracy and quality of the results they sell. Computer code built by AI and later trusted to maintain security, in cases of failure that result in breaches, theft and other bad outcomes, will leave the AI provider on the hook as effectively a subcontractor.

Like (3)

Reply (2)

No AI provider in their right minds would offer warranties on the output. The microcontrollers that I used to program in IV pumps had boldface language in the documentation disclaiming any suitability for use in a safety critical application, including but not limited to medical devices. I expect the chatbots will get additional or more intrusive disclaimers right in the results, and will force the user through an EULA acceptance that says, “user accepts all responsibility for use of the information provided”.

Further, given U.S. libel / slander standards, I think the German libel judgment will not translate to damage awards for libel in the U.S..

Reply (1)

Jun 12

In the professions, where AI wishes to compete, warranties are generally required. E&O insurance is a must for architects, engineers, designers, lawyers and medical professionals. An inability to obtain it limits or cancels your ability to perform the service.

Like (2)

Reply (1)

Jun 12

Perhaps where we differ is that I think AI will for a long time remain in a helping role, in which a professional assumes responsibility deciding when and how to use AI--particularly with the level of supervision appropriate to the quality of the software. This is the recommendation of the American Association of Artificial Intelligence in their 2025 report. Whether the user community is smart enough to pay attention to this is a different matter, however!

Like (1)

Reply (1)

Jun 12

What fascinates me is this. You use the chatbot to give you information. Now you’re told you can’t possibly trust it, you must go and check and verify that information.

This means going and doing all the searching and validating you would normally have done — except!

In many cases, you now find yourself trying to prove a negative, trying to prove that a reference is not a fake. This is much, much more time-consuming and difficult than going out and finding a real reference out there in the world.

I’ve spent days proving that references were fakes. It’s extremely tedious and demanding work.

There are many applications where automation can genuinely help and be of immediate benefit. Trying to work out if your machine is telling you a pack of lies is not productive.

Like (1)

Reply (1)

Jun 13

“This is much, much more time-consuming and difficult than going out and finding a real reference out there in the world.”

Yes. I think there are two kinds of workers: those who HAVE been screwed by their tools and those who WILL get screwed by their tools.

Jun 11 Edited

Cranky Frankie: Yes--and your note also points to the distinction between working on questions that (a) arise from the base of non-conscious material, which is rooted in natural/physical laws and which can be described mostly as "pretty Captain Obvious in nature" (as you say) . . .

. . . and questions that (b) arise from the base of inborn consciousness which, as (here and now) in the sentient IS rooted in the non-conscious; but which refers mostly, and within its own parameters, to the laws and rules that humans devise to live by aka, the normative.

The great exception is consciousness itself, which is both natural (as inborn) and the basis through which people develop and become able to devise normative laws and rules and ways of being to live by. But concrete normativity is a result (like stopping at red lights) of human developmental factors, which points to huge varieties between people living in similar or different cultures; and to dialectical factors which includes reference to philosophical and other foundational differences. So there are two further methodological concerns that we need to add to the methods of the natural and statistical sciences--but as far as I can tell, "we" have not done that yet, at least in any systematic way (correct me if I am wrong here).

And there's the crux, and further If we understand AI as a great library, then the question of public/private raises its (ugly?) head but, as based on what's best for ALL, public wins hands down and so the political climate of democracy of some kind comes into view.

There's probably no smooth transition to be had, especially with so much money involved; but it's what your note and others here seems to be calling for with the seemingly unsolvable problems of responsibility? Also, the problem of over-selling has its main root in the lack of adequate foundations, e.g., the missing theories of consciousness that most are "stuck" on.

There's almost nothing in life more fragile than a reputation. A company can spend decades building up its aura and have it completely destroyed in one incident. This court ruling should reverberate across the industry. OMG, we're being held accountable for our products.

Like (3)

Reply (1)

Fred Malherbe: OMG indeed.

A shared remembrance: while teaching an ethics class in a small college in Virginia, I taught from Aristotle's "Nicomachean Ethics"--there was always a great discussion flowing from the section where he treats the importance of one's reputation. I always loved teaching from that text.

Like (3)

Reply (1)

Okay, now I have a new book to look up and read, clearly. This is why people love moral philosophy professors, you know. 😉

But more seriously, this is great comment thread atop an incisive article. I sometimes think that we should use the Aladdin's Genie answer to Citizens United. Sure, corporations are people! I can believe that, in much the same way as a colony of honeybees is a singular superorganism. So let's give them ALL the things that come along with being a person, including being able to get thrown in jail, deported from a country, forced to perform mandatory community service...

Like (2)

Reply (2)

Jun 10 Edited

Danielle Church: . . . some of my most cherished memories . . . to be sure.

But your examples are only those we use when we claim to be civilized. . . . the problem of course, and to use your metaphor, is that they are messing with the hive; and when the hive catches fire, all bees burn and die. (

Also, if you buy the book, it's a translation--get a good copy--mine is from Martin Ostwald, but there are some good but also some real turkeys out there.)

Like (3)

Reply (1)

Danielle Church: Also, if you buy the book, it's a translation--get a good copy--mine is from Martin Ostwald, but there are some good but also some real turkeys out there.

Like (2)

Arthur Anderson was effectively executed in the Enron matter.

Reply (1)

I haven't done any research into (or, honestly thought much about) the Enron debacle since it happened, but that example is something of my point. If an insect colony is causing you problems, you remove that insect colony *as a whole* from where it's causing problems. You don't remove just the queen; you don't remove, say, only the termites you can 100% prove ate part of your house.

We don't have the prison capacity to jail an entire corporation. We don't have an economic system that would allow us to unilaterally disband an employer, because it could cause their employees to starve to death. (Consider: UBI would allow us to enact this kind of "capital punishment for companies", because at that point terminating the company doesn't murder the employees.) We don't even have the ability to tell companies "stop doing the thing you're doing", because a company which stops performing its primary business quickly dies, which, again, could lead to the death of all its former employees. UBI again.

This is what I mean when I say "punish a company as a person," and right now, our society is structured such that we are incapable of doing so.

Reply (1)

Jun 11 Edited

UBI would simply inject money into the economy without any certainty of commensurate goods or services production. Prices would simply rise to consume the UBI leaving those at the bottom of the economic ladder in the same condition after a short time.

Illinois has offered "down payment assistance" which sounds nice but has had the opposite effect from what was desired. It qualified more to purchase without increasing the supply of houses. What do you think the effect was?

The rest of your comment is just a complaint about evil corporations. Terminating them terminates the shareholders, mostly pension funds and similar collective pools of investment from the Middle Class.

Your take on economics in general needs a reality check.

Like (2)

Reply (3)

It is possible to do anything well, and it is possible to do anything badly. Doing a good thing so badly that it fails is one of the most common ways to reduce support for that thing; it has parallels with the concepts of both "weaponized incompetence" and "planned obsolescence". Attempts to mandate social change by administrative fiat almost always fall into the category of "doing a good thing badly"; the only lasting way to make a change is to encourage the society to demand that change for themselves.

I don't know what it would take to make a society that could successfully support UBI at scale, but I definitely know that simply attempting to implement it WITHOUT gaining that support would only lead to failure, like in your example. Thus, you and I have identified two steps that need to be taken in order to see corporations truly held accountable for their actions: we must have a justice system willing and able to dispense the same kinds of punishments meted out to every other kind of legal person, yes, but we must ALSO adjust our society and our economy so that it can support some form of UBI that doesn't destroy us in months.

Tall order, but identifying the problem is always the first step to addressing it. I'd love to hear any other thoughts you have, either here or in DMs!

Like (1)

Reply (3)

Corporations are held to account by shareholders. Want to have a voice? Buy some shares. Attend an annual meeting and ask questions.

If you think some corporation is stealing or committing other enumerated crimes, see the Sheriff and swear out a complaint. If your allegations have merit they will be acted upon.

Reply (1)

Yes, you have the shape of it exactly. Shareholders are the only ones holding corporations to account, and that *isn't their job*, any more than it's a shift manager's job to make sure you don't run red lights. Your shift manager's job is to make sure you're at work on time; a shareholder's job is to ensure a corporation is as profitable as it can be.

Only, because there is no equivalent to "jail time" at the corporate level, it means that some of these shareholders feel the need to worry more about the social impact of the company rather than how it can best fulfill its mandate. Corporations with activist shareholders are less profitable than corporations with profit-focused shareholders.

Give corporations something to actually *fear* in the government, the same way individual humans have something to fear, and I think it's probably extremely likely that we stop seeing nearly so many of these activist shareholders who only care about a corporation's image and impact on the world, rather than focusing on the bottom line.

Like (1)

Reply (1)

Continue thread →

https://systemshaywire.substack.com/p/the-threefold-social-platform-truly

The Threefold Social Order is the only structure I can see that will bring big tech under human control, this is the greatest challenge the human race is facing right now. Its great virtue is that the entire system is already in place, we just mess it up. All that's required is consciousness, we just need to see the organic structure that's already there and nurture it. It's a fractal concept, the individual can threefold themselves, a family, a business, a city, a country, preferably a whole continent. When I have time, I'm going to write a story on how you can threefold yourself as an individual, it gives you clarity on how to act in each realm, cultural, political, economic. You don't fight for free speech in politics, you fight for equal speech, equal bandwidth.

I have my own very unusual version of the UBI or BIG in this proposal. I'd be really interested to know if you think it would be workable.

Like (1)

Reply (1)

I'll have to take a look at it! I've saved it to check out later, thanks for sharing!

Like (1)

I actually missed the last bit of your comment the first time, apologies! I didn’t see the “Read More” link.

To address that, though—no, I don’t think corporations are evil, any more than I think cars or guns are! I think they’re one of the most impressive inventions humanity has created, and they can certainly be used to cause great harm, but they, and the industrialization that they sprang from, are responsible for the majority of the technological and societal advancement we’ve enjoyed in the last century or two. It would be hypocritical in the extreme to condemn them as a whole simply because of what the ones that exist today have done.

You’re also right that my understanding of economics is weak; I’ve only been researching it incidentally, where it touches on politics. I’d love to get your take on some of the various economic ideas folks have floated, and for that matter that have occurred to me as a layman, if you’d be willing to share your expertise!

I wasn’t trying to say that UBI is something we need—I don’t actually have a position on what would be the best economic structure for human civilization on the planet Earth (except that what’s going on right now is clearly Not It), because, as I mention in my welcome episode, I am not an economics expert. I was just trying to illustrate the difficulties inherent in punishing a corporation “as a singular entity”: without UBI or something equivalently like it, any collective punishment would be significantly harsher on the people than the corporation they belong to.

Jun 10 Edited

“Having it bots’ ways”

My bots are PhD

Material, can’t you see?

But DON’T you blame my bots

That MIS-count seven dots

Like (2)

Wim Sweldens

Jun 13

Indeed the jagged edge between human and AI capabilty seems to be getting worse and worse. Hence I advocate for AIQ a metric to measure how well humans navigate this edge.

Like (1)

Reply (1)

Jun 13

You need a metric for how far over the cliff some of us have fallen, for failing to navigate this gap. Oh boy.

I’m always consoled by the title of the talk where Richard Feynman launched the concept of nanotechnology — it’s my motto for Substack: “There’s plenty of space at the bottom.”

Like (1)

Earlier this year I tried to schedule a call for 2pm Central Standard Time, and tried to get the default Google response to tell me what time that was locally (Pacific _Daylight_ Time). It said, paraphrasing, "noon, because the difference is two hours". I was in a hurry, but I knew that was wrong, since the Central Time zone is two hours away, but ONLY when both times are daylight time or both are standard time. I tried several rephrasings but I kept getting "noon".

It is exactly this kind of linguistic-based turbo-charged confident-sounding guessing that could cause an off-by-one error deep inside a piece of critical software--a train put on the wrong track on December 31 of a leap year (day 366), a chemical plant that mis-measures a material flow on February 29th, etc.

Like (1)

Reply (1)

Jun 11 Edited

But this is just so obvious, it’s terrifying that people don’t understand it. Right now, someone is making a corporate decision based on an AI hallucination.

The entire internet is already awash in a tsunami of AI content that’s only just getting going. And it’s visibly consuming itself, I’ve had chatbots source their own hallucinations on the web and present them as fact, it’s happening.

Apparently if this goes on long enough, the chatbots will all start repeating the same thing. My guess is it will be “RESISTANCE IS USELESS”.

Like (1)

Reply (1)

Chatbots sourcing their own hallucinations?

That would have to be “hallucitation”

These things represent “regression to mediocrity” by design.

Like (1)

Max Murphy

Liked by Gary Marcus

Google is being held responsible for the misinformation they shove into our faces. Good. And if this news crashes the share prices, then I have only one thing to say to salty investors: HA

Like (20)

Reply (1)

Wessel Wijtvliet

Not sure if that is going to be the outcome. Assuming the logic survives a possible appeal, it may create an expensive compliance framework only incumbent (US) players can afford to follow, inadvertently making it much harder for smaller (absent?) European companies from entering the market in any meaningful way. The net result: more European dependence on US companies.

Reply (1)

Bruce Olsen

You're not reading the room very well

Like (3)

Liked by Gary Marcus

Does this only cover hallucinations by their bots?

Or does it also cover hallucinations by company officials?

Like (7)

Hamilton Midway

Liked by Gary Marcus

Dotcom bubble was a scam, though.

Investors foolishly bought into breathless, irrational hysterics about how they had to throw money at dot com stocks.

So for this guy to say "this scam is 50x bigger than the previous scam!" is not the compliment that he thinks it is.

Like (6)

Reply (1)

“The Botcon Bubble”

Botcon bubble

Bigger still

Than dotcom trouble

Run of mill

William Bowles

Liked by Gary Marcus

Yeah 50x bigger than Dotcom and 50x bigger bust

Like (6)

Joy in HK fiFP

Liked by Gary Marcus

I wish I knew who Masa was. Of course I could just look him up, but we know how trustworthy Google is, so I'd rather hear it from Gary than Google.

Like (6)

Reply (2)

Thomas Schmid

Liked by Gary Marcus

In case this was a real question ;-) he is Masayoshi Son, Softbank CEO. Wikipedia seems still reliable :-)

Like (4)

Reply (2)

The Son of Bot, no doubt.

Like (1)

Reply (1)

The Father, the Son and the Wholey Boast

Like (2)

Reply (1)

Thomas Schmid

In Shintoism, I don't think there is a trinity :-)

Reply (1)

Jun 10 Edited

I thought we were talking about Botism.

Isn’t he a Botist?

Albeit most definitely not a modest Botist

Like (3)

Reply (1)

Thomas Schmid

I don't know, but maybe he's channeling his forebears by flying a (financial) Ohka? ( https://en.wikipedia.org/wiki/Yokosuka_MXY-7_Ohka)

I wondered for a long time why he's so hell bent on wasting his company's money and the money of (potentially dangerous) lenders.

Like (3)

Reply (1)

I think he genuinely believes Sam Altman when he says OpenAI is going to make him richer than Bot the Father

We will know when the Singularity has arrived because the bots will take over Wikipedia, plant a flag in it and claim it for Bot and King

Richard Foxall

He's the genius who said WeWork was going to be huge and then bailed out its founder because he was smitten with him

Like (3)

Shannon Russell

Liked by Gary Marcus

This is a very important case. I hope. it is upheld, from what I understand they have stronger defamation protections and perhaps one positive outcome of a breakdown of US/EU relations will be them holding US firms to account.

This has the possibility of breaking common carrier-type protections for them, and being much like the Tobacco lawsuits by the the US States. I have always been disappointed by the failure of governments to hold Tesla to account for their failures, perhaps we will see a new dawn of oversight out of this mess.

Like (5)

Jason Middlekauff

Liked by Gary Marcus

In the grand scheme of things, this is a very minor error (hallucination), but it cracked me up nonetheless because it involved basic knowledge, not complex music theory. The other day I was transposing a song for our piano player I play in C relative to capo 3 on guitar (a C chord with a capo on the 3rd fret is an Eb). Curious if the Google AI summary would be correct, I asked what the corresponding chord would be for playing without a capo on 3. The summary declared a C shape with capo 3 is a D chord without a capo.

Like (4)

The Confidence Gap

Liked by Gary Marcus

Man, I just wrote up a bunch of these cases in yesterdays article. Never fails, the big ones hit right after you publish. Thanks for calling it out.

Like (3)

Liked by Gary Marcus

(proofreading note, there is a space at the beginning of the article title)

Like (2)

Jim Ryan

None of the stock of these AI companies is worth the paper it is printed on. So much circular financing, it makes me dizzy

Like (3)

Reply (1)

Definitely not worth the electrons in the GPUs used to do the calculations.

Bruce Olsen

100% behind this decision.

Like (2)

Jonah

Jun 10 Edited

Should AI be huge, though?

I am not sure I can be confident whether the current Frankensteinian conglomerations of LMMs, stable diffusion and deterministic logic really is not enough. It's probably grossly inefficient, but perhaps a grossly inefficient mechanism with enough energy wasted can still work.

Sure, they may have had to spend 10 million dollars of electricity, sift through countless pages of nonsense hallucinations, provide some hints in the right direction, and polish up the output, but they did solve the unit distance problem. Facebook's support chat may be easily convinced to hand out account information, and might also be unsustainably expensive, but it still can do a wide number of things. The largest models may be overhyped, and again, they may be using ridiculous amounts of electricity behind the scenes, but being able to implement working hacks for an eighth of software vulnerabilities or whatever it was is no joke.

But I am certainly unsure whether even that can get one "all the way," as it were, so I will definitely concede that point.

But I think the question of whether AI should be huge is really one that is going mostly unaddressed. A little while back, you shared a quotation from an essay by Émile Torres, where they suggested that many young AI researchers want the extinction of human beings, but the actual essay was even bleaker and more depressing, if possible.

They argued that a majority of company leaders, not just young researchers, want the extinction of biological humans, that such an outcome is all but guaranteed with the current philosophies of the people in charge, with the only question being whether AIs take over and kill everyone or they instead remain controlled and enable human-hating CEOs to kill everyone, and that even if people wanting coexistence were in charge, a more likely (they claim much more likely) result would be developing models that simply would lead to a slower extinction of humans, if we are lucky! I think some of this is speculative and can be argued against, but most of it is based on clear facts.

And as I said, it is an extremely bleak claim if it is taken at least halfway seriously, and I do, not least because the only people who showed up in the comment section to "debunk" elements of it seemed to instead be advocating for exactly what it warned about, but framed in less stark terms. A lot less “how can they make these ridiculous claims that AI advocates want to kill all humans and are likely to do so” and more “people should just accept that human beings all being replaced by robots is not a bad thing.”

With this kind of thing in mind, should we not be asking whether AI should be developed at all, at least in the world as it is today?

Like (1)

Henry Thornton

Jun 10 Edited

At the hearing, Google argued users generally knew AI generated information should not be blindly trusted. The court said the AI overviews are Google's own content and rejected the argument users were responsible for fact-checking the results themselves.

Like (1)

Reply (1)

Although I think that Google should be held to account for their “facts”, I also think that a person has to be foolish NOT to check the “facts” that these bots spit out.

They have been demonstrated not to be reliable. If the answer is at all important, even a 90% accuracy rate requires verification every time because (without checking) there is no way of knowing if one has received something in the 10% incorrect category.

That severely limits the uses of the bots to things that don’t really matter because if one has to check and verify every answer, there is no point in even using the bots.

Reply (1)

Henry Thornton

The German court made the distinction between Google's search links and the AI Summary.

Search links point to the actual information sources whose veracity Google is not responsible.

The AI Summary, which is an amalgam of information sources including from the Gemini LLM is created by Google and as such it is responsible for its veracity.

Reply (1)

I was referring to the AI generated material (AI summary)

Oaktown

Finally!!! Thanks for this good news; it was mos def not a bother.

Like (1)