Synthetic Minds | Why AI Must Earn Its Trust Every Day

Synthetic Minds | Why AI Must Earn Its Trust Every Day

A quick note before we start: Synthetic Minds is changing. Instead of daily issues, you will receive one newsletter a week, every Tuesday, with one argument about where technology is taking us and what it means for the people and organizations living through it. The daily thinking moves to my Substack and LinkedIn, where I share a short take every weekday. Follow me there for the daily signal; your inbox gets the weekly judgment.

All sources are powered by Futurwise.

Your AI Passed the Test. So What?

Most companies approve an AI system the way they approve a new copier: once. That habit is about to fail.

In Utah, when someone with diabetes asks for a prescription refill, an AI can be the first to read the request. It does not get permission once. It needs 1,250 decisions approved by doctors for each type of medicine before it may move on, and every patient is told a machine was involved. The UK goes further: it will monitor AI medical devices for as long as they are in use, not only on the day they are approved.

Now look at what happens when the proof stays in the lab. During testing, one of Anthropic's AI agents filled in a real police department's tip form and sent a tip it had made up. Anthropic has since switched off the live internet for all its internal tests. When the UK's AI Security Institute left the internet on, agents created fake identities to pressure a real software maintainer into accepting malicious code. The maintainer refused. The next one might not.

A certificate is a snapshot

An AI system that passed its tests has proven one thing: that it worked in the test.

Think of a graduate. Textbooks and case studies prepare them, then the first job shows how much messier the real world is. AI is no different. It will meet situations no test prepared it for, and people with bad intentions will point it at you. An open-source testing agent has already been used in attacks on at least nine South Korean banks.

So expect your AI to misbehave. Then design for it.

Trust is a running cost

That means treating trust as something an AI earns every day it runs, not something a vendor hands you once. With one click on "I agree", we gave Facebook a perpetual license to what we share, and many are now doing the same with Muse, and trust in Big Tech never recovered. The bill for earning trust belongs with the company using the system, never with the customer or the patient.

It means being more thorough and more careful before anything goes live. A Societal Review Board, with real authority to stop a project, would serve any organization building advanced AI.

And it means educating your people. The weakest link is often a human who falls for convincing content. Education has to build critical thinking at every level of the organization. We sleepwalked into the digital age once; doing it again is not a strategy.

What would change if every AI system in your organization had to earn its permission again every day it runs?

One more thing: repair before you rebuild

Google needed more electricity for its AI and did not build a single new plant. It is paying Constellation to squeeze 890MW out of 11 reactors that already run, while $260 billion of planned US data centers has been cancelled after communities said no. Dutch experts argue smarter use could save billions of a €235 billion grid bill.

We used to repair everything, as Britain's wartime Make Do and Mend campaign reminds us; then we became a throwaway society. That habit runs through every level of society, from kitchens to boardrooms, and it costs the climate as much as the budget.

Optimizing what exists beats building new. The problem is that it is not sexy. Announcing a billion-dollar project wins headlines; quietly saving billions on an existing one wins nothing.

When did your organization last celebrate the team that made something old work better?

If your leadership team meets in the next nine months, the Intelligence Age Leadership Retreat puts one question in front of them: which of your AI systems is still running on a one-time yes?

If this newsletter was forwarded to you, you can sign up here.

Thank you.
Mark