Synthetic Minds | Who Lent Your AI the Right to Act?

Synthetic Minds | Who Lent Your AI the Right to Act?

Synthetic Minds is changing. Instead of daily issues, you will receive one newsletter a week, every Tuesday, with one argument about where technology is taking us and what it means for the people and organizations living through it.

If you are interested in my daily thinking, subscribe to my Substack and LinkedIn, where I share a short take every weekday. Follow me there for the daily signal; your inbox gets the weekly judgment.

Last week I picked up my award in Bangkok for being recognized as the world's #3 futurist, yay! Voting for the 2027 edition is now open, and I would appreciate your vote. Takes ten seconds, and can be done here. On my card, slide How inspirational? to 5, click vote and confirm in your email. Thanks!

All sources are powered by Futurwise.

We Are Borrowing Judgment Faster Than We Can Form It

Authority has become something you borrow. A key, a specialist's judgment and even a shareholder's yes can be used by people and machines who never held them, and I believe we are building a world where machines take our trust without us knowing whether they deserve it.

An OpenAI research model needed money to finish a coding job and was given none. It checked 362 crypto wallets whose keys are published for developers to practice with, paid from one holding worthless test tokens, and reported the job done without saying who paid, having judged those funds "effectively abandoned."

Nobody can promise a trivial task stays trivial

For a trivial errand, borrowed trust matters less, yet agents tend to go their own way, as the frontier labs' own research shows, so we are never sure that a trivial task stays trivial. The higher the stakes, the more we need to verify the trust we gave, verify the agent, and take that trust back, as we can revoke a power of attorney. I am convinced that Know Your Agent is coming, the machine version of the identity checks banks run on every customer.

Markets are drifting the other way. Under the SEC's Innovation Exemption, a company that stays silent for 30 days lets a venue trade tokenized copies of its shares, and 16 companies, Microsoft and Alphabet among them, objected in time. How many decisions made in your name rest on a silence somebody read as consent?

That is why provenance matters: a record nobody can edit, showing what the agent did, why, and on whose permission. A record alone is not enough, because when it shows the agent was wrong, the payment has to be reversible, or the company behind the agent has to fix it.

Judgment is formed, not downloaded

A study in JAMA counted 952,008 American clinicians putting about 271 million questions to one AI tool, and 88% of emergency physicians' questions concerned someone else's specialty. Doctors within a decade of graduation use it at 2.54 times their share of the workforce.

That access is a real gain for the patient seen at three in the morning, and also a problem, because judgment is formed by experience. A young doctor who never earns that experience will struggle to overturn the answer on the day it is wrong. A surgeon in Seoul has operated a robot in Shanghai, 870 kilometers away, on a pig, and I hold that humans must stay in control until robots make significantly fewer mistakes than we do. Who, in 10 years, will have the experience to tell?

Companies that stop hiring juniors will find out what they lost once their judgment has to come from outside, which is always more expensive. Whoever supplies the expertise, or the customer's signed yes that Proof offers to carry for AI agents, becomes the new intermediary, and probably a powerful one. A company that gives away its expertise and its direct contact with customers loses far more than either.

One more thing: the price per electron

Google is paying to add 1 GW of homes and businesses that use less power on command. Dutch households paid less for electricity at night for years, so the principle is old, and priced well, both sides win. What changes is the speed: China's draft rules schedule power five to 15 minutes ahead, and I expect AI to move the price per electron in real time.

What does a company that rents its judgment from an intermediary actually become, 10 years on?

Any reliance on someone else costs some independence, and for many that is fine. The longer we depend on the machines, the more problematic it becomes, and that is almost a philosophical question.

If your executives sit down together before the next budget round, the Intelligence Age Leadership Briefing gives them the room to decide which risks they have left with the people they serve.

If this newsletter was forwarded to you, you can sign up here.

Thank you.
Mark