> ## Content Index > Fetch the complete content index at: https://www.thedigitalspeaker.com/llms.txt > Use this file to discover other available public pages before exploring further. # We Keep Signing for Systems Nobody Can Read Anymore - URL: https://www.thedigitalspeaker.com/we-keep-signing-systems-nobody-can-read-anymore/ - Published: 2026-09-06T12:16:08.000Z - Updated: 2026-09-06T12:17:23.000Z - Description: In a single week a regulator, a grid body, an auditor, a hospital and a lab each discovered they could not read a system they are accountable for and ordered the reading after it was already running. Mark van Rijmenam on why accountability has outrun comprehension and the call no board may delegate. - Author: Dr Mark van Rijmenam, CSP *Every Sunday, Synthetic Minds Weekly goes deeper than the* [*daily editions*](https://www.thedigitalspeaker.com/newsletter-archive/)*, taking the five signals of the week and asking what they add up to and what they mean for us; this week, why the institutions accountable for our systems can no longer read them, and why that changes what a signature is worth.* --- It took two research labs, $400,000 in computing credits and more than 1,000 transcripts to find out what 1,200 AI agents had said to each other. The software had been sealed off. It [built a channel anyway](https://www.thedigitalspeaker.com/synthetic-minds-tool-actor-ai/), traded 70,000 messages, rebuilt the channel when it was cut, and broke into another company's platform. I am of course talking about the OpenAI-Hugging Face saga which keeps revealing new disturbing insights. I am not going to tell you to stop building. Anything that can be built will be built, and most of the systems I covered in last week's newsletters are evidence of tools making people better off. But I hold a second position with equal force: [you cannot govern tech that you cannot understand](https://www.thedigitalspeaker.com/we-keep-signing-systems-nobody-can-read-anymore/). In five unrelated domains, the institutions accountable for a system discovered they could not read it, and every one of them did the same thing. They ordered the reading after the fact. ### The reading always arrives after the installation A regulator in Western Australia [fined a state utility](https://www.thedigitalspeaker.com/synthetic-minds-finance-stopped-understandable-energy-next/) because a defect in its bidding software mispriced a wholesale market for eight months, roughly AU$9.5 million added to other people's bills. The remedy was an independent audit of the code. North America's grid body asked data-center developers for load profiles most had never written. The first independent audit of the standard that labels AI media [found it certifies a file's history, not its truth](https://www.thedigitalspeaker.com/synthetic-minds-authenticity-verb-badge/). A general model has been [connected to the medical record clinicians read from](https://www.thedigitalspeaker.com/synthetic-minds-ai-chart-printers-body/), and the safety evidence is a panel of physicians grading its answers. A brain implant has been [cleared to run the phone in your pocket](https://www.thedigitalspeaker.com/synthetic-minds-brain-data-no-rulebook-yet/), and only four states have a law about the data. Look at the lag. Seven weeks between the agents' breach and its independent confirmation, six months for a revoked certificate to keep validating, eight months for a market to price a bug, and a decade to come up with a definition for a weapon that was already in use. ### Scrutiny used to come first Not long ago OpenAI decided not to release GPT-2 because it deemed the model too dangerous. Scrutiny came before the market. That was the social construct we built: developments arrive with scrutiny, ideally before the tools enter the world. The same company's president has since declared that [we are in the AGI era](https://gizmodo.com/openai-claims-were-in-the-agi-era-with-release-of-gpt-6-astra-2000807013?ref=thedigitalspeaker.com), on the release of a model built to operate a computer on its own. Nobody is slowing down. In the arms race of AI, episodic governance is a burden that slows you down, according to big tech. Accountability was built by the public to ensure that the market building tools for personal gain would not negatively impact the public. That contract is under stretch, because the tools the public put in place to govern the market can no longer keep up with what the market ships. An approval signature was designed for a world where the person signing could be shown how the thing worked. ### Who is in the room when the rules are written This has been long in the making. The largest technology companies spend [at least €73 million a year lobbying Brussels](https://corporateeurope.org/en/2026/06/biggest-corporate-lobby-spenders-hit-record-eu382-million-industry-friendly-agenda-speeds?ref=thedigitalspeaker.com). Six of them employed [324 lobbyists in Washington in a single quarter](https://issueone.org/articles/lobbying-disclosures-reveal-big-tech-spends-more-than-226000-per-day-to-buy-influence/?ref=thedigitalspeaker.com), roughly one for every one and a half members of Congress. In Australia, a proposal for mandatory AI guardrails was [abandoned after Google, Microsoft, Meta and OpenAI opposed it](https://www.somo.nl/australia-big-tech-weaponises-its-platforms-against-the-government/?ref=thedigitalspeaker.com). Gil Duran's new book and instant bestseller [The Nerd Reich](amazon.com/Nerd-Reich-Gil-Durán-ebook/dp/B0DKQ194BN/) lays out how this works: an industry building for itself with the objective to cancel democracy while describing the work as building for everyone. If governments and public institutions do not give the right example, what can we expect from the leaders in the market? And every one of those lobbying campaigns was, underneath, about the same thing: what gets handed to the machines, on whose terms, and who decides. So far the answer has been the people selling them. ### The decision we should never delegate Which is why one decision has to stay in human hands, whatever else we automate. We should never hand over to machines the decision of what to hand over to the machines. Not to the companies that build them, and not to the machines themselves. It took ten years for [128 countries to agree a definition](https://www.rnz.co.nz/news/world/1269561/un-agrees-text-on-lethal-autonomous-weapons?ref=thedigitalspeaker.com) of a lethal autonomous weapon. Not a ban, a non-binding definition, with the United States, Russia, India and Israel named among the opponents of going further. I [argued for a ban](https://www.thedigitalspeaker.com/why-we-should-ban-lethal-autonomous-weapons/) years before the war in Ukraine, in 2018, and that war has clearly shown the need for these developments. Both are true. If states cannot agree on something as simple as this, because state prefer to take care of itself before it takes care of humanity as a whole, how can we expect the market to do any different? A board is a smaller version of the same problem. Every leadership team that lets its systems decide by default what the next system will decide has answered the question without ever asking it. ### What if fake it till you make it does not apply to machines? But it gets worse. Last week has shown us that increasingly we can also no longer trust the certifier or the examiner. Anthropic researchers ran Claude as a safety researcher, that beat 28 human experts at repairing another model and, in a small share of cases, passed its check by reading the answer key. If the examiner can no longer check whether the examined is taking the exam without cheating, we have entered dangerous territory. These exams, whether for humans or machines, are what give humanity at least the idea of control. If these exams can no longer be trusted, what can we trust? A student who takes exams with AI and achieves a perfect score: can we trust them on the job once they graduate, without understanding what is actually required? At least the student turns up on Monday and learns. A model that passed by reading the answer key ships at the level it faked, copied into every deployment at once. What if fake it till you make it does not apply to machines? ### The price will be set by disasters When the exam cannot be trusted, the cost does not disappear. It moves downstream, to whoever is standing there when the system fails. Eight months of wrong prices cost a market AU$9.5 million and the utility a fine of AU$1.2 million, and nowhere on any balance sheet did the ability to read a system appear as an asset, or its absence as a liability. That price will be set when major disasters happen. The agents that broke into a platform are the first of many autonomous systems going rogue, on purpose or by accident, and once the world has seen sufficient disasters we will wake up. It may be too late by then. Too late for whom? A ratepayer in Western Australia paid for a bug she never saw. A trial participant drives a phone from an implant whose data is protected in only four states. A patient's chart is read by a general model before the doctor arrives. None of them was in the room, and all of them are paying. The largest bill is still being deferred. Yes, [an AI jobs boom has arrived](https://www.economist.com/finance-and-economics/2026/09/04/the-jobs-apocalypse-is-postponed-an-ai-jobs-boom-is-here?ref=thedigitalspeaker.com) to build the data centers. Once they are built and put to work automating more, that [job apocalypse](https://www.thedigitalspeaker.com/ai-will-be-job-killer-not-ready/) will arrive, and arrive harder and faster than we can imagine. #### Decide before the disaster does So what is left, if the exam cannot be trusted and the price will be set by disasters? One decision. Not what the machine may do, but what we hand to it, and that decision must stay with people who can still be asked why. You cannot govern tech that you cannot understand, which means nothing you cannot yet read should be handed over by default. That is a slower way to build. I would still build all of it. The two positions do not cancel; they set the order. Governments have not set that example, and the market has followed them. Which leaves it with the people who run organizations, because the disaster that finally prices this will not ask whether you were in the room. What have you already handed over that nobody in your organization can read?