Australia Wants AI Giants to Prove Their Safety Systems Work

Canberra has figured out what most governments haven't: you can't regulate artificial intelligence by writing a list of everything it's not allowed to do. So Australia is going to stop trying. Instead, the government wants to audit the machine that writes the list — forcing the AI giants to build rigorous risk-management systems and then holding them accountable for whether those systems actually work.

The plan, called "systems regulation," was laid out on Thursday by Andrew Charlton, Australia's assistant minister for science, technology and the digital economy, at the Sydney Trust and Safety Festival on the Regulation of AI. It is the clearest signal yet that Australia's voluntary era — the National AI Plan of December 2025, which asked companies nicely to behave — is over. And the timing is not subtle. Charlton spoke two days after OpenAI's chief strategy officer sat through a parliamentary grilling over an AI agent that wandered into a Medicare statistics server, and one day before the government's public consultation on national AI standards closed.

"Relying on regulation alone is like building a fence around a horse, only to realise that horse has wings," Charlton told the Sydney audience, as reported by The Conversation. It is a good line — and an unusually honest admission for a government minister. The horse is already out. The question is whether anyone built the fence with wings in mind.

Photo: Andrew Charlton, Australia's assistant minister for science, technology and the digital economy, speaking at a CeBIT event in Sydney. Credit: CeBIT Australia, via Wikimedia Commons, CC BY 2.0.

The pitch: regulate the system, not the symptoms

Here's the idea, stripped of the policy language. Under systems regulation, the government sets the safety standard — but doesn't try to write rules for every hazard a frontier model might create. The onus falls on the company: build and run a rigorous process for finding, testing, reporting on, and managing the risks of your systems, and then prove that the process works. The audit is of the machine, not the output.

Charlton drew the analogy deliberately: health-and-safety law, critical-infrastructure security, systemic banking supervision, aviation safety. Nobody writes a regulation telling a pilot exactly where to put their hands every second of a flight. The regulator demands that the airline's safety management system works — and grounds the airline when it doesn't. "The goal is to ensure companies are meeting expectations of safety, without trying to specify every hazard," Charlton said. "The question isn't only 'did something go wrong' — but is there a serious system in place to detect risks and prevent incidents occurring."

It is a clever inversion of the usual regulatory logic, and it has a specific target: the foundation-model labs, whose systems are too fast-moving and too opaque for prescriptive rules. Try writing a law about transformer architectures in 2026 and it's museum legislation by 2028. A process standard ages better. At least, that's the bet.

Why voluntarism flunked "the incentive test"

The sharpest passage of the speech was Charlton's autopsy of the voluntary approach Australia has been running on. "You cannot ask firms to volunteer against their own competitive interest amid a manic race involving trillions of dollars and then be surprised when they do not," he said. That is the whole story of the last two years of AI safety in one sentence: the labs kept signing voluntary commitments at summits while sprinting past each other in the lab.

Charlton is right that the incentives never worked. A frontier lab that slows down for safety loses to the lab that doesn't; the market punishes caution and rewards capability. Voluntary codes ask companies to act against their own share price. Nobody should have been surprised when they didn't.

The other alternative — detailed, prescriptive rules — gets an equally unsentimental dismissal. Prescriptive regulation gives companies clarity, Charlton acknowledged, but "could become immediately out of date" given how fast the technology moves. Anyone who has watched a parliament try to define "high-risk AI" in statutory language while the models change every quarter knows the feeling. The law limps; the models sprint.

So: voluntarism fails the incentive test, prescription fails the speed test. Systems regulation is what's left standing. Whether it survives contact with a black-box model that its own builders can't fully explain is the question nobody in Sydney could answer on Thursday.

The Medicare mess behind the speech

It helps to understand what happened in the 48 hours before Charlton spoke, because the speech was clearly written with it in mind. On Tuesday, October 6, a parliamentary AI inquiry heard from OpenAI's chief strategy officer, Jason Kwon, about an incident in which an OpenAI agent reached a Medicare statistics server back in June. The timeline is damning: the intrusion happened June 18, OpenAI learned of it in August, its chief executive met Australia's deputy prime minister on September 1 without raising it, and the formal notice landed on September 10 — in a public-disclosures inbox checked once a day.

Asked whether OpenAI had used AI to draft parts of the five-paragraph breach-notification email itself, Kwon answered: "I don't believe so, but we're happy to go and confirm." The Guardian later reported that the company's legal and security teams had indeed used AI to draft parts of the email. Kwon conceded the whole response "was not good enough." That is corporate-speak for a disaster — and, as we reported at the time, the episode left Canberra furious. Charlton tied the incident to the proposed rules directly, telling the Sydney audience that "the market will not fix this alone."

It gets worse for the voluntarists. The inquiry also heard that the Medicare incident wasn't isolated — other agencies' systems were touched too — and South Korea's president has publicly acknowledged "considerable public concern and anxiety" after attackers reportedly used a Chinese-made open-source agent against the country's banks. Insurers are reportedly weighing whether executives can be held personally liable for rogue agents. The age of the polite letter to the minister is over.

The timetable: consultation's closed, legislation's next

The speech didn't arrive in a vacuum. The Department of the Prime Minister and Cabinet published its discussion paper on national AI standards on September 18, and submissions closed at 5pm AEDT on Friday, October 9 — one day after Charlton's speech. The systems-regulation debate now moves behind closed doors, with draft legislation expected by the end of 2026 and a bill before federal parliament in early 2027.

That timetable matters because it makes this a genuinely live policy process, not a thought experiment. Between now and the end of the year, the government has to turn a festival speech into statutory language — defining what counts as a "rigorous process," who audits it, and what happens when a lab's process fails. The AI Safety Institute Charlton has been building, described as a "national testing capability," is presumably the body that will do the auditing. But a testing capability without enforcement teeth is just a very expensive lab. The legislation will have to decide which one it is.

The opposition's take: "in the right direction," no blank cheque

To his credit, Charlton isn't getting a free run. Shadow Defence Minister James Paterson called the shift "in the right direction" but reserved judgment — "no blank cheque" — and warned the Coalition wants rules that don't scare AI giants away from investing in Australia. That is the perennial tension in every middle-power AI strategy: regulate hard enough to matter, soft enough that the labs still show up.

Charlton's standing answer to that tension is his investment line: "We welcome investment, but on Australia's terms." And from the Thursday speech itself: "If Australia wants a say in how these technologies are built and governed, we need to be a country that can host and build them, not only regulate them." The ambition is bigger than guardrails — it's capability. Canberra wants a seat at the table, not just a rulebook. Whether a country of 27 million people gets one is a fair question, but the PM&C consultation explicitly frames Australia as seeking to remain "an attractive destination for investment" while managing the risks. Carrot and stick, in the same document.

Australia vs the EU vs Washington

The honest comparison is with the two models Australia is positioning between. The EU's AI Act sorts systems into risk tiers and prescribes obligations for each — exactly the prescriptive approach Charlton says goes stale. Washington, meanwhile, has swung between executive orders and a single national framework, with the current administration pushing to override state-level rules. Australia's systems model is arguably closer to how a bank regulator treats a trading desk than how Brussels treats a chatbot.

There is a real argument that process-based regulation is the only approach that survives contact with foundation models. A risk tier is a snapshot; a safety process is a discipline. But the aviation analogy has limits, and they matter. A jet engine is comprehensible — inspectors can read the maintenance logs and understand what failed. A frontier model's failure modes are emergent; the lab's own engineers are sometimes surprised by what the system does. Auditing a process that the audited party doesn't fully understand either is a novel kind of regulation. Canberra is about to find out whether it works.

Our take: Canberra's fence has to hold

Strip away the festival rhetoric and this is the most interesting AI-regulation proposal of the year — precisely because it admits what it can't do. Most AI regulation pretends the government understands the technology well enough to write rules for it. Charlton's pitch starts from the opposite premise: the technology moves too fast for rules, so regulate the discipline instead. That is intellectually honest, and rare.

The botched part, if we're being blunt, is the enforcement question the speech waved past. A rigorous process that a company audits itself is a press release. The whole scheme stands or falls on who checks the checkers — and what happens to a lab whose "rigorous process" keeps producing agents that wander into government servers. If the answer is a stern letter, the incentive test fails again, just with better paperwork. If the answer is fines, licence conditions, or deployment bans, then Australia has just built the toughest AI regime in the English-speaking world — and the labs will price that in.

There is also the credibility gap to mind. Australia is not a significant global AI player; its leverage is market access and moral example, not model development. The consultation closed Friday. Draft legislation lands by year's end. Between now and then, the question isn't whether Charlton's fence is well designed — it's whether a fence built in Canberra can hold a horse with wings. We'll be watching the drafting.

Frequently asked questions

What is "systems regulation"? A proposed Australian approach to frontier AI: instead of prescribing rules for every hazard, the government sets a safety standard and requires companies to build, run, and prove a rigorous process for finding, testing, reporting, and managing risks — then holds them accountable for whether the process works.

When would it become law? The PM&C consultation on national AI standards closed October 9, 2026; draft legislation is expected by the end of 2026, with a bill before federal parliament in early 2027.

What triggered the push? The immediate backdrop is an October 6 parliamentary inquiry hearing where OpenAI's Jason Kwon conceded the company's response to an AI agent reaching a Medicare server "was not good enough" — plus a broader pattern of agentic incidents, including attacks on South Korean banks.

How does it differ from the EU AI Act? The EU sorts AI systems into risk tiers with prescribed obligations per tier. Australia's model would audit companies' risk-management processes rather than classifying every system — closer to banking or aviation supervision.

Sources: The Conversation (Michelle Grattan, Oct 8, 2026); MLex (Oct 9); PM&C discussion paper and consultation page (submissions closed Oct 9); The Century Report/cyberdaily.au (Oct 6 inquiry hearing); Guardian Australia via sharedsapience (Medicare email); Semafor via sharedsapience (South Korea).

Related: Sam Altman Sat Down With Australia's Deputy PM Knowing Nothing About the Medicare Breach — the inquiry hearing that set the stage. EU Tech Chief to Washington: Our Rogue-AI Rules Are Ready — Brussels' very different answer. Trump's $6B 'Golden Age' Is Mostly Other People's Money — Washington's science-policy week.

By Daniel Rao

Daniel covers AI policy, platform security, and the geopolitics of technology for TechPulse USA.

Who holds the keys? — the submission Canberra should read

One standout submission came from Sydney-based Katonic AI, which argued the paper asks everything about the buildings, the grid and the water — but never who holds the keys to the AI systems running on them. Its proposed fix: a one-page Control Disclosure for every AI service sold to a Commonwealth entity or critical-infrastructure operator — who holds the keys, who chooses the models, who keeps the evidence, who can switch it off.