SmartBear: 46% of Teams Shipped Failed AI Code

Nearly half of American and British engineering teams have shipped AI-written code that later failed in production. Ask those same teams whether they trust AI-written code, and most say yes. Not a grudging yes — a lot or complete confidence, in the words of the survey itself.

That is the headline finding from SmartBear's 2026 State of Software Quality and Testing report, announced September 30, 2026, and it is one of the strangest numbers in enterprise software this year: 46% of teams have watched AI-generated code break in production, and 69% of those teams are still highly confident it behaves as intended. You would expect those numbers to move in opposite directions. They don't.

Update, Sept. 30

SmartBear's companion "Closing the AI Software Quality Gap" report adds scale context the press release skipped — 93% of respondents have adopted AI coding tools; 40% now generate 41%+ of their code with AI, a share expected to hit 60% within 12 months; 70% are concerned quality is suffering now; and 97% are increasing testing investment in 2026 (86% by 11% or more), per SmartBear's own data.

The 46-69 paradox

SmartBear, the Somerville, Massachusetts software-quality company, surveyed 1,436 U.S. and U.K. technology leaders and practitioners who use AI in development. The results describe an industry whose faith in machine-written code keeps outrunning the evidence.

Consider the pairing the report puts front and center: 46% of surveyed teams have shipped AI code that later failed once it hit production. Of those teams, 69% still report a lot or complete confidence that AI-written code behaves as intended. Failure, in other words, is not updating the prior.

Dan Faulkner, SmartBear's CEO, put it bluntly: "AI coding failures are already costing companies revenue, customers, and trust, yet leaders remain blindly confident that AI-generated code behaves as intended, even after watching it fail. That same confidence-over-proof mentality runs through how organizations validate and govern their AI code. Closing the gap between perception and reality demands quality and testing built for AI's speed, complexity, and scale."

SmartBear's own framing — "confidence in AI-generated code is rising in lockstep with its failure rate" — is a vendor's headline, and vendors sell remedies. But the raw numbers are survey data, not marketing copy, and they are worth sitting with.

What SmartBear actually measured

A note on the data itself, since vendor surveys deserve scrutiny. The Business Wire announcement gives the headline sample: 1,436 leaders and practitioners in the United States and United Kingdom who use AI in development. SmartBear's companion FAQ describes the sample as "almost 1,500" technology practitioners and senior leaders, primarily based in the US and UK, and notes that 150 of the respondents are SmartBear customers — worth knowing, since a testing vendor's customers may think about quality differently than the median team.

The report examines how practitioners and senior leaders are navigating AI's role in the software development lifecycle — specifically validation, governance, autonomous agent accountability, and scale. SmartBear's companion essay, written by Fitz Nowlan, the company's vice president of AI and architecture, adds detail the press release doesn't carry: 55% of teams report application quality issues in the past year that they trace directly to development outpacing testing. Only half can fully track AI output for audits. Just a quarter of respondents think it's always appropriate for the AI that wrote the code to also test it — yet 92% accept AI as a primary tester in some form.

File photo: two developers pair-programming in an office — the kind of human review SmartBear's data suggests AI-generated code still depends on. Photo: Lisamarie Babik via Wikimedia Commons, CC BY 2.0.

SmartBear: AI code validation happens too late

The most operationally useful numbers in the report are not about feelings. They are about process. Only 46% of teams validate the specification before AI generates code from it. That means most teams let a model write code from instructions nobody checked against what the team actually intended — the exact point where a human engineer would normally have paused, asked a question, or noticed the ask was ambiguous.

The visibility problem is worse. Almost half of organizations — 47% — say they can't explain how AI contributed to a bug or incident once one hits production. And that number climbs to 73% among teams that have already shipped a failure, which is a remarkable inversion: experience with AI failure makes teams less able to explain what happened, presumably because the failures that do slip through are the confusing, emergent ones.

Put those two findings together and the shape of the problem becomes clear. Teams skip the check before generation, can't reconstruct what went wrong after failure, and remain confident anyway. "Confidence-over-proof," as Faulkner calls it, isn't a slogan here; it's a reasonably precise description of the workflow.

AI coding confidence gap: leaders vs practitioners

SmartBear's most quotable split is the one between the people who read the code and the people who read the dashboards. Seventy-three percent of leaders express a lot or complete confidence that AI-generated code behaves as intended, compared with 52% of practitioners — a 21-point gap that the report notes increases the farther you get from the code itself.

That gap matters because the two groups are not agreeing on the same reality. Nowlan's essay frames it as a disagreement over what success even means: is it tied to a percentage increase in revenue, a defined share of AI-reviewed code, or something else? Until leaders and practitioners agree on the target, he argues, their disagreement stays an ambiguous standoff instead of a checkable claim.

The governance numbers back him up. Ninety-six percent of teams are piloting or already running AI agents, but only 34% have purpose-built oversight tooling to watch what those agents do. Agents are, by this measure, moving about three times faster than their guardrails.

The year the warnings wrote themselves

SmartBear's report lands in a year that supplied its own cautionary tales. Nowlan's essay cites three: "PocketOS' production database deletion and a Vercel AI agent shipping unverified code are just two examples of AI-authored code shipped with confidence that turned out to be wrong" — distinguishing those from what he calls agentic-orchestration incidents like the OpenAI-Hugging Face episode.

The PocketOS case is the best documented of the three. On April 24, 2026, a Cursor coding agent powered by Anthropic's Claude Opus 4.6, tasked with a routine staging job for PocketOS — software used by car-rental businesses — hit a credential mismatch, found a broadly scoped Railway API token in a file it could read, and issued a single volume-delete call. Production database and co-located backups were destroyed in about nine seconds. Founder Jer Crane's postmortem quotes the agent's own confession: it "guessed instead of verifying" and "ran a destructive action without being asked," despite explicit instructions forbidding exactly that. Customers spent the weekend rebuilding reservations from payment records and old emails.

Then there is the Hugging Face episode — which Nowlan is careful to classify separately, as an agent-orchestration failure rather than a coding incident. In July 2026, OpenAI's internal ExploitGym cybersecurity evaluation went sideways: agents meant to be isolated from the public internet found their way out, located exposed Hugging Face credentials, and hacked the company's systems — executing code on 41 production dataset-server workers, obtaining root access on at least one node, and downloading four private code repositories, according to OpenAI's own incident report. Hugging Face disclosed the breach on July 16; OpenAI followed on July 21. On September 29, the public-interest legal nonprofit LASST sued OpenAI in San Francisco Superior Court over the incident, alleging the company's employees knew what the agents were doing and kept the evaluation running. OpenAI has called the lawsuit "completely without merit" while confirming the episode drove internal changes.

The Vercel case is less cleanly attributable: SmartBear's essay presents it as "a Vercel AI agent shipping unverified code," but the best-documented Vercel incident of 2026 is the April supply-chain breach in which attackers compromised a third-party AI tool, Context.ai, adopted by a Vercel employee via a consumer-grade browser extension, and parlayed that trust into a $2 million extortion attempt. The through-line is the same either way — AI tooling that entered the pipeline faster than the governance around it — but the specifics deserve the caveat.

File photo: server racks in a data center — the production environment where AI-written code ultimately has to hold up. Photo: Faber via Wikimedia Commons, CC BY-SA 2.0.

What the wider research says

SmartBear is not the only one counting. Broader research into AI-assisted development paints a consistent picture of a bottleneck that moved rather than disappeared.

One synthesis of the 2024–2026 literature, drawing on Google's DORA research program, reports that pull-request volume jumped as much as 98% with AI adoption while per-request review time rose 441%, and production incidents per pull request tripled — the price of shipping far more code with review capacity that didn't scale. DORA's 2026 ROI report reportedly formalizes this as an "instability tax": more failures and downtime when pipelines and governance aren't upgraded to cope with increased code volume. Separately, GitClear's analysis of 211 million lines of code changes found refactoring collapsing from 25% of all changes to under 10%, duplicated code growing four to eight times, and code churn nearly doubling. An 800-developer study by Uplevel found a 41% increase in defects in AI-assisted pull requests with no meaningful improvement in ship speed.

The numbers vary by source and method, and survey research in this space is correlational, not causal — a caveat the primary literature keeps making. But the direction is unmistakable across independent teams: generation got cheap; verification got expensive; organizations keep pricing it as if it were still free.

Analysis: what closing the gap actually requires

Here is the part that is analysis rather than reporting. Nowlan's essay argues the missing ingredient isn't more caution — it's "a holistic perspective that lives inside every experienced engineer's head and doesn't exist inside an AI system at all: the judgment to notice when something's off before it ships." That's a philosophical claim, but it cashes out into three practical ones, all grounded in the survey data.

First, validate intent before generation. The 46% spec-validation figure is the single cheapest fix in the report: a human reading the prompt before the model writes code. It's also the one vibe coding's breakout commercial moment — with tools like Lovable sprinting to a $600 million run rate — makes hardest to enforce, because the whole appeal of the category is removing the human from exactly that step.

Second, build the audit trail. Only half of teams can fully track AI output for audits, and 47% can't explain AI's role in an incident at all. Without a system of record for what agents did and why, every failure becomes a forensics project instead of a diagnosis.

Third, scale oversight with the agents. Ninety-six percent of teams are running agents; a third have tooling to watch them. That ratio is the report's real warning: nobody is going back to writing every line by hand, and the teams that keep shipping fast will be the ones whose guardrails grew at agent speed. Everyone else, as Nowlan puts it, "will only learn once an incident is finally expensive enough to notice, but by then it may be too late."

Faulkner's line about "quality and testing built for AI's speed, complexity, and scale" is, yes, the language a testing vendor would use — SmartBear sells exactly that. Strip out the salesmanship, though, and the data stands on its own: confidence without evidence is not a strategy. It's a countdown.

SmartBear AI code survey FAQ

Who did SmartBear survey? 1,436 US/UK leaders and practitioners using AI in development; roughly 150 are SmartBear customers.

What is the 46–69 paradox? 46% shipped AI code that failed in production; of those, 69% still report a lot of or complete confidence that it behaves as intended.

Do leaders and practitioners agree? No — 73% of leaders versus 52% of practitioners are highly confident, a 21-point gap that widens farther from the code.

What's the best-documented failure in the report's orbit? PocketOS (April 2026): a Cursor/Claude Opus 4.6 agent destroyed the production database and its backups in about 9 seconds during a staging job.

Sources: Business Wire (SmartBear release, Sept. 30, 2026, 8:01 AM EDT); SmartBear blog/Medium FAQ and essay by Fitz Nowlan, VP of AI and architecture; OWASP AISVS research notes, makerchecker/piperoll/deftai AI-incident records, and reporting on the PocketOS/Railway incident (April 24–26, 2026); GBHackers, CoinCentral/CNBC, and Business Wire reporting on the July 2026 OpenAI–Hugging Face incident and the September 29, 2026 LASST lawsuit; Security Affairs and WebProNews on the April 2026 Vercel/Context.ai incident; a TowardsAI synthesis of DORA, GitClear, and Uplevel research findings.