Weak AI safety regulations may backfire, creating products that are potentially more dangerous than AI products developed under no regulation at all, according to a new study published in the Proceedings of the National Academy of Sciences. The research, conducted by a team from Cornell University and Carnegie Mellon University, applies theoretical economics and game theory to model the interactions between AI developers and downstream companies that integrate AI into real-world applications.

The core finding is that regulation must be strict and must target the entire AI supply chain — from the creators of general-purpose AI models like OpenAI, Google, and Anthropic to the specialized companies that deploy AI in medical diagnostics, customer service, or autonomous vehicles. Focusing only on downstream users might seem logical at first, but the study argues it can reduce overall safety by encouraging free-riding behavior among the model developers.

The Study's Key Findings

The researchers built a theoretical model that simulates how companies respond to different regulatory regimes. When regulators focus exclusively on downstream companies — those applying AI in specific contexts — the general-purpose AI developers tend to cut back on their own safety investments, such as third-party audits, red-teaming, and robustness testing. They assume that the downstream companies will be forced to ensure safety of the final product, so they shift the burden.

This free-riding behavior leads to a net decrease in safety compared to a scenario with no regulation at all. Without any rules, each company might make some voluntary safety efforts to avoid reputational damage or liability. But with weak regulation that only targets one part of the chain, the unregulated players have an incentive to reduce their own safety spending, hoping that the regulated players will pick up the slack. The net effect is a less safe overall AI ecosystem.

The Game Theory Behind AI Safety

The situation is a classic example of the prisoner's dilemma, a foundational concept in game theory. In this scenario, two rational actors have the option to cooperate or betray each other. If both cooperate by investing in meaningful safety, they achieve the best collective outcome. If both betray by skimping on safety, they get a mediocre outcome. But if one cooperates while the other betrays, the one who cooperates suffers the worst outcome — high costs with little safety gain — while the betrayer enjoys low costs and a free ride.

When each player is uncertain about what the other will do, the rational choice for each is to betray (i.e., not invest in safety), leading to a suboptimal outcome for everyone. Strict regulation that applies across the entire supply chain changes the incentive structure. It ensures that both parties are required to make adequate safety investments, turning the game into a cooperative equilibrium where both benefit from higher safety and shared responsibility.

Benjamin Laufer, the study's principal author, explained that people often think of AI as a single object, but it involves a complicated set of stakeholders and actors, each with their own contributions to the technology. To regulate thoughtfully, policymakers must consider the whole supply chain, not just a single provider or entity.

The Free-Riding Problem in Detail

The free-riding problem becomes especially acute when general-purpose AI developers release models that can be fine-tuned for many different downstream uses. If a downstream company builds a medical diagnosis system using a foundation model, the safety of that system depends both on the robustness of the base model and the precautions taken during the specific application. If the upstream developer knows that the downstream company will be held liable for any errors, they may underinvest in making the base model safe, assuming the specialist will catch and fix issues. However, the downstream company may lack the resources or expertise to fully audit the base model's behavior, especially if it is a complex black box.

The result is a gap in safety coverage — each party assumes the other is handling it. This dynamic has been observed in other industries, such as software supply chain security and pharmaceutical manufacturing, where weak regulation on one node of the supply chain can lead to catastrophic failures.

The Broader AI Regulation Debate

The study comes at a time when the United States government and Silicon Valley are fiercely debating how to regulate artificial intelligence. Two main camps have emerged. On one side are anti-regulation technologists who advocate for light federal guardrails, aligning with the current administration's approach to AI governance. They argue that the AI industry should be free to innovate as quickly as possible to win the global AI race against China, and that overregulation would stifle progress.

This camp often labels proponents of stricter regulation as doomers or alarmists who are trying to capture the regulatory process for their own benefit. On the other side, supporters of stricter federal AI regulation warn that the industry, in pursuit of wider profit margins, is underestimating or underselling the risks of unconstrained AI development. The list of potential downsides includes:

• AI systems exhibiting unpredictable or harmful behaviors, sometimes called AI psychosis.
• Negative community health consequences from the energy and water consumption of massive data centers.
• A feared unemployment crisis as AI adoption automates jobs across sectors from customer service to law.
• Amplification of bias and discrimination when models are deployed without adequate testing.

But the researchers argue that safety versus revenue does not have to be an either-or situation. According to their model, stronger, well-placed regulation can benefit all players by improving both the safety of end products and the utility that general-purpose AI creators and downstream specialists derive from their investments. Utility in the model is defined as revenue share minus investment cost.

This sweet spot exists when regulators set meaningful safety standards and enforce them uniformly across the supply chain. The model shows that if both types of companies are required to meet a minimum safety threshold, they can coordinate on higher investment, leading to higher quality and potentially higher revenue as trust in AI grows.

Historical Context and Examples

Similar dynamics have played out in other technology sectors. In the early days of the internet, weak regulation of cybersecurity allowed companies to pass the buck, leading to widespread vulnerabilities that were exploited by criminals and state actors. It wasn't until stricter rules like the GDPR and sector-specific cybersecurity frameworks were introduced that companies began to take shared responsibility seriously. However, these regulations were often applied after major incidents, highlighting the cost of delayed action.

In the AI space, several incidents have already illustrated the dangers of weak oversight. For example, when general-purpose language models were released without adequate safety testing, they were quickly used to generate harmful content, spread misinformation, or perpetuate biases. Downstream companies that built products on these models faced reputational crises, while the original developers faced little accountability.

The authors of the new study hope their work provides a quantitative foundation for designing smarter regulation. They emphasize that regulation should be seen not as a burden but as a coordinating mechanism that enables the whole industry to move forward safely. Laufer stated that the regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist — but that dynamic is exactly what leads to the prisoner's dilemma. By regulating everyone, the government removes the temptation to free-ride and creates a level playing field where safety becomes a shared priority.

As the United States moves closer to enacting federal AI legislation, the study offers a clear warning: piecemeal or weakly enforced rules could be worse than having no rules at all. True safety requires comprehensive regulation that covers every layer of the AI supply chain, from the foundational models to the final consumer products.

Source: Gizmodo News