Gavin Newsom vetoes California’s contentious AI safety bill

California Governor Gavin Newsom has vetoed a contentious artificial intelligence safety bill that would have required companies to make sure their AI models don’t cause major harm.

The bill, called SB 1047, was poised to be one of the most consequential pieces of AI regulation in the U.S., given California’s central position in the tech ecosystem and the lack of federal legislation for artificial intelligence. In recent weeks, the bill had divided the AI industry and drawn significant criticism from some tech leaders and prominent Democrats.

“While well-intentioned, SB 1047 does not take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data,” the governor wrote in a statement on Sunday. “Instead, the bill applies stringent standards to even the most basic functions — so long as a large system deploys it. I do not believe this is the best approach to protecting the public from real threats posed by the technology.”

In a letter explaining his decision to veto the bill, Newsom wrote that he agrees with proponents of the bill that “we cannot afford to wait for a major catastrophe to occur before taking action to protect the public,” but that any regulation “must be based on empirical evidence and science.” The governor pointed to his executive order on AI as well as several other bills he has signed in recent weeks that regulate the technology around “specific known risks” such as deepfakes.

‘Reasonable care’

SB 1047 mandated that companies developing powerful AI models take “reasonable care” to ensure that their technologies don’t cause “severe harm” such as mass casualties or property damage above $500 million. The bill, which was introduced by Democratic state Senator Scott Wiener and passed the state Senate in May, would have required companies to take precautions such as implementing a kill switch that could turn off their technology at any time. It also called for AI models to be submitted to third-party testing to ensure they are minimizing grave risk.

Additionally, the bill would have created whistleblower protections for employees at AI companies that want to share safety concerns. Companies that weren’t in compliance with the bill could have been sued by the California attorney general.

Supporters argued that SB 1047 would create common-sense legal standards to hold AI companies accountable. But venture capitalists, startup leaders and companies like OpenAI said the bill could hurt innovation and perhaps even drive AI companies out of the state. Critics of SB 1047, including OpenAI, tech incubator Y Combinator and VC firm Andreessen Horowitz, have registered lobbyists working on the bill.

“The AI revolution is only just beginning, and California’s unique status as the global leader in AI is fueling the state’s economic dynamism,” Jason Kwon, chief strategy officer at OpenAI, wrote in a letter last month opposing the legislation. “SB 1047 would threaten that growth, slow the pace of innovation, and lead California’s world-class engineers and entrepreneurs to leave the state in search of greater opportunity elsewhere.”

Lawmakers opposed

Lawmakers including former House Speaker Nancy Pelosi, Representative Ro Khanna and San Francisco Mayor London Breed also voiced their opposition, echoing concerns from the tech industry that the bill could impede California’s leadership in AI innovation. Newsom recently said he was concerned the bill might have a “chilling effect” on AI development.

The bill had also earned backing from some notable names in tech late last month in the days leading up to its passage by California’s legislature. Elon Musk unexpectedly voiced his support, even though he said it’s a “tough call and will make some people upset.” OpenAI rival Anthropic, which has a reputation for being safety-oriented, said the bill’s “benefits likely outweigh its costs,” though the company said some aspects of SB 1047 remained “concerning or ambiguous to us.”

Wiener had defended the bill, stressing that its provisions only apply to companies that spend more than $100 million on training large models or $10 million fine-tuning models, which would exempt most smaller startups. The lawmaker has also said that while he would support federal legislation, Congress has been historically slow to regulate tech and that in the absence of national action, he believes the state has a responsibility to lead.

Along with his SB 1047 veto announcement, Newsom said he will conduct an analysis of leading AI models’ risks and capabilities led by several academics, including AI scholar and entrepreneur Fei-Fei Li.

The governor also signed a bill on Sunday, SB 896, that regulates how state agencies use AI and conducts several agencies to produce a report on the risks and benefits of the technology.