Main Insight
With increasingly powerful models becoming widely adopted, serious incidents driven by AI will also become more common. We explore how accident prevention approaches in other industries can strengthen the EU's AI governance regime.
Serious Incident Prevention for AI: Lessons From Other Industries and Recommendations for the EU AI Office
November 20, 2025
As any industry grows, there are inevitably tough lessons learned when things go wrong. Whether it’s how toxic disasters have shaped the chemical industry or the implications of airliner collisions for the aviation industry, there are several clear examples of major accidents that have led to the development of industry policies and institutions to prevent future occurrences.
Today, the AI industry must grapple with the increasing likelihood of serious incidents, as frontier AI models become more capable and widespread and AI-related incidents are rising. Instead of learning the hard way, what can the AI industry take away from other industries to avoid a major AI disaster? And how can these insights be incorporated into AI governance approaches, particularly in the European Union?
Why We Should Act Now on Serious AI Incident Prevention
With powerful frontier AI advancing rapidly towards more capable and autonomous systems, the risk that serious AI incidents of disastrous scale will occur in the near future is growing significantly.
The OECD’s AI Incidents and Hazards Monitor tracks events, circumstances, or series of events where the development, use, or malfunction of AI systems directly or indirectly leads to harm–or could plausibly lead to harm. According to the Monitor, AI incidents are becoming more and more common. For example, the number of monthly AI incidents covered by reputable news outlets increased by 50 percent over the last six months.
While not all of these incidents have consequences on a wider scale, accidents like the 2010 Flash Crash demonstrate how the expected greater autonomy of AI systems could lead to cascading, high-velocity malfunctions and cause severe harm. General-purpose AI adoption is quickly expanding across industries, and its economic impact will likely increase in the future. The uncomfortable reality of today’s AI boom is that we’re setting up the dynamics for an AI-related disaster to have negative impacts on a large portion of humanity.
This leads to a strong case for prioritizing the prevention of serious incidents from frontier AI models. Rather than wait for something to go wrong and then take steps afterward, measures should be implemented early to avoid a major disaster in the first place. Fortunately, other industries can illuminate a roadmap for the way forward.
How Major Disasters Pushed Other Industries to Implement Accident Prevention Policies
In various critical sectors, major accidents have had devastating consequences for public health and the environment. Notable examples include: the Seveso chemical disaster, the Chernobyl nuclear power plant accident, the Deepwater Horizon oil spill, the collapse of the Tacoma Narrows Bridge, and the Grand Canyon airline collision.
In reaction to these major accidents, regulators have made significant adjustments and enhancements to safety requirements. In some industries, operators are required to develop Major Accident Prevention Policies to anticipate and prevent major incidents. In response to the Seveso chemical disaster, the European Parliament and Council passed the first Seveso Directive. The current Seveso-III Directive is recognized as a global benchmark for industrial accident policy, and the latest implementation report by the European Commission concluded that it led to significant improvements in industrial safety.
Other jurisdictions have implemented equivalent requirements in the chemical industry, and the EU and its member states (as well as other countries) have adopted similar frameworks in industries such as offshore oil and gas and mining. Other industries such as the nuclear, aviation, and pharmaceutical sectors rely on strict authorization and certification requirements, as well as independent investigations, to reduce the risk of major accidents.
Existing Accident Prevention Approaches and What This Could Look Like for the Frontier AI Industry
We analyzed existing EU and international regulations in the chemical, offshore oil and gas, mining, and transportation industries to identify policy requirements used for major accident prevention. Many of these could be replicated in some form for frontier AI, with clear analogies that we cite below.
Policy Requirements and Frontier AI Industry Analogies for Incident Prevention
Full version here.
| Category | Policy Requirement | Frontier AI Industry Analogy |
|---|---|---|
| Major accident prevention policy | Major Accident Prevention Policy (MAPP) | Frontier AI companies draw up serious incident prevention policies (SIPP). |
| Safety management system (SMS) | Frontier AI companies establish a safety management system (SMS) for serious incidents. | |
| Safety report demonstrating that a MAPP and SMS have been put into effect | Frontier AI companies produce a model safety report demonstrating that SIPP and SMS have been put into effect. | |
| Emergency plans in case of major accidents | Frontier AI companies draw up emergency plans for measures to be taken in case of serious incidents. | |
| Reporting requirements after a major accident | As soon as possible after any serious incident, frontier AI companies inform the relevant authorities. | |
| Independent oversight and verification | Independent body overseeing the implementation of the MAPP | An independent AI agency is responsible for overseeing and enforcing the implementation of the SIPP |
| Official notification mechanism | Frontier AI companies notify authorities in advance of new frontier model releases and substantial modifications. | |
| Independent verifications and inspections by competent authorities | Frontier AI companies establish schemes for independent verification of their SIPP. The regulatory authority performs independent safety testing of the models (either directly or by outsourcing to a reliable third party) and collects data to verify organizational procedures of the companies. | |
| Penalties for infringement and prohibition of use if measures to prevent major accidents are seriously deficient | The relevant authority can issue appropriate penalties to frontier AI companies, including prohibition of deployment/withdrawal from the market, if measures to prevent serious incidents are seriously deficient. | |
| Requirements for relevant authority to take action after a major accident | Following a serious incident, the relevant authority ensures that any necessary urgent, medium-term, and long-term measures to prevent harm are taken and makes recommendations for future preventive measures. | |
| Confidential reporting mechanisms (“whistleblowing”) | Employees of frontier AI companies and other individuals are able to confidentially report safety concerns relating to AI models, both internally and to relevant authorities. | |
| Independent investigations | Independent safety investigation authority | A safety investigation authority, independent of the general AI regulatory authority, investigates each serious incident. |
| Investigation authority to recommend preventive actions to regulatory authority | After each investigation, the safety investigation authority recommends relevant actions to the frontier AI company and the regulatory authority to prevent further serious incidents. | |
| International collaboration | Information exchange between competent authorities | AI regulatory authorities and AI safety institutes exchange relevant information on SIPPs and their implementation. |
| International agreement on minimum standards | The international community agrees on clear, verifiable minimum safety standards for AI models. |
What the European Union Should Do Next on Serious AI Incident Prevention
With the EU AI Act (AIA) and the Safety and Security Chapter of the Code of Practice (CoP), EU law already contains some serious-incident-related regulation. Article 55(1)(c) AIA requires providers of General-Purpose AI (GPAI) models with systemic risk to report “relevant information about serious incidents and possible corrective measures to address them” and Measure 1.2 of the CoP requires Signatories to “implement systemic risk mitigations [that] also address serious incidents as appropriate.” However, other policy requirements identified are missing, in particular an explicit requirement for a Serious Incident Prevention Policy (SIPP).
Based on existing provisions in the AIA and CoP, we recommend that the European Commission/EU AI Office:
- Issue clear guidance that requires the Safety and Security Framework (Commitment 1 CoP) to take into account all elements of a safety management system for serious incidents.
- Issue clear guidance on how the Model Report required by Commitment 7 in the CoP shall demonstrate that possible incident scenarios have been identified, and the necessary measures to prevent serious incidents have been taken, so it can also serve as a safety report in relation to serious incident prevention measures.
- Develop internal procedures for verifying the identification of serious incident scenarios and measures taken to prevent serious incidents in the Safety and Security Framework and Model Report submitted by the Signatories.
- Establish a mechanism for confidential reporting of safety concerns relating to frontier AI models (“whistleblowing”) to the AI Office and the investigation of such reports.
Additionally, in consultation with GPAI model providers and independent experts, the EU should work on clarifying and strengthening the CoP in light of the above requirements. This should include explicitly requiring Signatories to develop a SIPP and emergency plans, and clarifying Commitments 1 and 7 to include all relevant elements of SMS and Safety Reports. It should also be made explicit that SIPP, SMS, Model Safety Reports, and emergency plans need to be communicated to the AI Office in advance of model releases or major modifications, ideally via a structured dialogue.
Furthermore, legislation could be prepared to:
- strengthen the EU AI Office by providing it with additional enforcement powers, including a clear mandate to prohibit the placing on the market of models or to issue a recall if needed;
- transform the EU AI Office into an independent AI agency with dedicated staff and legal personhood; and
- create an independent safety investigation authority with an obligation to investigate serious incidents and recommend relevant actions. The authority should be functionally independent from the Commission and EU AI Office.
Finally, in consultation with their international partners, the EU could create international collaboration mechanisms to exchange information between regulatory authorities and AI safety institutes. This should ultimately also lead to clear, verifiable minimum safety standards for AI models.
The risks from ever-more powerful AI systems cannot be ignored. We do not have to wait until serious incidents occur in order to take action. Leveraging the lessons learned from other safety-critical industries can help us avoid major AI disasters, and should be a priority for policymakers.
Co-authored by Sven Herrmann, Visiting Fellow at The Future Society, and Toni Lorente.
