How to Set Red Lines in the Real World

Last week, our Global Call for AI Red Lines captured international attention. More than 70 organizations and over 200 respected thinkers and leaders, including nine former heads of state and ministers and ten Nobel Prize recipients, signed on to a collective statement demanding that governments prevent unacceptable AI risks. 

Within days, the call was covered by more than 300 mainstream and specialized outlets from a variety of sectors, including TIME, Forbes, Barron’s, Newsmax, Semafor, Associated Press, ComputerWorld, and CryptoRank. The message was also carried to the world’s top international stages, at both the United Nations General Assembly’s opening and the UN Security Council, highlighting the global geopolitical relevance of the issue.

Co-organized by The Future Society, the French Center for AI Safety, and the Center for Human-Compatible AI, together with more than 20 partner organizations, the call also sparked serious policy discussions. Many in civil society and governments are now debating what specific red lines should look like. Our focus with the Global Call was not to define the exact substance of red lines, but to elevate them on the political agenda as a necessary priority for both national and international security. 

After a decade of work on international AI governance, we know that real (geo)politik ultimately shapes these decisions. It will be governments that lead dialogues and efforts to establish these red lines, defending both their common interests and sovereignty. The debate on international AI red lines needs to start with decision makers in Washington, D.C. and other key capitals. While realism about power dynamics is necessary in these negotiations, cynicism is not. For example, given the cross-border and security implications of some AI technologies, there are shared interests. The prudence and cooperation of many independent and sovereign nations will be essential to set enforceable limits.

Some agreements that certain risks are unacceptable already exist at various levels:

1. Major AI companies have long said limits are necessary, whether at official hearings, on the record, in their Responsible Scaling Policies, or in joint voluntary agreements. These limits are meant to delineate what is acceptable and unacceptable in AI development, and under what conditions. At the Seoul AI Safety Summit for example, 16 U.S. and Chinese firms, including Google, Microsoft, Meta, OpenAI, Anthropic, Zhipu.ai, 01.AI, and MiniMax, committed to set risk thresholds and to not develop or deploy the AI systems that go beyond these thresholds. This is in addition to several calls by scientists echoing the tech industry’s concerns.
2. In the United States, the AI Action Plan is an example of reconciling AI leadership with prevention of unacceptable risks–through its Incident Preparedness mandate. At the state level, numerous initiatives exist, some of which have been already enacted into law (e.g. Texas Responsible Artificial Intelligence Governance Act and California’s SB 53). Many of these initiatives prohibit some harmful AI use or developments, such as generation of child pornography or behavioral manipulation–indicating what American citizens consider unacceptable with AI.
3. In the rest of the world, the UK, the EU, China, and many other jurisdictions have unilaterally imposed domestic restrictions on what constitutes unacceptable risks of AI. They have written out under which circumstances some AI models cannot be used on their markets or in their national strategies. Whether implied in legislations, Action & Opportunities Plans, consumer protection, regulatory guidance, or codes of practice, these all embed de facto red lines.  

AI-related agreements with implied red lines already exist even internationally. The best example is the U.S.-China joint affirmation in 2024 that AI must never make decisions about nuclear weapons, setting a critical precedent for bilateral red lines. This should not come as a surprise. Coordination and even cooperation between geopolitical adversaries has often been the trigger for building broader international agreements, notably for nuclear non-proliferation and biological weapons control. The leadership of a superpower has also often been the catalyst for addressing global challenges–whether post-war reconstruction, terrorism finance, arms control, or dual-use technologies.

Together, this shows governments and industry are already converging on intolerable risks from AI. These form a “lowest common denominator” of governance. The opportunity now is to build on this existing patchwork of explicit and implicit red lines, and multilateralize what nation states and industry have already been willing to impose unilaterally. 

However, regardless of the exact content, three elements will always be essential to making red lines real, whether set domestically, bilaterally or multilaterally. These are the practical foundations for turning principles into enforceable limits. The Future Society is committed to working with policymakers and the wider ecosystem to move these priorities forward:

Coalition-building and track II dialogues 

The first element is building coalitions to discuss the existing suggestions for limits on unacceptable risk, and building reciprocal confidence among relevant parties. Through our policy research and convenings, we have already initiated multiple conversations on the governance and geopolitics of powerful AI with American, European, and international policymakers. Our work at France’s AI Action Summit and our convenings such as The Athens Roundtable on AI at the OECD have helped bring together a broad range of decision-makers, disentangle the stakes, and build confidence among stakeholders who would otherwise suspect foul play. Ongoing fora like the AISI network and various working groups at OECD.AI can also serve that purpose.

The next Athens Roundtable, held in an invite-only format at the UK’s Royal United Services Institute, will focus on the shared security challenges posed by powerful AI and international protocols needed to manage them, even in times of heightened geopolitical tensions. As mentioned above though, progress can only be made in key capitals–so no single organization can hope to convene all these dialogues single-handedly. If you are interested in contributing to these efforts, contact us. 

Advancing verification mechanisms 

Any limits on AI risks are only credible if they can be verified in an objective, legitimate way by all stakeholders. History shows that verification is what turns agreements into reality. Nuclear treaties only became credible once verification systems were in place. The same will hold for AI. The most trustworthy verification approach will always include human auditing, on-premise testing with representatives, or observers from all parties present in person. Progress is already underway with the creation of the AISI network and pilots of joint evaluative exercises. 

Enforceable AI red lines will also require robust methods to detect and monitor compliance at scale. Unlike nuclear technology, AI is more widespread, and verification will require automation. Just as seismographs can be used to detect nuclear tests, new tools are emerging to detect large AI training runs and discriminate between compliant and non-compliant compute usage. President Trump’s recent announcement that the United States will develop an AI-based verification system to enforce the Biological Weapons Convention demonstrates continued American leadership in using technology to strengthen international agreements. Likewise, The Future Society is fostering the maturation of an assurance and verification technology ecosystem to ensure that cost-effective and scalable solutions continue to improve and remain available to governments. If you are interested in contributing to these efforts, contact us. 

Formal enforcement and investigative processes

Beyond verification, any successful multilateral agreement requires agreed-upon processes to maintain the regime. For example, agreement on when and how to trigger verification, investigate, assess whether non-compliance is malicious or accidental, trigger emergency responses, trigger legitimate penalties, learn from accidents, refine the tooling and relevant safety standards, integrate new nation states to the regime, etc. There again, a lot can be learned from international agreements that have not only maintained themselves but also self-improved over time, such as the International Civilian Aviation Organization.  

Fortunately, these are questions that many relevant governance actors have already asked themselves, whether in industry-driven fora such as the World Economic Forum, multilateral engagements such as the OECD and AISI Network, or domestically to enforce existing restrictions. The Future Society has been an active participant to many of these discussions, and has even played a leading role in shaping several of them. Now more than ever, we will redouble our efforts towards proposing and refining these processes, until they can be mutually agreed upon by all parties. If you are interested in contributing to these efforts, contact us. 

At The Future Society, we are committed to helping governments, civil society, and industry navigate this landscape together. Join us in building this coalition, whether as funders, startups, NGOs, civil society, policymakers, or otherwise–your participation can help make real progress on all these fronts.