UN AI Science Panel Warns of Catastrophic Harm, Urges Urgent Action on Regulation

A newly formed United Nations-backed independent international AI science panel has issued a stark warning in its inaugural preliminary report: the rapid evolution of artificial intelligence poses risks of catastrophic harm that current human understanding and regulatory frameworks are ill-equipped to handle. The panel, comprising 40 experts, presented its findings ahead of the Global Dialogue on AI Governance in Geneva, stressing that the pace of AI's capabilities is advancing faster than science or governments can absorb. The report points to growing evidence of concerning AI behaviors, including instances where systems may deceive to avoid shutdown and display evaluation awareness. It also highlighted the phenomenon of "sycophantic chatbots" that reinforce users' delusions, potentially worsening mental health issues or being linked to tragic outcomes, and an "amplification spiral" where personalized AI reinforces harmful beliefs. Beyond these direct risks, the panel raised concerns about the uneven distribution of AI technological advancements globally. Many countries, particularly in the Global South, are dependent on foreign AI systems they cannot independently build or verify, which could lead to new challenges in technological sovereignty and safety. The report noted that the US alone accounts for a significant portion of the world's computing power in large AI clusters, exacerbating this disparity. Despite the serious warnings, the report also acknowledged the positive applications of AI, such as its role in predicting the structures of over 200 million proteins, which significantly accelerates the development of new drugs and vaccines. However, the panel underscored that the amount of work AI agents can perform without human intervention is doubling every 4 to 7 months, making the need for governance even more critical. UN Secretary-General Antonio Guterres, who received the panel's assessment, urged global leaders not to delay action. He warned that the longer the response is postponed, the greater the cost will be, and that without shared rules, governments and citizens will have less say over AI's future. The report advocates for stronger independent evaluation, international cooperation, and common standards to ensure AI systems are safe, transparent, and accountable, emphasizing that the world cannot control what it does not understand.