News Briefing 23 September – 04 October 2024

Autonomy in weapons systems, and AI in security and defence

• International: International Humanitarian Law and the Challenges of Contemporary Armed Conflicts: Building a Culture of Compliance for IHL to Protect Humanity in Today’s and Future Conflicts: A report by the ICRC outlines key challenges in modern warfare, including nuclear weapons, protection of civilians, new military technologies like autonomous weapons, and humanitarian issues such as missing persons and separated families. It emphasizes the importance of upholding IHL to protect victims of war, calls for nuclear disarmament, and highlights the need for improved compliance to reduce the human cost of conflicts. The report aims to foster reflection on IHL’s role in addressing these contemporary challenges.
• US: Pentagon announces Replicator 2 initiative as US eyes more AI weapon capabilities: The Pentagon has launched Replicator 2, a new initiative aimed at countering drone threats to protect critical military installations and assets. Defense Secretary Lloyd Austin directed top military officials to focus on overcoming challenges in production capacity, technology innovation, and system integration related to small drones. This follows the original Replicator initiative, announced in 2023, which aims to deploy thousands of AI-powered autonomous weapon systems by 2025. The goal is to enhance U.S. defense capabilities against larger adversaries, like China, but concerns remain regarding funding and the program’s progress.
• International: Pact for the Future directs states to advance discussions on killer robots with urgency: Stop Killer Robots commented on the ‘Pact for the Future‘ that was adopted by world leaders at the opening plenary of the Summit of the Future at the United Nations Headquarters in New York on 22 September 2024. SKR welcomes the reference to autonomous weapons which calls to ‘advance with urgent discussions’ on the issue.
• International: Will AI fundamentally alter how wars are initiated, fought and concluded?: Erica Harper, Head of Research and Policy at the Geneva Academy of International Humanitarian Law and Human Rights, wrote this piece in Humanitarian Law and Policy, an ICRC blog. As AI aided decision support systems are being actively incorporated into militaries, the piece raises concerns on how this might impact the decision making of leaders to engage in hostilities. Further, distancing the combatants from the battlefield might have impacts on the cognition and human judgment, which are yet to be studied. Similarly, the piece highlights the centrality of human emotions and judgment for cessation of hostilities. 
• US/ International: Trae Stephens Has Built AI Weapons and Worked for Donald Trump. As He Sees It, Jesus Would Approve: This piece in Wired covers an interview of Trae Stephens, co-founder at Anduril by Steven Levy from Wired. The interview covers a broad range of topics including the growth of Anduril, its stand in 2018 against building lethal weapons and eventual pivot to building fighter planes, underwater drones, and other deadly weapons of war, as well as regulation of automated military technologies. 
• International: Unicorns for Uniforms: On the Problematic Allure of VC Investments in Defence: Elke Shwarz, Professor in Political Theory at Queen Mary University London, writes for OpinioJuris on the impact of venture capital on the lucrative and growing market of military AI. Shwarz says that venture capital and silicon valley startups that are funded by them have a ‘mandate of growth-at-all cost and at high speeds’, which cannot accommodate ‘ethical considerations about war and its human consequences.’ 
• International: Transcending weapon systems: the ethical challenges of AI in military decision support systems: This blog post by Matthias Klaus explores the ethical implications of AI-assisted decision support systems (DSS) in military operations. It highlights concerns about biases, reduced human autonomy, and erosion of moral responsibility. Klaus calls for ‘robust ethical frameworks and guidelines’ as well as ‘continuous training’ to ensure responsible use of AI in decision-making, emphasizing the importance of maintaining human oversight in military contexts.
• International: UK, US and Canada to collaborate on cybersecurity and AI research: The UK, US, and Canada have agreed to collaborate on cybersecurity and AI research to enhance defense and security. Led by the UK’s Defence Science and Technology Laboratory (Dstl) and Canada’s Defence Research and Development Canada (DRDC), the partnership focuses on ‘real-world challenges’, aiming to reduce duplication of efforts and leverage existing research. Current projects include the CASTLE program, which trains AI for autonomous cyber defense. 
• International: WCC co-sponsored side event at UN General Assembly tackles faith, ethics, and AI: At a side event during the 79th session of the UN General Assembly, the World Council of Churches (WCC) co-sponsored a discussion on the ethical implications of AI for global peace and security. The panel brought together religious leaders, technology experts, and policymakers to address concerns about AI, especially regarding autonomous weapons systems. Key participants emphasized the need for responsible AI development that promotes peace, justice, and human rights. Dr. Ryan Smith from the WCC highlighted the evolving role of faith communities in influencing policies on AI governance, aiming to ensure that such technologies respect human dignity. The event underscored the urgency of ethical considerations in technological advancements to safeguard humanity. Isabelle Jones said ‘technology can and should be used to promote peace, justice, and human rights, but in order for that to happen, we need responsibility for the development and the use of new technologies that we create and the roles that they play in our lives. We have the opportunity to do that.’

AI, algorithms and autonomy in the wider world

• International: AI is a double-edged sword that requires proper governance handling, according to Professor Marwala: Professor Tshilidzi Marwala, the UN Under-Secretary General and Rector of United Nations University, emphasized both the potential and risks of AI during his keynote speech at the 2024 ICEGOV Conference in South Africa. He highlighted that AI, while transformative in addressing societal challenges like poverty and climate change, requires careful governance to prevent exacerbating inequalities. Citing data from the 2023 Sustainable Development Goals (SDGs) Summit, Marwala noted underperformance in achieving the 2030 Agenda, with AI potentially ‘enabling 79% of SDG targets’. He stressed the importance of aligning AI governance with human rights principles and ‘behavioral science’ for responsible deployment across sectors such as health and education.

Facial recognition, biometric identification, surveillance

• US: Commission on Civil Rights rings alarm bell on law enforcement use of AI tool: A U.S. Commission on Civil Rights report warns that facial recognition technology (FRT), increasingly used by law enforcement, airport security, and public housing surveillance, ‘disproportionately harms people of color and women.’ The report highlights the lack of federal regulations governing FRT use, raising concerns about privacy violations, wrongful arrests, and discrimination. The technology, deployed by federal agencies like the FBI and Department of Homeland Security, has been criticized for its inaccuracies and bias. The Commission calls for stronger regulations, fairness testing, and legal recourse for those harmed by the technology. Civil rights advocates hope the report will spur policy changes.
• UK: Biometric data: Misuse, use, and collation: The POSTnote research briefing report highlights the use of biometric data, such as fingerprints, faceprints, and voice recognition for personal identification. While it offers convenience in security and identity verification, it poses significant risks, including the inability to change biometric traits if compromised. The growing adoption of biometric systems is driving a global market expected to reach £46 billion by 2032. Advances in AI have raised ethical concerns, as biometric data could reveal sensitive personal information, prompting calls for stronger regulations to safeguard privacy and civil liberties.
• US: Harvard Kennedy School’s Carr Center Holds Inaugural Event for ‘Surveillance Capitalism’ Initiative: Harvard Kennedy School’s Carr Center for Human Rights Policy launched the “Surveillance Capitalism or Democracy?” initiative, discussing the risks of surveillance capitalism. Key speakers, including Nobel laureate Maria Ressa and author Shoshana Zuboff, highlighted privacy threats, the exploitation of personal data, and the urgency for stronger regulations. Zuboff argued that current regulations are ineffective, stressing the need for a decisive response to protect privacy and democracy from the power of tech giants. The event emphasized the critical role of rules and collective action in shaping a surveillance-free future.
• US: US accuses social media giants of ‘vast surveillance’: The FTC has accused major social media platforms like Meta, YouTube, and TikTok of engaging in large-scale data surveillance to profit from users’ personal information. The report highlights concerns about privacy risks, identity theft, and stalking, especially for children and teens. FTC Chair Lina Khan emphasized that targeted advertising fuels mass data collection, endangering privacy. Despite industry claims that users understand the trade-off for free services, the report called for stronger data protection practices and federal privacy laws to limit these surveillance activities.