News briefing 06 December 2024 – 10 January 2025

Autonomy in weapons systems, and AI in security and defence

• International: Opportunities after the UNGA Resolution on Autonomous Weapons: Moving Toward a New Treaty: Writing for the Humanitarian Disarmament blog, Elizabeth Minor, Advisor at Article 36, highlights opportunities for the UN General Assembly informal consultations on autonomous weapons systems, due to be held later this year. Minor notes that ‘It is clear from international discussions that states have developed considerable common ground on how autonomous weapons systems might be regulated and thus have reached a point where negotiations on an effective legally binding instrument could commence.’ States have reached convergence in a number of significant areas, including on the issue of human control, and a growing number of states and stakeholders ‘have highlighted that autonomous weapons systems that target people raise specific concerns bearing consideration.’
• Ukraine: Ukraine collects vast war data trove to train AI models: Reuters says that Ukraine is ‘sitting on a valuable resource: millions of hours of footage from drones which can be used to train AI models to make decisions on the battlefield’. Over 2 million hours of battlefield footage from drones has been collected by OCHI, a non-profit Ukrainian digital system which centralises and analyses video feeds from over 15,000 drone crews working on the frontlines, since 2022.
• Israel/Palestine: Israel Loosened Its Rules to Bomb Hamas Fighters, Killing Many More Civilians: An investigation by The New York Times reveals more details about how Israel has been using its Lavender and Habsora (‘The Gospel’) systems: ‘In recent years, the Israeli military had developed computing systems, one of which was known as “The Gospel,” that could automatically cross-reference information from several different sources, including phone conversations, satellite imagery and mobile phone signals. In the chaotic opening weeks of the war, different intelligence units harnessed these automated computing systems in varied ways to triangulate data and locate militants. One common method involved automatically cross-referencing the location of a phone with its owner’s home address. When a phone appeared to be in roughly the same place as an address linked to its owner, the system flagged and recorded the owner’s phone calls. Arabic-speaking soldiers then listened to these calls to determine whether a wanted militant had been found. Some units used speech-to-text software to translate the conversations automatically. The military said officers always verified the information provided by the automated systems and it denied that artificial intelligence was ever more than the starting point of a human-led verification process. But the amount of verification varied from unit to unit, according to at least eight officers. Some officers said they would only confirm someone as a militant if they overheard the person speaking about their involvement in Hamas’s military wing. In other units, three officers said, an individual was considered a confirmed militant if he was simply listed in Lavender.’
• International: AI admin tools pose a threat to national security: Heidy Khlaaf, Chief Scientist at the AI Now Institute, writes that ‘In the pursuit of potential future efficiencies, the use of AI administrative tools by military agencies introduce real risks. This is a trade-off that their purported benefits cannot justify.’
• International/US: Meta-Powered Military Chatbot Advertised Giving “Worthless” Advice on Airstrikes: In a piece on Meta’s large language model (LLM), Llama, The Intercept highlights Meta’s partnership with Scale AI, ‘a $14 billion machine learning startup and thriving defense contractor.’ Scale AI is using Llama ‘to power a chat tool for governmental users who want to “apply the power of generative AI to their unique use cases, such as planning military or intelligence operations and understanding adversary vulnerabilities,” according to a press release.’ The Scale AI tool is called ‘Defense Llama’, and Scale AI says that it ‘was trained on a vast dataset, including military doctrine, international humanitarian law, and relevant policies designed to align with the Department of Defense (DoD) guidelines for armed conflict as well as the DoD’s Ethical Principles for Artificial Intelligence. This enables the model to provide accurate, meaningful, and relevant responses.’ However, demo examples used have been described as ‘useless’ by experts. The Intercept quotes Jessica Dorsey, a professor at Utrecht University School of Law and scholar of automated warfare methods, who said that ‘consulting Defense Llama seems to entirely circumvent the ostensible legal obligations military planners are supposed to be held to. “The reductionist/simplistic and almost amateurish approach indicated by the example is quite dangerous,” she said.’
• International: Beyond Human-in-the-Loop: Managing AI Risks in Nuclear Command-and-Control: This piece argues that, with the integration of AI into nuclear weapons systems becoming ‘a distinct possibility’, in order ‘to prevent nuclear calamity and ensure the responsible use of AI in nuclear command-and-control, states should move beyond mere prescriptive commitments to human oversight.’
• US: Deadly and Imminent: The Pentagon’s Mad Dash for Silicon Valley’s AI Weapons: Public Citizen, a member of the Stop Killer Robots coalition, has published a report on the U.S. Department of Defence’s Replicator programme (highlighted in numerous past news briefings; see more here), which recommends ‘that Deputy Secretary of Defense Kathleen Hicks and Secretary of Defense Lloyd Austin must clarify that the AI weaponry procured by Replicator will not be empowered to kill on the battlefield.’
• US: Eighteen ways Palantir wants the Pentagon to change: Palantir’s (see previous news briefing coverage here) Chief Technical Officer, Shyam Sankar, in an interview with Defense One, says that its government business ‘has re-accelerated dramatically’. When asked what defence areas the company will expand into, Sankar answered that ‘a big focus, obviously, Maven [one of the company’s systems] is at the pointy end of integrating AI into the kill chain… With Maven, you can perform targeting operations with 20 people that used to take us 2,000 people in Iraq. So that sort of efficiency really drives the ability to hide in a pure conflict. You’re much smaller, [there’s a] much smaller footprint, less support infrastructure, and [that] drives lethality.’
• Sri Lanka: LKI – ICRC seminar discusses Contemporary Challenges Concerning AI and Autonomous Weapons Systems: The Lakshman Kadirgamar Institute of International Relations and Strategic Studies in Sri Lanka recently hosted a panel discussion titled, ‘Artificial Intelligence (AI), Autonomous Weapons Systems (AWS) and Contemporary Challenges’ in collaboration with the ICRC. In a keynote speech, Additional Secretary for East Asia, the Middle East, and Africa at the Ministry of Foreign Affairs, Yasoja Gunasekera, underscored the urgent need for global action to regulate Artificial Intelligence and Autonomous Weapons Systems. Also speaking at the event was Yanithra Kumaraguru, Sri Lankan Coordinator for the Global Campaign ‘Stop Killer Robots,’ who underscored the critical role of ethics in shaping both the legal frameworks and the development of autonomous weapons, especially in areas where current laws may not provide clear guidance.
• Report: Bias in Military Artificial Intelligence: The Stockholm International Peace Research Institute (SIPRI) has released a new report titled ‘Bias in Military Artificial Intelligence’. The report finds that ‘the humanitarian consequences of bias in military AI relate principally to its use in targeting, but broader applications such as surveillance and humanitarian services should not be overlooked. The presence of implicit assumptions around gender, ethnicity, ability and other sensitive characteristics in military AI systems can result in misidentification of threats and nonthreats, flawed assessments of humanitarian needs, and invasive surveillance and monitoring practices.’
• US: Defense industry could see big shakeup under Trump: 2025 Preview: This piece by BreakingDefense covers the possible changes in defense procurement under the upcoming Trump administration. It says that ‘one way this dynamic could play out is greater preference for inexpensive uncrewed systems — an area where defense startups have firm footing — instead of the exquisite, expensive weapons platforms traditionally made by defense primes.’
• Gaza/ Israel: How Israel Uses AI in Gaza—And What It Might Mean for the Future of Warfare: Time covers the use of artificial intelligence by the Israeli military through systems such as ‘Lavender’ and ‘Gospel.’ The piece also notes that the use of AI systems in the military is not restricted to Israel, and that dual use technologies such as facial recognition are also quite concerning. 
• Israel: Israel creates hub to hasten military AI, autonomy research: Defense News reports on the Israeli Ministry of Defense establishing a new office of AI and autonomy administration which will ‘operate under the auspices of the Directorate of Defense Research & Development.’ The ministry’s director general, Eyal Zamir has said that ‘future battlefields will see integrated teams of soldiers and autonomous systems working in concert.’
• US: Palantir and Anduril join forces with tech groups to bid for Pentagon contracts: The Financial Times reports that Palantir and Anduril, among about a dozen other companies, are in talks to form a consortium which will be able to jointly apply for U.S. defence contracts. The piece reports that likely members of this consortium include ‘SpaceX, ChatGPT maker OpenAI, autonomous shipbuilder Saronic, and artificial intelligence data group Scale AI.’
• International: What’s next for AI in 2025: This piece in MIT Technology Review predicts five major advances in the field of artificial intelligence in 2025. Apart from advances in gaming, more sophisticated large language models, applications of AI in scientific research and emergence of new players in chip production, the piece highlights that 2025 is likely to see defense companies and mainstream AI companies collaborating more actively with military applications. 
• China: China’s use of AI, biometrics pose significant, persistent threats, DOD says: Biometric Update reports on a Department of Defense report titled ‘Military and Security Developments Involving the People’s Republic of China’ that highlights ‘advancements in AI and biometrics by the People’s Republic of China (PRC) pose significant risks to global security, particularly to the United States and Western nations.’ 

Facial recognition, biometric identification, surveillance 

• US: Federal joint agency report on law enforcement use of biometrics released: A joint report by the U.S. Department of Homeland Security, U.S. Department of Justice, and the White House Office of Science and Technology Policy highlights the need for transparency, oversight and accountability of using biometric data by law enforcement agencies. Biometric Update reports that the document ‘calls for agencies to make public disclosures about the types of biometric systems they use, the purposes they serve, and the policies governing their implementation.’ The report further ‘identifies facial recognition technology’s frequent use without individuals’ explicit knowledge or consent as a significant concern’ as well as its ‘propensity for errors, particularly among minority demographics, which underscores the potential for privacy violations and discriminatory practices.’ 
• Israel: Israel keeps emergency biometrics measures in place, more than a year into Gaza war: Regulators, legal experts and rights organizations are questioning the necessity of the expedited legislation allowing complete access of National Biometric Database for Israeli security forces by Israel’s Ministry of National Security, reports Biometric Update. The legislation was brought into force in response to the October attacks and is being regularly extended by the ministry and has also included access to security cameras without court approval since November 2023.   

AI, algorithms and autonomy in the wider world

• US: She didn’t get an apartment because of an AI-generated score – and sued to help others avoid the same fate: Saferent, an AI powered tenant screening tool, is being sued under the fair housing act for ‘disproportionately scoring Black and Hispanic renters who use housing vouchers lower than white applicants’, reports The Guardian. Mary Louis and Monica Douglas settled with Saferent and reached an agreement that ‘the company can no longer use a scoring system or recommend whether to accept or decline a tenant if they’re using a housing voucher.’ 
• US: DHS unveils ‘playbook’ for deployment of AI by the public sector: Biometric Update reports on the Playbook for Public Sector Generative Artificial Intelligence Deployment by the Department of Homeland Security which ‘outlines a comprehensive framework for integrating generative artificial intelligence (GenAI) responsibly into public sector operations. The report prioritizes key considerations such as privacy, security, civil rights, transparency, and online authentication.’ 
• International: The Petrov dilemma & AI: This piece in the New Indian Express uses the Petrov Dilemma, where the human intuition of Stanislav Petrov (officer on duty during 1983 Soviet Union’s early warning system for possible nuclear attacks) helped in averting a nuclear catastrophe, to contextualise present concerns in AI governance, reliability, and safety. 

Academic research papers

• Academic research paper: Basic Research, Lethal Effects: Military AI Research Funding as Enlistment: This research paper by David Gray Widder (Digital Life Initiative, Cornell University), Sireesh Gururaja (School of Computer Science, Carnegie Mellon University), and Lucy Suchman (Department of Sociology, Lancaster University) shows ‘how funding calls framed as basic research nonetheless enlist researchers in a war fighting agenda’ and ‘affirmation of progress in military technologies is qualified by acknowledgement of outstanding problems, becomes justification for additional investments in research.’ The paper notes that ‘an indicative analysis of a subset of DARPA calls focused specifically on AI’s promise for the battlefield highlights DoD imaginaries of technological solutions to longstanding problems, refigured and arguably exacerbated by previous investments in data-driven warfighting.’