News Briefing 1 March 2024

Autonomy in weapons systems, and AI in security and defence

• US/International: AI warfare is already here: An in-depth feature from Bloomberg explores the United States’ continued development and use of Project Maven, which is ‘a system built around powerful algorithms intended to identify personnel and equipment on the battlefield. Relying on breakthroughs in machine learning, the system can teach itself to pick out objects based on training data and user feedback. The Pentagon’s AI models can also learn whether changes to objects are likely to be significant—suggesting, for example, that an adversary is building a new military facility. All this is fused with information such as satellite imagery and geolocation data from communications intercepts on a single computer interface, called Maven Smart System.’  The software ‘determines what data to gather, collates and analyzes the resulting information and communicates a commander’s decision to act—potentially to a weapon system’, adding that ‘It’s up to humans to decide whether to pull the trigger.’ The piece quotes UN Secretary General Antonio Guterres: ‘Machines with the power and discretion to take lives without human involvement are morally repugnant and politically unacceptable and should be prohibited’.
• US: US Used AI to Help Find Middle East Targets for Airstrikes: Bloomberg reports that the United States used Project Maven to ‘narrow down targets for more than 85 US air strikes on Feb. 2.’ The report quotes Schuyler Moore, chief technology officer for US Central Command (which runs US military operations in the Middle East), who said that ‘We’ve been using computer vision to identify where there might be threats’. Bloomberg notes that ‘Moore’s comments mark the strongest known confirmation that the US military is using the technology to identify enemy targets that were subsequently hit by weapons’ fire.’ Moore emphasised that these capabilities ‘are being used to help find potential targets but not to verify them or deploy weapons against them’, and that ‘Humans constantly check the AI targeting recommendations.’ 
• France: French AI firm to process drone footage for automatic threat detection: DefenseOne reports that French AI firm Preligens, which ‘deploys artificial intelligence to observe Chinese and Russian military activity’, is ‘integrating video footage gathered by drones into its algorithms for detecting adversaries’ military movements.’ According to Sophie Hue, head of communication at Preligens, the company ‘recently developed a helicopter detector, that can now classify and identify precise models’, and ‘have also improved the vehicle detector, which is now capable of detecting and categorizing civil, armored, military and electronics (radars) vehicles.’
• International: The Perilous Coming Age of AI Warfare: Paul Scharre, Executive Vice President and Director of Studies at the Center for a New American Security, writing for Foreign Affairs, argues that ‘there is an urgent need for international agreement’ on rules around autonomous weapons systems, and that ‘International regulations, if carefully crafted and successfully implemented, could help mitigate some of the worst harms of autonomous weapons.’ Scharre provides a snapshot of current developments in the field of weaponised AI and suggests ‘five complementary initiatives that countries could pursue to reduce the threat of autonomous weapons’, including adoption of ‘a broad principle that establishes the minimum necessary human involvement in lethal decision-making’, including that ‘countries could also agree that in order for human control to be meaningful, any use of autonomous weapons must be limited in geography, time, and the targets being attacked’, and a ‘ban on autonomous weapons that target people.’ Similar measures have been called for by the Campaign to Stop Killer Robots for a number of years. For more on meaningful human control, see our report on Convergences in states positions on human control. For more on a prohibition on systems that target people, see here and here. 
• Gaza/Israel: How Israel Is Using AI as a Weapon of War:  Branka Marijan, a CIGI senior fellow and a senior researcher at Project Ploughshares, writes on Israel’s use of AI for ‘AI-assisted decisions’ in Gaza, noting that ‘Though we don’t know the full extent of the Israeli military’s use of AI, it’s clear what is happening in Gaza is far from the promised vision of “clean” war. The ongoing outcome also suggests that advanced militaries might be prone to accept speed in decision making over moral and perhaps even legal obligations.’
• US: Navy pulls trigger on new robotics warfare specialist rating: The U.S. Navy is reportedly ‘establishing a new “robotics warfare specialist” general rating’, as the Navy seeks to ‘build and develop a team for the next generation — a team who has the reps and sets in sensors, platform autonomy and mission autonomy programs, and who can provide input and machine learning feedback processes’, according to Chief of Naval Operations Adm. Lisa Franchetti.

Facial recognition, biometric identification, surveillance 

• US: Officials learn about impacts of facial recognition in Montana: Early last week, Senate Bill 397 that was adopted last year was under discussion where ‘Montana’s Law and Justice Committee spoke with representatives from law enforcement and public schools’. Amongst several concerns, the committee urged caution on the use of technology by mentioning that the ‘privacy of citizens should not be infringed upon.’
• Argentina: Buenos Aires’s facial recognition system to be examined by court: During 2019, The Fugitive Facial Recognition System (SNRP) was introduced in Argentina with the intention for the system to ‘accurately identify fugitives’. Two years later it was discovered that ‘the system had been used to monitor public figures and feed data from people with no criminal record’. An independent investigation also confirmed that the system is flawed because several ‘errors’ have led to ‘arbitrary police checks or arrests.’ The SNRP was suspended and is currently under review in an Argentinian court.’
• Canada: ‘Facial recognition’ error message on vending machine sparks concern at University of Waterloo: Students at the University of Waterloo are raising privacy concerns given that a dispenser/vending machine ‘displayed a facial recognition message despite no prior indication it was monitoring students.’ Invenda, the manufacturer of the vending machine claims that ‘the software is compliant with the European Union privacy law General Data Protection Regulation,’ but it has not been confirmed whether the system is legal in Canada. 

AI, algorithms and autonomy in the wider world

• International: Google pauses Gemini’s ability to generate AI images of people after diversity errors: ‘Gemini’, an image and language model developed by Google DeepMind, has been paused due to ‘the inaccuracies in some historical images its AI model generated’. Google has released an apology for the inaccuracies and has also mentioned that they are working on an improved version of Gemini that will be launched soon. 
• International: The frenzied hype around AI kept expectations high. The earnings calls are disappointing. 2024 is going to be a year of reckoning: Elizabeth Lopatto, writing for The Verge, says ‘Investments in AI haven’t paid off as handsomely as the market had been hoping.’ Given that in recent years, investors and high expectations for inflated profit margins within the sector but this has not happened. Lopatto notes that ‘for all the hype around incorporating AI into its Bing search engine, Microsoft’s market share grew by less than half a percent.’ She brings to attention that the reality is that AI tools could lead to a dip in profit margins also considering that AI tools could potentially ‘lead to bad publicity, lawsuits, and congressional hearings, with minimal productivity gains.’
• International: The AI series: AI and Surveillance Capitalism: This piece, part of Al Jazeera’s AI series, features a conversation between AI experts Camille François and Meredith Whittaker. The conversation centres around ‘the pivotal moment for AI’, fueled by the availability of massive amounts of data and computational power in the hands of a few large companies driven by the surveillance advertising business model. The speakers also touch upon topics such as biased algorithms in AI systems, human labour in content moderation, and the role of regulation.

Research and reports

• US: A.I. Joe: The Dangers of Artificial Intelligence and the Military: This new report from Public Citizen outlines some of the primary concerns around military applications of AI use and the dangers of autonomous weapons. The authors, Robert Weissman and  Savannah Wooten, also ‘provide a brief overview of the Pentagon’s AI policy’ and conclude with a set of recommendations for consideration, including that ‘The Pentagon should pledge not to use autonomous weapons and the U.S. government should support an international treaty prohibiting and regulating the use of such weapons.’