News Briefing 29 July – 09 August 2024

Autonomy in weapons systems, and AI in security and defence

• EU: Europe Is Pumping Billions Into New Military Tech: Luca Zorloni, writing for the Wired provides some analysis on the recent move by the European Commission to dramatically increase its defense budget to ‘€7.3 billion’ for 2021-2027, up from ‘€590 million’ previously. The funding will advance military technologies, including autonomous drones, AI systems, and next generation weapons. This investment aims to enhance Europe’s military capabilities and integrate cutting-edge technologies. As Automated Decision Research Manager Catherine Connolly noted when speaking to The Guardian, ‘the money that we’re seeing being poured into autonomous weapons and the use of things like AI targeting systems is extremely concerning.’
• New Zealand: NZ Defence Force joins US deploying artificial intelligence for ‘kill chains’: New Zealand has been actively involved in joint military exercises with the US and other allies by focusing on integrating advanced AI-powered systems into military operations. These exercises are aimed at enhancing the efficiency of “kill chains” and developing what is termed a “kill web’’.
• Australia: Australia takes aim with US-made loitering munitions: This piece in C4ISR Net reports on the procurement of US manufactured Switchblade 300 from AeroVironment by Australian Defense Force (ADF). This comes at a time when the ADF has been criticized for delaying the incorporation of loitering munitions into their arsenal. Minister for Defence Industry Pat Conroy has said that ‘new technology and asymmetric advantage’ in autonomous weapon systems is a priority highlighted in the Defense Strategic Review. 
• US: Anduril unveils new ‘Arsenal’ factory in bid to scale up weapons production: Breaking Defense reports on Anduril’s announcement to open a factory, named ‘Arsenal’ in the US, which will ‘boost the firm’s autonomous weapons output dramatically, with a focus on systems that can be rapidly manufactured at scale.’ Chief Strategy Officer, Chris Brose has said that the factory ‘needs to be capable of producing tens of thousands’ of systems for projects ‘such as CCA or large, undersea drones for the Navy.’ Previous news briefing coverage of Anduril can be seen here. 
• International: The Risks of Artificial Intelligence in Weapons Design: Harvard Medical School News covers a conversation with Kanaka Rajan, associate professor of neurobiology in the Blavatnik Institute at Harvard Medical School. The conversation covers a position paper written by Rajan, Riley Simmons-Edler and Ryan Badman and Shayne Longpre for the 2024 International Conference on Machine Learning. Rajan outline three major risks of AI enables military technology as ‘first, how these weapons may make it easier for countries to get involved in conflicts; second, how nonmilitary scientific AI research may be censored or co-opted to support the development of these weapons; and third, how militaries may use AI-powered autonomous technology to reduce or deflect human responsibility in decision-making.’ She concludes by saying that the realistic way forward would be ‘ to ban and regulate the most egregious classes of AI weapons as soon as possible and for our communities and institutions to establish boundaries that should not be crossed.’ 
• International: ‘I’m afraid I can’t do that’: Should killer robots be allowed to disobey orders?: This piece in the Bulletin for Atomic Scientists by Arthur Holland Michael covers a paradoxical situation created between balancing the need for human control, and the ability for an autonomous weapon to reject an illegal order. Michael says that while allowing an autonomous weapon to reject an illegal order would undermine human control, disallowing the same would counteract the aim to create ‘ethical and yet controllable killer robots.’ For this situation, he further says that militaries ‘need to either demonstrate that it’s possible to build ethical, responsible autonomous weapons that don’t say no, or show that they can engineer a safe and reliable right-to-refuse that’s compatible with the principle of always keeping a human in the loop.’

Facial recognition, biometric identification, surveillance

• UK: UK government to expand facial ID to fight rioters – will it work?: This piece in Context reports on the ongoing anti-immigration riots in the UK, and the use of facial recognition technologies by law enforcement agencies through ‘live facial recognition, which scans in real-time and checks faces against a watchlist; retrospective facial recognition, which is applied to images taken from CCTV; and operator-initiated facial recognition which works via a mobile phone app for in-person checks.’ 
• US: US Army moves closer to fielding next-gen biometrics collection: Biometric Update reports that the US Department of Defense (DOD) has completed field tests of ‘Next Generation Biometric Collection Capability (NXGBCC) hardware and software’ which will replace ‘Biometrics Automated Toolset-Army (BAT-A)’ which has been used in combat zones since the past 20 years. It says that ‘NXGBCC has been designed to support access control, identify persons of interest, and to provide biometric identities to detainee and intelligence systems.’
• UK: UK school found using facial recognition on children without consent: Last week, Chelmer Valley High School was reprimanded by the UK Information Commissioner’s Office (ICO) for using facial recognition technology (FRT) for ‘cashless canteen payments without proper consent’. The school failed to conduct a required data protection impact assessment (DPIA) and used “assumed consent,” which was deemed inadequate. The ICO emphasized that ‘proper risk assessments’ and explicit consent are crucial. The school has since started obtaining explicit consent and conducting a DPIA to rectify the situation. This case underscores the importance of complying with data protection laws and consulting stakeholders before implementing new technologies.
• Argentina: Argentina’s plan to fight crime with AI draws concerns from rights groups: Argentina has launched the Artificial Intelligence Unit Applied to Security (UIAAS) to use AI for crime investigation and security, including facial recognition, drone surveillance, and social media analysis. This move has raised concerns among civil rights groups due to the country’s history of privacy issues and misuse of surveillance technologies. Previous problems include ‘wrongful arrests’ and tracking of activists. Despite these concerns, the Argentine government is promoting a relaxed regulatory environment to attract tech companies and position itself as ‘the world’s fourth AI hub.’
• US: Meta to pay $1.4 billion to settle Texas facial recognition data lawsuit: Reuters reports that Meta Platforms has agreed to pay ‘$1.4 billion’ to settle a lawsuit with Texas over the illegal use of facial-recognition technology to collect ‘biometric data without consent.’ This settlement, said to be ‘the largest ever by any single state’, resolves a lawsuit filed in 2022 under Texas’ biometric privacy law. Meta denies wrongdoing but has previously settled a similar case in Illinois for ‘$650 million’. Texas Attorney General Ken Paxton praised the settlement and said ‘the settlement marks the state’s commitment to standing up to the world’s biggest technology companies and holding them accountable for breaking the law and violating Texans’ privacy rights.’
• India: Privacy concerns over facial-recognition system at Indira Canteens: The Deccan Herald reports on privacy concerns about the use of a facial-recognition system at Indira Canteens in Bengaluru, Karnataka. While the system aims to enhance service efficiency and prevent misuse, it has sparked fears regarding personal privacy and data security. Critics worry that the technology might infringe on privacy rights and could be misused or inadequately protected. ‘Shweta Mohandas, a researcher with the Centre for Internet and Society, highlighted concerns about compliance with the Digital Personal Data Protection Act. “The bare minimum requirement of the act is notice and consent,” she said, questioning whether not consenting to use facial recognition would result in denial of food.’

AI, algorithms and autonomy in the wider world

• International: Competition authorities’ intent to share insights on AI policy a positive step: On July 23, 2024, major competition authorities from the EU, UK, and US issued a joint statement signaling their intent to collaborate on regulating generative AI. ARTICLE 19 welcomed this move, noting it supports fair and competitive digital markets and addresses concerns about market concentration. The statement emphasizes principles like ‘fairness, interoperability, and choice.’ ARTICLE 19, represented by Dr. Simonetta Vezzoso, views this as a crucial step to prevent monopolistic control and ensure a competitive AI landscape. The hope is that this commitment will lead to effective regulatory actions and prevent potential harms from concentrated power in AI development.
• US: Silicon Valley’s Trillion-Dollar Leap Of Faith: This article discusses the immense investments in AI by tech companies and the skepticism surrounding their returns. Despite the hype that AI will revolutionize industries and generate vast economic benefits, there are concerns that the spending may be excessive and ‘very premature’. Companies are investing billions to develop and scale AI, believing that its transformative potential justifies the costs. Matteo Wong, writing for the Atlantic says ‘amid the hype, it’s important to remember that this future is not guaranteed. Many of the productivity gains expected from AI could be both greatly overestimated and very premature.’
• EU/South Africa: EU AI Act has implications for SA-based firms: The EU AI Act is creating a global impact, including in South Africa. For South African firms handling significant data or deploying AI systems in the EU, this legislation introduces increased scrutiny and compliance requirements. Essentially and of interest, the EU AI Act affects any company involved with AI systems in the EU, irrespective of its location, much like the GDPR does for personal data processing.
• International: Mind the AI Divide: Shaping a Global Perspective on the Future of Work: Last month, The UN International Labour Organization,  published a report, “Mind the AI Divide,”. The report addresses the uneven adoption of AI, which is exacerbating global inequalities and threatening social justice. It highlights that disparities in digital infrastructure, technology access, and education are widening economic and social divides, especially within less developed countries. The report emphasizes the need for targeted efforts to bridge the digital divide to realize AI’s potential for sustainable development and poverty reduction.