News Briefing 7-18 October 2024

Autonomy in weapons systems, and AI in security and defence

• International: Palmer Luckey: Every country needs a ‘warrior class’ excited to enact ‘violence on others in pursuit of good aims’: In an interview at Pepperdine University in California last month, Palmer Luckey, CEO of weapons company Anduril, insisted that there is ‘a shadow campaign being waged in the United Nations right now by many of our adversaries’ to trick Western countries into not aggressively pursuing AI, adding that ‘[Our adversaries] use phrases that sound really good in a sound bite: ‘Well can’t you agree that a robot should never be able to decide who lives and dies’. Anduril has developed a number of weapons systems with autonomous capabilities, including the recently launched Bolt-M ‘autonomous air vehicle’ which the company claims ‘delivers on-demand precision firepower against static or moving ground-based targets.’ The Bolt-M is covered further below. Previous ADR news briefing coverage of Anduril can be found here. 
• Anduril Launches Compact Bolt & Bolt-M Autonomous Drones for Precision Firepower and Reconnaissance: As noted in the item above, Anduril has recently launched the Bolt-M ‘autonomous air vehicle’, which the company claims has ‘advanced onboard computer vision and machine learning software’, enabling ‘operators to follow any target from custom standoff positions and maintain track custody through occlusion’, along with the ability to carry out ‘precision strike from any angle’. 
• International: Shield AI’s founder on death, drones in Ukraine, and the AI weapon ‘no one wants’: The CEO of Shield AI, another US-based weapons company which primarily develops AI to make air-based systems autonomous, recently stated in an interview that Ukraine ‘has been a great laboratory’. Previous ADR news briefing coverage of Shield AI can be found here. 
• UK: Rise of the robots: AI to shape UK defense review: Politico reports that the UK Ministry of Defence is using an AI system built by Palantir ‘to sift through submissions for a comprehensive review of the nation’s defense capabilities’, and notes concerns that ‘using AI for such a crucial review could lead to key submissions being overlooked.’ For previous ADR news briefing coverage of Palantir- whose CEO is a proponent of autonomous weapons systems and adoption of military AI more broadly – see here. 
• International: A new military-industrial complex: How tech bros are hyping AI’s role in war: Related to the stories above, the Bulletin of Atomic Scientists carries an article which notes that ‘The current debate on military AI is largely driven by “tech bros” and other entrepreneurs who stand to profit immensely from militaries’ uptake of AI-enabled capabilities.’
• US: Prototypes Arrive for US Army’s Robotic Combat Vehicle Competition: Arms company General Dynamics recently announced that it has ‘submitted two prototypes of its TRX (Tracked Robot 10-ton) robotic vehicle, featuring a hybrid-electric propulsion system and an artificial intelligence-enhanced design’ to the U.S. Army.
• Ukraine/International: Opinion: The Geneva Conventions – Now It’s Time to Introduce an AI Weapons Protocol:  Steve Brown, a former British Army Ammunition Specialist and Bomb Disposal Officer and now a senior writer and English language editor with the Kyiv Post, argues that ‘The “powers that be” should begin the undoubtedly lengthy process of devising and adding a protocol to the Geneva Conventions that will enshrine the control of the threat posed by AI on the battlefield into international humanitarian law’. Brown observes that ‘The rapid advancements in AI and drone technology have accelerated as a result of expediency on the battlefield in Ukraine’, and notes that this year’s Nobel Laureate in Physics, Geoffrey Hinton, has raised many concerns around autonomous weapons. 
• International: Wars are won with code, says German military AI maker: Co-founder and co-chief executive officer of Defense company, Helsing, Gundbert Scherf, said in an interview with POLITICO that ‘defense is turning more and more into a software problem.’ The company has defense contracts with British, German, French, Estonian and Ukrainian governments, and ‘processes millions of data from sensors and weapon systems of European militaries’, which Helsing claims will ‘enable “faster and better decisions” by humans and increase the lethality of weapons.’ See here for previous news briefing coverage of Helsing.

AI, algorithms and autonomy in the wider world

• International: Artificial intelligence: rooting out bias and stereotypes: As AI becomes increasingly prevalent, there are growing concerns about its potential to reinforce stereotypes and gender bias. During the recent UN General Assembly’s High-Level Week, discussions highlighted the stark underrepresentation of women in the tech industry, particularly in AI roles. Mita Hosali of the UN noted that only about ‘18% of AI researchers are women’, and hiring rates for women in tech are declining. Experts emphasized the need to address bias in AI systems, which can have real-world implications. Amandeep Singh Gill, the UN Secretary-General’s Envoy on Technology, stressed the importance of bridging the AI divide, ensuring that local researchers have access to necessary data and promoting international cooperation. 
• US: Port workers’ next demand: Stop robots from taking our jobs: The article discusses the concerns of port workers, particularly those in the International Longshoremen’s Association (ILA), regarding job security amidst rising automation. After a successful strike that secured a wage increase, union leader Harold Daggett is pushing for contract language that prohibits automation without union consent. While US ports lag in automation compared to their global counterparts, analysts suggest that resistance may be futile due to economic pressures. The piece highlights the ongoing struggle between technological advancement and job preservation for dockworkers.
• International: This Prompt Can Make an AI Chatbot Identify and Extract Personal Details From Your Chats: This piece in Wired reports on eight researchers behind testing a new attack method on two LLMs, LeChat by Mistral AI and Chinese chatbot ChatGLM, which allowed them to stealthily retrieve personal data of users. 

Facial recognition, biometric identification, surveillance

• Brazil: Brazilian groups call for ban on facial recognition: This piece by Biometric Update covers a report titled ‘Surveillance Through Opaque Lenses: mapping the transparency and accountability of facial recognition projects in Brazil’ which ‘seeks to take a critical look at the mechanisms of public transparency in the use of facial recognition cameras in public security in all five regions of Brazil.’ The report highlights Brazil operating ‘facial recognition for law enforcement and public security without meeting minimum standards of transparency.’
• International: NYT reporter Kashmir Hill discusses the privacy implications of facial recognition technology: New York Times reporter Kashmir Hill spoke at the University of Michigan about privacy risks associated with facial recognition technology, especially focusing on companies like Clearview AI. Hill discussed how Clearview AI collected billions of online photos to create a facial recognition tool used by law enforcement, raising concerns about the misuse of personal data. She emphasized the potential dangers of facial recognition in policing, as seen in cases like Robert Williams, who was wrongfully arrested due to a facial recognition error. Hill highlighted the need for stronger privacy protections in the US pointing to Illinois’ privacy law as a good model. While Europe has more robust privacy laws, the US. still lacks comprehensive federal regulations.
• International: Seeking the biometric bill of rights: In this piece,  Katina Michael, a professor in the School for the Future of Innovation in Society and School of Computing and Augmented Intelligence at Arizona State University calls for stronger protections of sensitive personal information such as DNA, facial recognition, and fingerprints. She warns that such data is vulnerable to misuse by hackers and companies, with limited legal protections. Michael advocates for a biometric bill of rights, ensuring individuals have control over their data, including rights to have it deleted and to know how it’s used. She emphasizes the urgency of addressing these issues as cybercrime and threats like deepfakes increase.
• Germany: Germany Rushes to Expand Biometric Surveillance: The article discusses Germany’s recent push for a new “security package” that expands biometric surveillance. Despite Germany’s reputation for strong privacy protections due to its historical context, this package introduces measures that could infringe on fundamental rights.
• US: Police seldom disclose use of facial recognition despite false arrests: This investigative piece by The Washington Post analyses records documenting police use of facial recognition in more than 1,000 criminal investigations over the past four years. Though ‘prosecutors are required to inform defendants about any information that would help prove their innocence, reduce their sentence or hurt the credibility of a witness testifying against them,’ the report highlights that the police fail to inform the accused of the role played by facial recognition. 

Research & Reports 

• International: Racial discrimination and emerging digital technologies: a human rights analysis: Published in 2020, this UN Human Rights Council report from the Special Rapporteur on contemporary forms of racism and racial discrimination, which highlights the urgent need to address racial discrimination in the context of digital technologies. It calls for a collaborative approach involving governments, corporations, civil society, and international organizations to ensure that technological advancements do not reinforce or exacerbate racial inequalities.