News Briefing 9-20 September 2024

Autonomy in weapons systems, and AI in security and defence

• International: Reaching Critical Will First Committee Briefing Book 2024: The 2024 edition of Reaching Critical Will’s UNGA First Committee Briefing book is out now. The chapter on autonomous weapons, by Stop Killer Robots, calls on states to ‘promote, co-sponsor and vote in support of a resolution on autonomous weapons that broadens the international debate on this issue’, and to ‘recognise in national statements that autonomy in weapon systems presents serious challenges to ethics, law, security, and the protection of civilians.’ 
• International: Governing AI for Humanity: The United Nations Secretary-General’s High-level Advisory Body on AI  has released its final report titled ‘Governing AI for Humanity.’ The report mentions autonomous weapons a number of times, and says that ‘on legal and moral grounds, kill decisions should not be automated through AI’ while urging member states to follow through on the UN Secretary-General’s call to complete the negotiation of a new international instrument on autonomous weapon systems by 2026. 
• International: Towards Human Rights — Broadening perception of risks posed by autonomous weapon systems: This blog piece written by ADR researchers Sai Bourothu with Gugu Dube highlights the importance of human rights and international human rights law in conversations on autonomous weapon systems. The piece notes that ‘it is crucial that the conversation on prohibiting and regulating these weapon systems and mitigating their impacts take cognizance of the critical role of human rights, and that international human rights law be central to any international discussions on AWS and the negotiation of a legally binding instrument.’ 
• US: Emails show cops knew buying drones broke state law. They did it anyway: The San Francisco Standard reports on legal compliance issues in the procurement and use of drones by the San Francisco Police Department. This piece brings to light emails by Asja Steeves, SFPD policy division manager which cautioned the department on compliance issues, and suggested holding off on procurement till there was more clarity. 
• US: Anduril unveils new cruise-missile like weapon, plus voice-controlled drones: Anduril has unveiled its new “Barracuda” family of autonomous air vehicles, The drones, resembling cruise missiles, come in three variants with increasing size, payload, and range, the largest capable of traveling ‘over 500 nautical miles’. The development of such autonomous systems raises concerns about the level of autonomy permissible in lethal decision-making, with Anduril emphasizing adherence to Department of Defense policies that allow for adjustable levels of human oversight.
• International: HD Hyundai showcases USV technology at REAIM Summit: At the REAIM Summit 2024 in Seoul, HD Hyundai showcased its unmanned surface vehicle (USV) technology, highlighting the ‘Tenebris’, which is being developed in collaboration with Palantir Technologies. The Tenebris, designed for reconnaissance near enemy lines, features advanced AI for autonomous navigation and mission capabilities. The 14-ton vessel is expected to be completed by 2026.
• US: US Air Force tests the use of autonomous flying capabilities in austere, contested scenarios: The US Air Force has successfully tested autonomous flying capabilities during logistics missions at the AGILE FLAG 24-3 exercise in California. Using technologies from Joby and Reliable Robotics, the exercise involved a remotely controlled Cessna 208B Grand Caravan that completed a nearly 2,000 km (1,150 miles) flight, delivering 5,500 kg (1,200 pounds) of cargo between military bases. This trial is part of the Autonomy Prime program, showcasing the potential for autonomous systems in contested environments.
• US: Oracle, Rafael Partner to Provide Cloud-Based AI to US Military: This piece by ExecutiveBiz reports on a partnership between Oracle, Rafael advanced defense systems and the US military. Under this partnership, the US military can utilize Rafael’s AI-driven imagery intelligence system and sensor-to-shooter system through Oracle Government Cloud, Oracle Cloud Isolated Region, Oracle National Security Regions and Oracle Roving Edge Infrastructure offerings.
• International: Threat of ‘killer robots’ in Ukraine increasingly realistic, warns expert: Speaking with Yahoo News, Stop Killer Robots vice-chair, Peter Asaro noted that ‘there are grey areas around what makes a “killer robot” and that even systems where humans still nominally make the decisions could cause issues.’ Asaro underscores that ‘there is a real possibility for a treaty’ on autonomous weapons, with the global public ‘in broad agreement that we really do not want to live in a world where machines are given the authority to decide who lives and dies, and that we should ban and regulate these things.’ 

AI, algorithms and autonomy in the wider world

• International: Beware of Big Tech Steering AI Regulation: The article by Peter G. Kirchschläger argues that allowing Big Tech companies like Meta, Google, Amazon, Apple, and Microsoft to help shape global AI and digital regulations is a mistake. These companies have historically prioritized profits over user safety, contributing to privacy violations, disinformation, mental health issues, and environmental damage. Kirchschläger warns that their involvement in regulatory processes could further entrench their power and harm society. Instead, the article advocates for input from independent experts and civil-society organizations to ensure regulations serve the public interest and promote global equality.
• International: Now we know the risks of AI, here’s how policy frameworks can mitigate them: The article discusses the increasing recognition of AI risks and the need for policy frameworks to address them. It emphasizes that while AI development has been driven by the private sector, ‘regulation is lagging behind.’ Various tools, such as technical, socio-technical, and manual approaches, are needed to mitigate these risks. The article calls for combining different risk mitigation methods to ensure AI is used safely and responsibly, balancing its potential benefits with the need to prevent harmful consequences. Immediate action is urged to manage the rapid adoption and impact of AI technologies.
• International: Pope warns of AI dangers, urges fair wages for migrants on Singapore visit: Pope Francis, during his visit to Singapore, warned about the dangers of AI, emphasizing that ‘it should be used to bring people closer together and to promote understanding and solidarity within society.’ He also called for ‘fair wages’ and better treatment of migrant workers, acknowledging their significant contributions to society. Singapore was the final stop on the pope’s 12-day Asia Pacific tour, which included visits to Indonesia, Papua New Guinea, and East Timor.

Facial recognition, biometric identification, surveillance

• Russia: ‘Significant’ Risks As Facial Recognition In Russia’s Subways Goes Regional: Russia is expanding its facial recognition payment system, ‘Face Pay’, from Moscow to six cities, raising concerns among human rights advocates. While officials claim it offers convenience and security, critics warn it poses significant risks to civil liberties, as similar technologies have been used to detain protest participants. Legal experts caution that the system may lead to self-censorship among citizens fearful of surveillance. Despite claims of user choice, the potential for increased state control and monitoring remains a significant concern.
• International: New research explores AI manipulation attacks on face biometric systems: Biometric Update reports on two research papers which ‘address sophisticated fraud attempts on biometric systems using AI, in one case to carry out morph attacks, and in the other using synthetic data to carry out template inversion attacks.’ 
• EU: Surveilling Europe’s edges: when digitalisation means dehumanisation: Caterina Rodelli, from Access Now, writes on the use of automated decisions and systems at EU borders, and their impacts on further dehumanising migrant populations. She notes that the  ‘use of automated decision-making and surveillance technology perpetuates structural oppression, discrimination, and exclusion of migrant and racialised people.’ To remedy this, she urges that EU policy makers ban the use of inherently harmful AI systems, and refocus migration policies on ‘principles of safety, protection, and justice.’