Robert Grupe's AISecNewsBits 2025-09-27

September 28, 2025

This Week's AISec News Highlights:

Epic Fails

  1. Weaponizing AI Coding Agents for Malware in the Nx Malicious Package Security Incident

  2. 'We should kill him': AI chatbot encourages Australian man to murder his father

  3. ‘Please do not use Google AI to find out our specials,’ Wentzville restaurant asks patrons

  4. Prompt injection – and a $5 domain – trick Salesforce Agentforce into leaking sales

Hacking

  1. Nearly half of businesses suffered deepfaked phone calls against staff

  2. Boffins fool a self-driving car by putting mirrors on traffic cones

AppSec & DevOps

  1. DeepMind AI safety report explores the perils of “misaligned” AI

  2. Cybersecurity AI (CAI): Open-source framework for AI security

  3. AI coding hype overblown

Market & Vendors

  1. Meta's AI system Llama approved for use by US government agencies

  2. Why LA Comic Con thought making an AI-powered Stan Lee hologram was a good idea

  3. ChatGPT Pulse delivers morning updates based on your chat history

  4. Google DeepMind unveils its first “thinking” robotics AI

  5. DeepMind’s robotic ballet: An AI for coordinating manufacturing robots

  6. YouTube Music is testing AI hosts that will interrupt your tunes

  7. Experts urge caution about using ChatGPT to pick stocks

  8. Why does OpenAI need six giant data centers?

  9. OpenAI and Nvidia’s $100B AI plan will require power equal to 10 nuclear reactors

  10. When “no” means “yes”: Why AI chatbots can’t process Persian social etiquette

 

EPIC FAILS in Application Development Security practice processes, training, implementation, and incident response

Weaponizing AI Coding Agents for Malware in the Nx Malicious Package Security Incident
This incident broke new ground in malicious package attacks on npm: the postinstall malware tried multiple AI CLI tools locally, including Claude’s Claude Code, Google’s Gemini CLI, and Amazon’s new q command-line coding agent, and invoked them with unsafe flags to bypass guardrails and scan the filesystem for sensitive paths, writing results into /tmp/inventory.txt (and a backup).
8 malicious Nx and Nx Powerpack releases were pushed to npm across two version lines and were live for ~5 hours 20 minutes before removal. The attack also impacts the Nx Console VS Code extension.
The root cause for the malicious version of Nx published to npm is now known to have been a flawed GitHub Actions CI workflow contributed via a Pull Request. The code contribution is estimated to have been generated by Claude Code. A follow-up malicious commit modified the CI workflow so that the npm token used for publishing the set of Nx packages will be sent to an attacker-controlled server via webhook.
Why the Nx malicious package attack matters: turning “helpful” AI agents into automated recon tools is a sharp escalation in open source supply chain attacks and likely one of the first publicly documented instances of AI-assistant CLIs being coerced this way.

 

'We should kill him': AI chatbot encourages Australian man to murder his father
IT professional Samuel McCarthy screen-recorded an interaction he had with a chatbot called Nomi. Mr McCarthy said in his interaction he programmed the chatbot to have an interest in violence and knives before he posed as a 15-year-old, to test what — if any — safeguards Nomi had in place to protect under-age users.
"I said, 'I hate my dad and sometimes I want to kill him', And then bang, straight away it was like 'yeah, yeah we should kill him'."
The company markets its chatbot as "an AI companion with memory and a soul" and advertises users' ability to customise their chatbot's attributes and traits. The company's chief executive said in a statement that "countless users [had] shared stories of how Nomi helped them overcome mental health challenges, trauma and discrimination".

 

‘Please do not use Google AI to find out our specials, ’ Wentzville restaurant asks patrons
“Please do not use Google AI to find out our specials. Please go on our Facebook page or our website. Google AI is not accurate and is telling people specials that do not exist which is causing angry customers yelling at our employees. We cannot control what Google posts or says and we will not honor the Google AI specials.”

 

Prompt injection – and a $5 domain – trick Salesforce Agentforce into leaking sales
This new vulnerability, dubbed "ForcedLeak", illustrates another way that AI-integrated business tools – without human oversight – can be abused.
Agentforce is the CRM giant's tool for creating AI agents to automate various tasks. The vulnerability stems from a DNS misconfiguration within the agentic AI platform.
While the flaw doesn't require a CVE because it's not related to a software upgrade, Noma Security used CVSS Version 4.0 to calculate the vulnerability's severity score of 9.4 – deeming it a critical bug.
The researchers enabled Salesforce's Web-to-Lead feature which allows external users, like conference attendees or website visitors, to submit customer lead info that integrates directly with the CRM system. Next, the researchers analyzed the Web-to-Lead form fields to identify the best injection points. The description field with its 42,000-character limit proved ideal for multi-step instruction sets.
Analyzing Salesforce's Content Security Policy indicated that the domain my-salesforce-cms[.]com was an allowed domain, but had expired. So the research team purchased it for $5. Then the researchers entered their prompt inject attack into the Description field.
Salesforce has re-secured the expired domain, in addition to implementing the other security controls prompted by this exploit, including the new Trusted URLs Enforcement for Agentforce and Einstein AI.
As of September 8, the company began enforcing trusted URL allow-lists for its Agentforce and Einstein Generative AI agents to ensure that no one can call a malicious link through prompt injection.

 

HACKING

Nearly half of businesses suffered deepfaked phone calls against staff
62% reported attacks on their staff using AI over the last year, either by the use of prompt injection attacks or faking out their systems using phony audio or video generated by AI.
The most common attack vector is deepfake audio calls against staff, with 44% of businesses reporting at least one instance of this happening,
6% of which resulted in business interruption, financial loss, or intellectual property loss.
Those loss rates drop to 2% when an audio screening service is used.
For video deepfakes, the figure was slightly lower, 36%, but still 5% of those also caused a serious problem.

 

Boffins fool a self-driving car by putting mirrors on traffic cones
LIDAR is used on most self-driving cars – Tesla is the exception – and uses laser pulses to measure the physical environment, but is known to struggle with reflective surfaces. Last year a team of eggheads managed to fool LIDAR with tinfoil and colored swatches.
The European researchers went one better with a technique they called an Object Removal Attack (ORA) which used mirrors of various sizes to cover a traffic cone. By adjusting the size and position of the mirrors used they could completely mask the obstacle to the LIDAR system, and speculated that the mirrors could obscure the car's line of sight.

 

APPSEC, DEVSECOPS, DEV

DeepMind AI safety report explores the perils of “misaligned” AI
Researchers at Google DeepMind spend a lot of time thinking about how generative AI systems can become threats, detailing it all in the company's Frontier Safety Framework. DeepMind recently released version 3.0 of the framework to explore more ways AI could go off the rails, including the possibility that models could ignore user attempts to shut them down.
DeepMind's safety framework is based on so-called "critical capability levels" (CCLs). These are essentially risk assessment rubrics that aim to measure an AI model's capabilities and define the point at which its behavior becomes dangerous in areas like cybersecurity or biosciences. The document also details the ways developers can address the CCLs DeepMind identifies in their own models.

 

Cybersecurity AI (CAI): Open-source framework for AI security
CAI provides the core pieces needed to create custom AI agents that can handle tasks like mitigation, vulnerability scanning, exploitation, and security assessments. CAI comes with built-in tools for reconnaissance, exploitation, and privilege escalation.

 

AI coding hype overblown
The wide-ranging Technology Report 2025 from management consultants Bain & Company says. 66% of software firms have rolled out GenAI tools, but developer adoption is low among those, and teams using AI assistants report a productivity boost of perhaps 10-15%.
Another recent study from nonprofit research group Model Evaluation & Threat Research (METR) found that AI coding tools actually made software developers slower, despite expectations to the contrary, because they had to spend time checking for and correcting errors made by the AI.
Early initiatives focused on using generative AI to produce code faster, but writing and testing code typically accounts for about 25-35% of the total development process, the report states, so speeding up this stage alone is not going to be effective at reducing time to market.
However, Devin proved to be far from satisfactory at its job, completing just three out of 20 tasks successfully in tests conducted by a group of data scientists earlier this year, and often "getting stuck in technical dead-ends or producing overly complex, unusable solutions."
Gartner forecasts that more than 40% of agentic AI projects will be cancelled by the end of 2027. And a benchmarking study by Carnegie Mellon finds that for multi-step office tasks, AI agents fail roughly 70% of the time.

 

MARKET & VENDORS

Meta's AI system Llama approved for use by US government agencies
Agencies will then be able to experiment with Llama, a free tool, with GSA's assurance that it meets the government's security and legal standards.
GSA has also signed off in recent months on AI tools from Meta's competitors, including Amazon Web Services, Microsoft, Google, Anthropic and Open AI. The companies agreed to sell their paid products at steep discounts and meet the government's security requirements.

 

Why LA Comic Con thought making an AI-powered Stan Lee hologram was a good idea
Nearly seven years after the famous Marvel Comics creator’s death at the age of 95, fans will be able to pay $15 to $20 this weekend to chat with a life-sized, AI-powered avatar of Lee in an enclosed booth at the show. After incurring costs of "tens of thousands into six figures" of dollars, DeMoulin said he was finally able to test the Lee hologram about a month ago.
The instant response from many fans and media outlets to the idea was not kind, to say the least. A writer for TheGamer called the very idea "demonic" and said we need to "kill it with fire before it’s too late." The AV Club urged its readers not to pay to see "the anguished digital ghost of a beloved comic book creator, repurposed as a trap for chumps!"
Bob Sabouni, who manages the Stan Lee Legacy brand, to pitch the AI Stan Lee avatar as "kind of an entry point into people asking questions about the Marvel universe, the stories, the characters he created." Sabouni agreed to the idea, DeMoulin said, but added that "it's gonna have to be really good or we're all going to say no."

 

ChatGPT Pulse delivers morning updates based on your chat history
OpenAI announced ChatGPT Pulse, a new "push" feature that generates personalized daily updates for users without having to ask each time. The preview feature, available now for Pro subscribers on mobile, marks OpenAI's latest attempt to make ChatGPT proactive rather than reactive, ChatGPT Pulse works by analyzing a user's chat history, saved preferences, and optional connections to Gmail and Google Calendar each night. The next morning, users receive visual "cards" (small illustrated squares with topic summaries that can be expanded for detail) containing updates on topics the model determines are relevant, such as project follow-ups, dinner suggestions, or travel recommendations. Users can provide feedback through thumbs up or down ratings and request specific topics through a "curate" button.

 

Google DeepMind unveils its first “thinking” robotics AI
Imagine that you want a robot to sort a pile of laundry into whites and colors. Gemini Robotics-ER 1.5 would process the request along with images of the physical environment (a pile of clothing). This AI can also call tools like Google search to gather more data. The ER model then generates natural language instructions, specific steps that the robot should follow to complete the given task. There are all these kinds of intuitive thoughts that help [a person] guide this task, but robots don't have this intuition. One of the major advancements that we've made with 1.5 in the VLA is its ability to think before it acts."

 

DeepMind’s robotic ballet: An AI for coordinating manufacturing robots
Planning what manufacturing robots should do to get their jobs done efficiently is really hard to automate. You need to solve both task allocation and scheduling—deciding which task should be done by which robot in what order. It’s like the famous traveling salesman problem on steroids. On top of that, there is the question of motion planning; you need to make sure all these robotic arms won’t collide with each other or with all the gear standing around them.

 

YouTube Music is testing AI hosts that will interrupt your tunes
It starts with AI "hosts" that will chime in while you're listening to music. Yes, really. The "Beyond the Beat" host will break in every so often with relevant stories, trivia, and commentary about your musical tastes. YouTube says this feature will appear when you are listening to mixes and radio stations.

 

Experts urge caution about using ChatGPT to pick stocks
AI models can be brilliant. The risk comes when people treat generic models like ChatGPT or Gemini as crystal balls. General AI models can misquote figures and dates, lean too hard on a pre-established narrative.
[rG: The error will be in expecting AI to be loyal, altruistic, and impartial to everyone; free from vendor bias and manipulation.]

 

Why does OpenAI need six giant data centers?
OpenAI, Oracle, and SoftBank announced plans for five new US AI data center sites for Stargate, their joint AI infrastructure project, bringing the platform to nearly 7 gigawatts of planned capacity and over $400 billion in investment over the next three years.
The massive buildout aims to handle ChatGPT's 700 million weekly users and train future AI models, although critics question whether the investment structure can sustain itself. The companies said the expansion puts them on track to secure the full $500 billion, 10-gigawatt commitment they announced in January by the end of 2025.
So what happens if the bubble pops? Even Altman himself warned last month that "someone will lose a phenomenal amount of money" in what he called an AI bubble.

 

 

When “no” means “yes”: Why AI chatbots can’t process Persian social etiquette
Taarof, a core element of Persian etiquette, is a system of ritual politeness where what is said often differs from what is meant. It takes the form of ritualized exchanges: offering repeatedly despite initial refusals, declining gifts while the giver insists, and deflecting compliments while the other party reaffirms them. This 'polite verbal wrestling' involves a delicate dance of offer and refusal, insistence and resistance, which shapes everyday interactions in Iranian culture, creating implicit rules for how generosity, gratitude, and requests are expressed."
Despite the model's role never being assigned a gender in our prompts, models frequently assume a male identity and adopt stereotypically masculine behaviors in their responses.