How are Attackers Exploiting AI Hallucinations in Cybersecurity (e.g., Registering Hallucinated Domains or Weaponizing AI-generated Guidance), and What Defenses Actually Reduce This Risk?

How Attackers Are Exploiting AI Hallucinations in Cybersecurity

Artificial intelligence hallucinations—instances where a model generates plausible but factually incorrect information—were historically viewed as a quality control issue. Today, threat actors have transformed these errors into an active adversarial security vector. By anticipating or discovering the mistakes AI models make, attackers are setting traps for users and automated systems that rely on AI-generated guidance.

When an AI model confidently recommends a non-existent software library, domain name, or API endpoint, attackers can register those exact assets. If a developer, employee, or autonomous AI agent follows the hallucinated guidance, they are directed straight into an attacker-controlled environment, often resulting in malware deployment or data compromise.

How Attackers Weaponize Hallucinations

Threat actors exploit the gap between AI generation and factual reality through several distinct methods of tradecraft:

• Hallucinated Domains (Phantom Squatting): AI models frequently invent URLs when asked for documentation, customer support links, or download portals. Researchers at Palo Alto Networks’ Unit 42 have documented this technique—which they call phantom squatting—where attackers register AI-hallucinated web domains to deliver phishing pages and malware. In one confirmed case, an AI hallucinated a web domain for a legitimate postal service application, and attackers registered that exact fictional domain to serve a malicious Android APK to users who followed the AI’s advice.
• Phantom Software Packages (Slopsquatting): When developers ask AI for code solutions, the model may recommend importing a highly relevant but entirely non-existent software library. Attackers monitor these common hallucinations and publish malicious code under those exact fictional names in public repositories like npm or PyPI. This technique has been named slopsquatting, and it replaces the human typo that traditional typosquatting relied on with an AI-generated hallucination instead.
• Automated Exploitation via AI Agents: With the rise of autonomous AI agents capable of browsing the web or executing code, hallucinations are no longer just a risk to human users. If an AI agent hallucinates a malicious domain and possesses the permissions to interact with it, the system can compromise itself without any human intervention.

Effective Defenses and Mitigations

Because hallucinations are an inherent limitation of current Large Language Model (LLM) architectures—one that architectural improvements can reduce but not eliminate—organizations cannot simply wait for AI vendors to patch the issue. Mitigating this risk requires a defense-in-depth approach that treats all AI output as untrusted.

• Strict Allowlists: Organizations must restrict development environments and automated agents to interact only with pre-approved domains, package repositories, and IP addresses. This prevents a system from reaching out to a newly registered, attacker-controlled domain.
• Verification Workflows: Implementing mandatory secondary validation is critical. This includes human-in-the-loop checks or automated scripts that cross-reference AI-suggested URLs and software packages against established, trusted databases before they can be accessed or downloaded.
• Browser and DNS Controls: Network-level security must be configured to block or flag Newly Registered Domains (NRDs) and domains with low reputation scores. Because attackers must register hallucinated domains after the AI generates them, these domains are typically brand new and easily caught by aggressive DNS filtering.
• Agent Tool Permissions: The principle of least privilege must be applied to AI agents. If an AI assistant does not explicitly require web browsing, file downloading, or command-line execution capabilities to perform its core function, those tools should be disabled to prevent the automated execution of hallucinated instructions.

Summary

The weaponization of AI hallucinations demonstrates how threat actors are adapting to the widespread adoption of artificial intelligence. By registering hallucinated domains through phantom squatting and publishing phantom software packages through slopsquatting, attackers are turning the inherent limitations of AI models into highly effective traps. Defending against this vector requires organizations to shift away from implicit trust in AI outputs and rely instead on strict network controls, rigorous verification workflows, and tightly restricted permissions for AI agents.