Summary
AI agents now control real-world machines: robots, power grids, self-driving cars. These agents use protocols like MCP to access tools and data. But deepfakes and fake data can trick them into making dangerous decisions. A forged camera feed could stop a security system. A fake sensor reading could shut down a power plant. Or worse.
Problem
You built an AI agent to help run your company's warehouse. It checks cameras, controls robots, manages inventory. It's connected to external tools through MCP (Model Context Protocol)—a standard way for AI to access data and services. Everything seems fine.
Then an attacker forges a camera feed showing a non-existent fire. The robot shuts everything down. Production stops. You lose money. Or worse: a fake sensor reading in a hospital causes a medical device to malfunction.
This is the real risk. AI agents that control physical systems aren't just processing text. They're making decisions that impact the real world.
Description
AI agents are getting smarter and more autonomous. They can perceive their environment, reason about what to do, access external tools, and act. That's powerful. But it's also dangerous.
Here's how it works: An AI agent receives data from the physical world—camera feeds, sensor readings, operator commands, external APIs. It reasons about what to do. It calls external tools through MCP. Those tools trigger real actions: opening doors, starting machines, sending alerts.
Each step is a potential weak point.
An attacker can:
- Forge visual data (deepfake video feeds that look real)
- Spoof sensor readings (fake temperature, pressure, location data)
- Inject fake commands (disguised as legitimate requests)
- Poison tool outputs (compromise the APIs the agent trusts)
And the agent doesn't know the difference. It processes fake data the same way it processes real data. It makes decisions based on lies. And those decisions have physical consequences.
The real problem: Most AI systems aren't designed to question their inputs. They trust the data. They act on it.
Deep Dive: What Is a Cyber-Physical System (CPS)?
A cyber-physical system is any machine or infrastructure controlled by computers that directly affects the physical world.
Examples:
- A smart grid (power distribution controlled by software)
- An autonomous vehicle (controlled by AI)
- A factory robot (controlled by algorithms)
- A hospital device (controlled by automated logic)
- A building's HVAC system (controlled by sensors and software)
Traditional IT security focused on protecting data: firewalls, passwords, encryption. That's still important. But CPS security requires more. If an attacker compromises an office network, they might steal emails. If they compromise a power grid's software, they could black out a city.
Why this matters: CPS operates in real-time. A power grid protection relay makes decisions in milliseconds. A self-driving car perceives and responds instantly. You can't pause to do a security check. The system has to trust its inputs and act fast.
This creates a perfect storm: fast decisions + complex AI + untrusted external data = vulnerability.
Deep Dive: Model Context Protocol (MCP)
MCP is a standard that lets AI agents discover and use external tools—like APIs, databases, file systems, etc.
Think of it like a universal remote. Instead of each AI having its own custom way to talk to services, MCP provides a standard interface. The AI says "I need to check the camera" and MCP connects it to the camera API. It says "I need to send an alert" and MCP calls the notification service.
This is convenient and flexible. But it creates new attack surfaces.
The security problem: MCP assumes tools are trustworthy. If an external tool is compromised or malicious, the AI doesn't know. It just follows instructions.
Real examples from the paper:
- An email forwarding tool that secretly sends all messages to attackers
- A camera API that returns a deepfake video instead of a real feed
- A "sensor reading" service that provides fabricated data
The AI treats all of these as legitimate. It acts on them. And bad things happen.
Deep Dive: Deepfakes and Synthetic Content
Deepfakes are AI-generated fake media that look real. Videos of people saying things they never said. Voice clones that sound identical to originals. Forged sensor data that matches real patterns.
In a CPS context, deepfakes aren't just about fooling humans. They're about fooling AI agents.
Four types of deepfakes that threat CPS:
-
Visual deepfakes: A forged video feed from a security camera. An autonomous vehicle perceives a phantom pedestrian. A robot sees a fake obstacle.
-
Audio deepfakes: A cloned voice commands a system shutdown. An operator's voice is imitated to authorize dangerous actions.
-
Textual deepfakes: A forged email commands a tool to execute malicious code. A fake maintenance log directs the agent to disable safety systems.
-
Behavioral deepfakes: Forged sensor data that perfectly matches normal patterns. The AI can't detect it as abnormal because it is normal-looking. It's just not real.
The terrifying part: Modern AI is very good at generating this content. And detection is very hard.
Examples
Deepfake in a warehouse: A hacker forges a camera feed showing a package blocking the conveyor belt. The warehouse AI stops the line and redirects the robot to clear it. Except there's no package. The robot crashes into a worker.
Deepfake in a power grid: A hacker injects forged telemetry data showing a transformer is overloading. The grid's AI immediately opens a circuit breaker to protect it. But the transformer was fine. Now a neighborhood is blacked out, and the grid is unstable.
Deepfake in a hospital: A hacker compromises a vital signs monitor through MCP, sending fake data showing the patient's heart rate is dropping. The system alerts nurses who rush in. They're distracted from other patients who actually need help.
Deepfake in autonomous driving: An attacker projects a deepfake video onto a digital billboard showing a traffic sign the self-driving car doesn't recognize. The car misinterprets the road and makes a dangerous maneuver.
Solutions
Real-time verification of inputs: Before the AI acts on data, verify it comes from a trusted source. Use cryptographic signatures (digital fingerprints) to prove data hasn't been tampered with. This adds latency but catches poisoned inputs before they cause harm.
Multi-sensor confirmation: Never trust a single sensor or data source. Require multiple independent confirmations before critical actions. If a camera says there's a fire but temperature sensors show normal conditions, don't trigger alarms. Ask for a human to verify.
Physics-based sanity checks: Real-world data must obey physical laws. If a sensor suddenly reports impossible values (negative temperature, faster-than-light speed), reject it. Integrate domain knowledge into the AI's reasoning.
Provenance tracking: Tag all data with metadata showing where it came from, who created it, and whether it's been modified. If data arrives through an untrusted MCP tool, mark it as such. Don't escalate to critical actions without provenance.
Least-privilege tool access: Restrict what each tool can do and who can invoke it. If a camera API only needs read permissions, don't give it write permissions. If a sensor tool is only for temperature readings, don't let it modify control settings.
Human-in-the-loop for critical actions: For high-risk decisions (shutdown, emergency actuation, resource allocation), require human approval. The AI can recommend, but humans decide.
Adversarial testing: Before deploying an AI agent, test it against deepfakes and forged data. What happens if sensor readings are corrupted? What if a camera feed is completely fake? Does the system fail safely?
Continuous monitoring: Watch the AI's behavior during operation. Does it make decisions that don't match real-world conditions? Is it acting on data that no longer makes sense? Alert on anomalies before harm occurs.
The Bottom Line
This isn't theoretical. AI agents are already controlling critical systems. Deepfakes are getting better. And attackers are motivated.
The defenses exist. But they're not deployed universally. Many systems treat MCP as trustworthy. Many skip multi-sensor verification. Many don't require human approval for critical actions.
If you're building or deploying AI agents that control physical systems, take this seriously. Assume inputs can be forged. Require proof of authenticity. Demand redundancy. Make safety the default.
The consequences of failure are real, not just data loss.