AI-enabled email accounts can be an insider threat

Barracuda has demonstrated a controlled attack that shows how attackers can weaponise an AI assistant to turn a single compromised account into fraud. The test shows that the greatest risk from a compromised AI-enabled account is how quickly an AI assistant can help attackers uncover sensitive information, identify targets, craft convincing communications, and advance an attack using access the victim already possesses.

In the controlled proof-of-concept attack, Barracuda’s red team demonstrated how a single compromised employee account can escalate into CEO compromise and wire-transfer fraud to net attackers $250k using wholly legitimate, existing processes. The controlled attack leveraged Copilot, but it applies equally to other widely available AI assistants.

How the attack unfolded

Using a compromised employee account, the attackers first asked the AI assistant to help establish persistence by creating inbox rules that hide sign-in alerts and other suspicious notifications from the user. They then used the assistant to uncover the organisation’s structure, identify high-value targets and surface relevant conversations buried within months of emails, attachments and calendar activity.

Armed with this intelligence, the attackers prompted the AI assistant to draft a highly convincing phishing message to the CEO in the employee’s natural writing style. Because the email originates from a legitimate internal account and reflects genuine business context, it is far more likely to succeed.

Once the CEO’s account is compromised through a session-token theft attack, the threat actors repeat the process, using AI to maintain persistence and uncover the most valuable financial information within the executive’s mailbox.

Why it matters

AI can transform an inbox into a searchable intelligence repository. In the proof of concept, a simple prompt asking for recent financial activity unearthed active invoices, wire transfers and approval workflows, including a pending $247,500 payment. The attackers then used the CEO’s account and the AI assistant to draft a convincing request to finance staff to change the destination bank account before the transfer is approved.

Because the request comes from the CEO’s legitimate mailbox, references a real transaction and matches the executive’s communication style, traditional email security controls have little reason to flag it as suspicious.

Attackers can also create forwarding rules to intercept confirmation messages and use the AI assistant to quickly locate and remove evidence of the fraud.

“A user’s email history is full of sensitive information and context that can be leveraged by attackers, including emails sent and received, documents shared, attachments and calendar invites. Company structure can be deduced from implied relationships in messages or directly viewed via organisational charts,” said Daniel Avulov, Senior Cybersecurity Researcher, red team at Barracuda. “The controlled attack shows how AI assistants can become unwitting malicious insiders and improve both the quality and speed of an attack. The most effective defensive approach is to recognise that attack patterns remain the same, take advantage of the telemetry that already exists, and ensure mean time to detect is as low as possible.”

As AI assistants become deeply embedded in business communications and workflows, organisations must treat AI-enabled accounts as high-value assets. Protecting identities, monitoring account compromise and securing AI-assisted access to corporate information will become an increasingly critical part of modern email and identity security strategies.

For more information, read the Barracuda blog post here.

Image credit: iStock.com/Thanadon Naksanee