Lakera launches framework for testing LLM security

Check Point Software subsidiary Lakera has released an open-source framework for testing the security of the large language models (LLMs) powering AI agents.

The backbone broker benchmark is designed to evaluate the security of LLMs based around the concept of threat snapshots, which aim to zoom into the critical points where vulnerabilities in LLMs are most likely to appear. By testing models at these moments, developers and model providers can evaluate how well their systems stand up to the threats they are likely to face in the real world, without needing to model a full agent workflow.

The benchmark developed by Lakera combines 10 of these threat snapshots with a dataset of nearly 19,500 crowdsourced adversarial attacks to evaluate the susceptibility of LLMs to attacks such as system prompt exfiltration, phishing link insertion, malicious code injection, denial-of-service, and unauthorised tool calls.

Lakera co-founder and Chief Data Scientist Mateo Rojas-Carulla said the company has built the benchmark in recognition of the fact that today’s AI agents are only as secure as the LLMs that power them.

“Treat Snapshots allow us to systematically surface vulnerabilities that have until now remained hidden in complex agent workflows,” he said. “By making this benchmark open to the world, we hope to equip developers and model providers with a realistic way to measure, and improve, their security posture.”

Initial results from testing 31 popular LLMs using the benchmark demonstrate that enhanced reasoning capabilities significantly improve security, but that model size does not correlate with security performance, Lakera said. Meanwhile currently closed-source models are generally outperforming open-weight models, but the top open models are narrowing the gap.

The crowdsourced adversarial attacks were compiled using Lakera’s hacking simulator game Gandalf: Agent Breaker, which challenges users to break and exploit AI agents in realistic scenarios and the 10 GenAI applications inside the game simulate how a real-world AI agent behaves. The software was initially developed during an internal hackathon, and released to the public in 2023.

Image credit: iStock.com/dem10