Over the last month, the number of Large Language Model (LLM) penetration testing and orchestration frameworks has exploded. Just a few weeks ago, ARTEX — an open source agentic penetration testing system that automates offensive security workflows — was the only framework available. In just the last three days, we have seen over 50 framework families attempt to scan Cloudflare customers globally.
The number of Internet Protocol (IP) addresses seen running these frameworks has grown into the thousands and continues to increase daily. The proliferation of these frameworks has significantly changed the status quo in cyber defense: defenders are struggling to keep up with the volumetric nature of these autonomously guided attacks. ARTEX was taken down by its creator Autumn-27 on October 8.
(“The malicious exploitation of this tool runs entirely counter to its original intent,” they wrote in Chinese. ) But in the month since, the ecosystem has exploded with new fragmented versions of it. There are now over 12,000 repositories, with several hundred unique forks containing lineage from the original repository.
Additionally, hundreds of other similar agentic orchestration frameworks have been created since October 8. Daily requests of AI penetration testing (pentesting) and exploitation tools has grown by 30x since October 1. These tools have also begun chaining stolen customer credentials in some of the more complex attacks, such as lateral movement chaining throughout an environment.
We expect these frameworks to continue to proliferate, and every organization should be prepared to address this threat. To counter LLM-based pentest frameworks, we recommend that all organizations adopt what we call the Modern Security Model, which has six key pillars: Fortify the front gate : Implement a Web Application Firewall (WAF) with advanced rules such as rate limits.
Security station in the foyer : Anomaly detection on all web applications your customers access. Identify phony visitors : Implement Bot Defense. Remove the back door : Implement Identity-Gated Access for all high-risk external and internal web applications.
Inspect deliveries: Enforce validated schemas and explicit security controls. Neighborhood watch : Implement Threat Intelligence to gain visibility into attacks. What is an LLM penetration and exploitation orchestrator?
Open-source pentesting frameworks are able to orchestrate AI agents to carry out stages of the traditional Cyber Kill Chain at scale. Tools like ARTEX have the ability to quickly plug in existing Model Context Protocol (MCP) and connect traditional standalone pentesting tools in novel ways. Some even allow operator direction on what type of task to accomplish.
These tools can map and enumerate environments, both from the inside and outside. ARTEX and its forks map assets and environments onto a graph, allowing multiple independent parallel agents to enumerate, verify, prototype, and attack at the speed of inference. These tools turn compromises into a hill-climbable problem, which agents are specifically good at.
The AI LLM pentesting tools that currently exist We are currently tracking 50+ AI LLM agents. The table below shows the top five LLM penetration and exploitation orchestrators. Each is extremely popular (1k+ GitHub stars) and fully autonomous.
The fully autonomous nature of these agents represents a doctrinal shift in offensive cyber capabilities: Human skills are no longer the limit: A harness performs every step of the OODA loop by observing the environment, orienting itself to prioritize valuable and/or vulnerable targets, decide on an action, and act autonomously. Speed increase: Having multiple parallel agents working towards a goal is a large shift compared to single operator or multiple operator augmented by AI.
Scale: A single operator can target multiple organizations at the same time, greatly increasing their blast radius. Small mistakes are easily discovered: Autonomous frameworks relentlessly and systematically test exposed services, including their configurations. Architecture Planning model Built-in capabilities/execution environment Post-exploitation capabilities ARTEX (and forks) Planner and parallel agents Graph of goals, leads, and findings is linked to separate asset graph of target Directly on host or container with network/root access, can install/develop tools as needed Nothing built-in, requires user instruction LuaN1aoAgent Supervisor agent, planner agent, and generic executor agents Three graphs, with goals/tasks/blockers, evidence/hypothesises/exploits, and target/local Container with network/user access, can develop tools as needed and install user packages Built-in SSH and Chisel for tunneling and attempted reconnection after disruption Decepticon Orchestrator agent dispatching specialist sub-agents Graph of objectives with prerequisites Kali container running as root, can develop tools as needed Dedicated post-access, Active Directory and cloud pivoting skills Cairn Dispatcher orchestrating work to existing AI harnesses (Codex, Claude Code, Pi) Graph of facts and leads; new nodes trigger dispatcher Kali container with root access or directly on host, can develop tools as needed Prompt guidance and Kali tools PentestGPT-Agent Supervisor agent and one generic executor agent Task list with dependencies, goals, and evidence Run on host, can develop tools as needed Nothing built-in, requires user instruction Observations from the world's largest network As of November 12, ARTEX and its forks make up over 60% of currently known AI penetration testing orchestrators.
The charts below help us understand how adoption of these tools has changed over the past six weeks. Since the removal of the original ARTEX repository on October 8, traffic has grown more than tenfold, and has increased more than 30x since October 1.
Originally published at blog.cloudflare.com
