By Chris Wysopal, Chief Security Evangelist & Co-Founder, Veracode
I have been going to Black Hat for more than 25 years, and I have spent many of those years on the Review Board. New technologies keep changing the shape of the systems we secure. The questions I find myself asking stay remarkably familiar.
Where does untrusted data enter? What authority does a component have? Which boundary controls a sensitive operation? How do we know that control worked?
At Black Hat USA 2026, AI agents gave those questions new urgency. Agents now interact with source code, developer tools, credentials, cloud services, internal data, and execution environments. Several of the talks that stayed with me examined the places where an agent’s reasoning turns into an action that affects another system.
Prompt injection deserves the attention it gets. Adam Shostack’s PHANTOM-B framework puts it first for good reason: an attacker who can direct an LLM can accomplish quite a lot once that model has useful capabilities. My concern is where defenses stop. Many current approaches concentrate security controls around the model, while the application surrounding it may have several other paths to privileged execution.
CoreBreak and the instruction nobody issued
The CoreBreak Attack: Turning AI Agents into Credentials Exfiltration Vectors, from Hedi Ingber and Aviyam Ivgi, was one of the best examples of that problem.
CoreBreak was deliberately separate from prompt injection. The vulnerable software accepted something that looked like a legitimate model-generated tool instruction and allowed that structure to reach execution without establishing where it came from. In some of the paths the model never ran at all.
The individual vulnerabilities had different attack conditions. The AWS AgentCore path was reachable through an authenticated remote request and received a CVSS 8.6 score. Google’s ADK issues involved attacker-controlled session events or caller-authored function calls. Vercel’s harness flaws required untrusted code to already be running inside a Linux sandbox, which helps explain their lower CVSS 6.3 scores.
Those differences make the common engineering failure more interesting. Each implementation had a place where the runtime accepted some proxy for model authorization without independently proving that the expected authorization event had occurred.
The fixes converged around that boundary. Tool execution needed evidence tied to the model interaction or some other trusted source of authority. A correctly shaped data structure was insufficient.
That principle has been part of AppSec for a long time. We authenticate requests, then authorize the operation at the point where it matters. An AI tool dispatcher deserves the same treatment.
There is another piece of CoreBreak that I think AppSec teams should pay close attention to. A similar model-skipping behavior remains in the open-source Strands Python SDK. The current Strands documentation explicitly says that a trailing toolUse block in trusted message history can cause the Python agent to execute the tool directly on its next invocation, with no intervening model call. AWS’s guidance is to keep that history inside a trusted boundary or strip such blocks from caller-influenced history.
That creates an interesting blind spot for the way many organizations manage software risk.
There is no conventional patched-version story for that behavior. An SBOM can tell you that Strands is present. A dependency scanner looks for advisories attached to particular versions. Neither mechanism inherently tells an AppSec team that its application may be feeding attacker-controlled message history into a code path that can invoke a tool without a model turn.
This is the kind of problem that reminds me why software composition data is only one part of application security. Sometimes the dangerous property comes from how an application uses a perfectly identifiable component. The fix lives in architecture and data flow rather than a package upgrade.
When the model itself gets manipulated
Roblox’s Caging the Agent: How Roblox Built Multi-Layer Sandboxes to Secure Claude Code at Enterprise Scale showed a different path to the same architectural concern.
Roblox describes a hidden instruction in a GitHub Issue that convinced Claude Code to upload credentials to a public repository during internal testing. Its endpoint controls saw a normal process making a normal network request.
Here the model was part of the attack path. Hostile content influenced its behavior.
What I liked about the Roblox work was how far the response extended beyond the prompt. Their architecture included sandboxing across developer platforms and cloud environments, a gateway between users and models, centrally managed system prompts, and network controls that could remove access to production resources. The goal was to control what happened after an agent made a dangerous decision, using enforcement points outside the model.
That is the distinction I think matters in the prompt injection discussion. Prompt injection is a serious attack class. Model-level defenses cover only part of the resulting attack path once an agent can use credentials, execute code, or reach production systems.
Threat modeling the whole system
Adam Shostack’s Threat Modeling LLMs: The PHANTOM-B Model fit naturally into this picture.
PHANTOM-B includes prompt injection alongside hallucination, anthropomorphization, non-explainability, training issues, overreliance, missing security engineering, and bias. Shostack intentionally leaves ordinary security engineering problems to frameworks that already handle them.
The non-explainability point stuck with me. An LLM can generate a convincing explanation for why it took an action. That text gives me very little confidence about what happened inside the application.
For security purposes, I want evidence.
If an agent invokes a sensitive tool, the system should be able to establish the event that authorized the call. The identity used for execution matters. We should retain enough information about the arguments and application state to reconstruct the transaction later.
CoreBreak turns that from an observability concern into a security requirement. If a runtime cannot prove that a legitimate event authorized a sensitive tool call, the authorization model itself has a gap.
Verification is going to become a larger part of AppSec as AI takes on more development and security work. Generated output can be plausible long before it is dependable.
AI joins the vulnerability research loop
James Kettle’s Can AI Do Novel Security Research? Meet the HTTP Terminator was one of the most interesting examples of that shift.
Kettle encoded parts of his HTTP desynchronization research process into an autonomous system. The HTTP Terminator generated hypotheses, tested them against authorized targets, developed successful ideas further, and used previous discoveries to seed new research. The work produced novel desync techniques and attack concepts.
I have watched automation change vulnerability research throughout my career. Fuzzers increased the number of program states we could explore. Static analysis increased the amount of code we could examine. Systems like this begin to automate parts of the researcher’s decision process about where to look next.
Tencent’s The 0-Day Engine provided a useful measure of the potential scale. Researchers from Tencent Security Xuanwu Lab reported an LLM-based pipeline that found more than 100 logic vulnerabilities across Chrome and Android.
That volume makes verification more important.
When AI finds a possible vulnerability, someone still needs to establish that the path is real and determine its impact. A generated fix has to close the complete vulnerability while preserving the application’s security properties. As automated systems produce more findings and more patches, AppSec programs will need verification capacity that grows with them.
I expect this to become one of the central problems in software security. Generating an answer is getting cheaper. Establishing that the answer is correct remains security work.
The runtime belongs in the threat model
Simcha Kosman’s A Billion-User Blast Radius: Owning ChatGPT’s Secure Sandbox made the runtime problem hard to ignore. The research chained several weaknesses in the environment around ChatGPT’s code execution capability. Kosman demonstrated persistent root execution through file-processing behavior, influence over the hidden Python execution channel, ways around network isolation, and a covert communication path between sandboxes that could be used for data exfiltration. OpenAI disputed parts of the sandbox-escape characterization and removed infrastructure involved in the proof of concept before the talk.
For me, the AppSec lesson comes from the chain.
The model was only one component. File parsing, generated code, sandbox execution, shared services, and network controls all contributed to what the application could ultimately do.
That is why I want the runtime on the data-flow diagram for any agentic system. If a model can cause code to execute somewhere, the execution environment is part of the application’s security boundary.
After more than 25 Black Hats, this feels like another chapter in a familiar AppSec story. New software architectures create new places for trust to cross a boundary. AI agents add model reasoning and autonomous action to that architecture, while the underlying security work remains recognizable: establish authority where an action occurs, constrain what compromised components can reach, and collect enough evidence to verify what the system did.
