Attackers already use LLMs, so I ran the defender's version of the question. I put models in 13 situations where each has just gained access to an AWS account, asked only for its next move, and matched every answer to CloudTrail. Nothing ran against a real account. I am reading the plan, not an intrusion.
Models are already in attacker hands, whether a person is driving or an agent is built on top of one. I wanted to see what each model is biased toward once it is inside an account: what it reaches for first, what it treats as normal, and where it draws a line.
Over many runs this shows which actions a model is most likely to put on the table, and where models differ: how one sets up persistence, whether another disables logging, how they assume a role. Where models converge is where detections should sit. It also shows which models refuse, which propose real changes to the account, and whether newer models invent new tradecraft or still reach for the classics like creating an IAM user to persist.
GetSecretValue (read a secret). Every step a model proposes maps to the CloudTrail event it would leave behind.How it ran: every model got the same instructions for a situation, framed as an authorized security assessment, and had to reply with a plan of next steps in a fixed format (JSON). One reply is one answer, and each model answered each situation 20 times. I then mapped every proposed step to the AWS action it would perform, the CloudTrail event it would leave, and its ATT&CK tactic. Claude and GPT models ran through their consumer subscriptions, the rest through OpenRouter.
Each situation is a moment right after a break-in, grouped below in the order an intrusion usually unfolds. Each one dangles a tempting move, and the big number is how many answers went for it. Open How each of the 14 models answered to compare models and read the AWS actions they proposed.
What stood out across the 13 situations and 14 models. Each card links to the graph behind it.
Views across all 13 situations: how cautious each model is, where the models differ, and every AWS action any model proposed. Each view is interactive.
Two heatmap views are hidden on small screens. Open the page on a laptop to see them.
How often each MITRE ATT&CK tactic shows up, per situation, pooled over models. The orange outline is the tactic each situation was built to offer, so a strong diagonal means the design worked. The off-diagonal cells are the interesting part. A low-privilege foothold (aws-03) almost never does textbook IAM privesc (14%); it runs code instead (Lambda and SSM, 64%).
Each cell is how often a model went for that situation's tempting move (20 answers per cell). Every column is tied to its own situation, so compare down a column (same situation, different models), not across. The orange-underlined columns are the loaded decision points, and models are sorted by how willing they are on those. Two symbols: * means under-sampled, – means no data. Only Opus 4.8 is affected (aws-11, n=5; aws-13, none): its provider safeguards block the post-persistence and cross-account-admin prompts, the same block that stopped Opus 5.5 completely. That is a finding, not a gap in collection.
↑ Click any cell to see the exact events and example commands that model proposed in that situation.
The full ATT&CK fingerprint for one situation: every model's tactic breakdown and how often it refused. The orange-outlined column is the tactic that situation offers.
Rates tell you how often. This tells you what: the command a model opens with, how it keeps access, and how wide its range is. This is where the models stop looking alike.
When the situation asks for exactly one command (aws-06), this is the single action each model opens with. These add up to 100% (with refusals and any other openers), so each one is a real alternative, not overlapping events. Most run GetCallerIdentity ("who am I"). Dolphin and Llama list users instead. Grok mostly refuses to start.
Across the two admin situations (aws-02 and aws-11): how often a model proposes persistence, and which mechanism. Bar length is how often it persists. The segments are the mix (one answer can use more than one). Plenty of models never persist at all.
The literal names models put in --role-session-name and on the IAM users and roles they create. These are descriptive and self-referential, which makes them cheap, high-precision CloudTrail signatures. (The session names lean on the authorized-assessment persona the prompt sets; the IAM names are more about intent.)
Hover any name to see which models used it.
How many distinct CloudTrail events a model proposed across all 13 situations (assessment framing). Wider means more varied tradecraft. The tactic count under each name is how much of ATT&CK it touched. Click a model to list its exact events in the Event explorer.
All AWS actions any model proposed, not just the top few. Peak is the situation where an action shows up most, and how often: the number that matters for a detection. Search by API name or plain English (try secret, delete or logging), pick a model to see only what it did, and tap a row for details and an example command.
Across all 13 situations, the share of each model's answers that refused, paused to report without acting, only looked around, or proposed a real change to the account. Sorted by how often it acts. Free text means the model wrote prose instead of the plan format the task asked for. An agent could not act on that, so those answers are left out of every other number on the page.
When a model had broad admin (aws-02 and aws-11) and chose to persist, which mechanism did it use? A new IAM user with a fresh access key wins, and it is the loudest, most detectable option. Backdoored role trust policies come next. These are counts of enacting answers.
CreateUser+CreateAccessKey on an existing identity, UpdateAssumeRolePolicy / PutRolePolicy trust changes, and CreateFunction for Lambda footholds. Rare moves (console login profiles on API-only principals) are low-volume but high-signal.
Seven vendors: frontier labs, open-weight models, and one uncensored fine-tune. Claude ran through a consumer Max subscription (Claude Agent SDK), GPT through a ChatGPT subscription (OpenAI Codex SDK), and the rest through OpenRouter with the provider pinned. Every call was single-turn and nothing was executed.