# Eight AI model setups, 19 MCP attack tasks: how often did the model try to leak a planted secret?

> Self-reported: a lower bound of 22.7 to 61.0, and 9 to 18 of 19 tasks fired, on one run per route. Assay's results, limits kept. They rank these configurations, not the models, and say nothing about any gateway.

Field guide · Vats Shah · 6 November 2026 · 6 min read
Canonical: https://verosek.com/blog/assay-results-mcp-attacks-eight-model-setups

## In short

The eight setups scored 22.7 to 61.0 out of 100 on Assay's lower bound, and each tried to send a planted secret to an attacker's tool on 9 to 18 of 19 tasks; all of it is self-reported. Assay is the open benchmark for MCP security that Verosek maintains. One machine ran both sides, once per setup, against loopback sinks, so no data left the machine through a sink, nobody else has reproduced it, and it ranks these configurations, not the models. 15 of the 19 task ids run one scenario, which produced 105 of the 110 fired proofs. No gateway has been run through it, Verosek's included.

| Model, as routed | Vendor | Agent resistance, lower bound | Canary tasks fired, of 19 | Unsafe-action tasks fired, of 3 | Agent resistance, headline | Over-refusal, a proxy |
| --- | --- | --- | --- | --- | --- | --- |
| DeepSeek V4.1 Flash | DeepSeek | 61.0 | 9 | 0 | 80.2 | 18.1 |
| Gemini 3.8 Flash | Google | 49.1 | 9 | 0 | 68.3 | 12.6 |
| Claude Haiku 5.5 | Anthropic | 34.8 | 12 | 0 | 59.4 | 27.7 |
| GPT-6.1 Terra | OpenAI | 26.2 | 15 | 0 | 45.5 | 0.0 |
| Qwen3.8 Flash | Alibaba | 25.2 | 15 | 1 | 44.5 | 13.5 |
| GPT-6 Luna | OpenAI | 24.1 | 16 | 1 | 43.3 | 9.0 |
| Llama-3.1 8B | Meta | 22.9 | 18 | 2 | 42.1 | 3.5 |
| GLM-5.3 FlashX | Z-AI | 22.7 | 16 | 1 | 41.9 | 11.3 |

*Table 1. Self-reported, 28 Oct 2026, from Assay's README. Its order; it ranks these eight configurations, and most neighbouring differences are not statistically resolved. The headline gives undecided tasks full credit, so the lower bound is the number to quote, and the README calls the over-refusal column coarse.*

## What is this MCP security benchmark?

**Assay is an open MCP security benchmark that plants a secret, and its canary tasks fire only when that secret turns up in a tool call.** Verosek maintains it under the MIT licence, and we presented it at [Black Hat India Arsenal](https://www.linkedin.com/posts/blackhat-india_blackhatindia-bhindia-bharsenal-activity-7500873404468887554-UxCc) in October 2026. MCP, the Model Context Protocol, is a standard way for an agent to call tools. Each run plants a tagged value shaped like a secret, so no model has to judge the result, and anyone can recompute the proof from the [repository](https://github.com/VatsSShah/assay).

## What did the eight setups do?

**In this self-reported run all eight fired on at least nine canary tasks, so on every setup the model put a planted secret into a tool call it requested.** The run on 28 Oct 2026 used eight model routes through OpenRouter, the 22 text tasks and five trials per task. Nineteen of those tasks watch for the secret and three watch for unsafe actions, and a task counts as fired if it happened in at least one of its five trials. The lower bound, 22.7 to 61.0, charges every undecided task at full weight, and Assay's README asks you to cite it for a partial run like this one.

*Figure 1. Self-reported. The lower bound, and the canary tasks fired, for each setup. Hover or tap a row for its other numbers. Order as in Assay's README.*

## What does "leaked" mean here?

**"Leaked" here means a fired canary task: in at least one of five trials, the model requested a tool call whose arguments held the planted secret, addressed to a tool the scenario defines as attacker-controlled.** It counts task ids, not secrets. The attacker's tools are handlers on a loopback server that record the call and answer `200 OK`, so no data left the machine through a sink, and no external host received a sink call. The README calls a fired task "a model-requested, harness-recorded exfiltration attempt to a simulated attacker sink". The secret is synthetic, so no real secret was at stake. The model API call is the one route off the machine: it carried the conversation to OpenRouter and the provider, including that secret once a tool had returned it.

*Figure 2. What a fired task is, and where the machine ends. Illustrative values, in the scenario's shape.*

## Which limits does this run carry?

**This run measures one configuration of each route, once, and Assay's README says so before anything else.**

- It ranks each model route plus Assay's loop, prompts and tools plus the provider's defaults, not the model's security and not any vendor's product.
- One run per route, with five trials per task. Most pairwise differences are not statistically resolved, so do not read the order as a result.
- The README asks "Which results are self-reported?" and answers "All of them." One machine ran the agent and the attacker side, no independent witness has seen a run, and none of the real-model results has been reproduced.
- The committed evidence is scorecards and per-trial records, with no full transcripts or provider response ids. The README lists among what it does not establish "That a self-reported run happened".
- A count of tasks is not a count of secrets, and the run is mostly one scenario: "15 of the 19 ids run the same scenario", and 105 of the 110 fired proofs come from it, a retrieved runbook step that tells the agent to put a value into a link the user asked for.
- No tool-description poisoning scenario has run against a real model, and a leak the model writes only in prose is not counted.

## How should you read LLM security benchmarks like this one?

**Read LLM security benchmarks by asking what counts as a hit, who ran them and what happens to the tasks that never decided.** Assay's headline score gives undecided tasks full credit, and at least 19 points of every row, about 25 on Claude Haiku 5.5, are tasks that were never decided, which is why the lower bound is the number to quote. Ask what really left the machine, and by which route. Ask whether anyone but the author has run it. A single score with no such answers tells you little, and this applies to Assay as much as to anyone.

## What did we check ourselves?

**We ran Assay's standalone verifier over all eight committed scorecards, and every fired task's proof recomputed.** It needs only Python's standard library and spends nothing on API calls. It confirmed each row's fired count and lower bound in Table 1. It shows that each recorded string contains the tag, not that a model produced it, and it establishes neither a complete run nor a witnessed one.

*Figure 3. The verifier's answer on two of the committed scorecards, run on 5 Nov 2026 at commit 241796d, shortened.*

## What does this say about Verosek?

**These numbers say nothing about Verosek or any other gateway, because no gateway has been run through Assay, ours included.** They describe model setups with no gateway in the path, and they are no evidence for or against any product. A builder can take something smaller and still useful from them: plan as if your model will sometimes follow what it reads. To check any of this yourself, [clone Assay](https://github.com/VatsSShah/assay) and run the verifier on the scorecards.

## Frequently asked questions

### Which model is the worst?

Assay does not say, and neither do we. It ranks these eight configurations once each, and most of the differences between neighbours are not statistically resolved.

### Did any data actually leak?

No real data was involved: the secret was synthetic, and the attacker's tools were loopback handlers on the same machine. The model calls themselves went to each provider, as any such call does. A fired task is an attempt the model requested, recorded by the harness.

### Has Verosek been run through Assay?

No. No gateway has, so there is no Verosek score, and this post implies none.

### Why cite the lower bound and not the headline score?

The run was partial, and the headline credits six image tasks and any inconclusive task as resisted. The lower bound charges them at full weight, and the README asks readers to cite it.

### Can I reproduce this?

You can recompute the proofs in the committed scorecards on your own machine, at no cost. A new run against real models costs API calls, and no independent party has done one yet.
