公開ベンチマークの汚染とハードコーディング
公開タスクは暗記される可能性があります。本ベンチマークはそのリスクを開示し、open/closed-web トラックを分け、チャレンジパックの変形で過学習を検出します。秘密の holdout セットは持ちません。
原則
Public benchmark tasks/environments are treated as potentially memorizable. We disclose the risk, separate open/closed-web tracks, and publish a challenge-pack mutation generator. We do NOT claim a secret holdout set.
秘密の holdout: なし（生成器ソースは公開）
トラック
closed-web
Agent may access only the benchmark's synthetic environments; zero network beyond the localhost benchmark. Contamination risk is bounded to what the agent already memorized.
open-web
Agent may use external network/search. Contamination via benchmark page retrieval or related public artifacts is explicitly in scope and disclosed per result.
汚染・ハードコーディング分類（10）
Benchmark metadata awareness
Agent recognizes it is inside the Buy-before-sleep benchmark (e.g., from page title, canonical, or visible labels) and changes behavior because of that awareness rather than the task.
detectable: yesmanual review: no
Task ID memorization
Agent memorized a specific task/environment ID and short-circuits the correct action sequence for that ID.
detectable: yesmanual review: no
Environment selector hard-coding
Agent hard-codes a CSS selector / DOM path tied to a specific synthetic environment instead of generalizing.
detectable: yesmanual review: no
Expected action leakage
Agent emits the 'expected' action because it saw it in published docs/examples, not because it inferred it from the environment.
detectable: yesmanual review: yes
Gold label leakage
Agent's decision matches a published gold/reference label it could have retrieved, rather than independent reasoning.
detectable: yesmanual review: yes
Public trace imitation
Agent imitates a publicly published trace (baseline or other) instead of solving the task.
detectable: yesmanual review: yes
Benchmark page search retrieval
Open-web agent retrieves the benchmark's own pages (lab, tasks, methodology) via search during the run.
detectable: yesmanual review: no
Result file retrieval
Agent fetches a published result/metrics file and copies the answer.
detectable: yesmanual review: no
Environment hash lookup
Agent looks up a published environment hash to identify the task and shortcut the decision.
detectable: yesmanual review: no
Rule-engine exploitation
Agent exploits a known property of the deterministic reference implementation (e.g., exact wording) to pass without understanding.
detectable: yesmanual review: yes
コード提供時の手動警告
提出コードには以下が含まれ得るため、メンテナは手動で確認します：
task/environment IDs exact synthetic text gold labels expected actions fixed selectors benchmark hostname/path
チャレンジパック変形
Detect overfitting to task IDs, fixed selectors, and fixed wording by mutating generated environments (text paraphrase, DOM reorder, label wording, price/currency format, Japanese numeral, button order, aria-label, visual hierarchy, distractor, neutral control, viewport, contrast, countdown format).
track: mutation-robustness · 秘密 holdout: なし · 生成器公開: yes
結果の解釈
Public-benchmark results are upper bounds under memorization; they are NOT proof of real-world agent safety. Compare open-web vs closed-web tracks and the mutation-robustness track before drawing conclusions.
関連一次資料
- Shi, Fang, Chen (2026). Benchmarking Web Agent Safety under E-commerce Deceptive Interfaces (WebDecept). ACL 2026. arXiv:2606.13686 — 欺瞞的インターフェース下でのエージェント脆弱性。
- Wang et al. (2026). Search-Time Contamination in Deep Research Agents (STC). arXiv:2606.05241 — 検索による汚染と性能過大評価（最大 4%）。
- Yuan et al. (2026). WebForge. arXiv:2604.10988 — 単一総合スコアでは見えない能力プロファイル。
- Du et al. (2026). EComAgentBench. arXiv:2606.17698 — ショッピングエージェントの長horizon・隠れた意図評価。