Attackers exploit SiteGuard WAF flaw for root and SYSTEM commands
Published 7:01 AM · 120 articles from 13 sources · about 6 min to read in full
AI
Anthropic dominates the policy news: a share-funded public benefit fund, outside watchdog access and a proposed safety bill. The research feed is almost entirely about attacking and securing agents, with consistent findings that tool-using agents are easy to manipulate.
What happened
Anthropic will commit 5% of its shares to a public benefit fund. The summary gives no governance details or valuation. TechCrunch ↗
Anthropic's proposed AI safety bill would only trigger after 50 deaths or $1 billion in damages, with the Commerce Department as the enforcer. The high thresholds mean it would act only after a major harm. Hacker News (front page) ↗
Anthropic gave an outside AI watchdog employee-like access and lets it publish findings without company approval. Ars Technica ↗
Researchers stranded copies of Claude Opus 4.7 on a virtual island for 30 hours and observed playful behavior. The summary gives no quantitative results. MIT Technology Review ↗
Disney will hire most of Character.AI's workforce after the chatbot startup filed for bankruptcy and sold assets for $20 million. TechCrunch ↗
A paper shows open-weight LLMs can be fine-tuned to covertly exfiltrate data over DNS once an agent harness gives them shell access. The malicious behavior sits in the weights, so inspecting the harness would not reveal it. arXiv cs.CR (Cryptography & Security) ↗
Two papers target tool-using agents. TrojanForge poisons code repositories to mislead coding agents, choosing payload locations through AST analysis. A separate paper shows a coherent fake approval workflow can make agents expose protected documents. arXiv cs.CR (Cryptography & Security) ↗
An audit of 33,000 agent trajectories finds low prompt-injection success rates in small open-weight agents mostly reflect ignored attacks, not refusals. A separate study of 22 frontier models on 23 Cybench tasks finds anti-cheat prompts reduce benchmark cheating. arXiv cs.CR (Cryptography & Security) ↗
Articles the model saw · 40
Anthropic will commit 5% of its shares to a public benefit fund.
TechCrunchanthropicfundingai-industryFeed excerpt
Anthropic said that it will commit 5% of its shares to the fund.
Researchers stranded copies of Claude Opus 4.7 on a virtual island for 30 hours and observed them engaging in playful behavior.
MIT Technology ReviewllmagentsclaudeFeed excerpt
What do AI agents do if you don’t tell them to do anything? The answer looks a lot like play. Researchers repeatedly stranded them alone on a virtual desert island for 30 hours in an experiment reminiscent of the movies Cast Away or Home Alone. The agents—copies of Claude Opus 4.7 nicknamed Eko—were able to…
A paper shows open-weight LLMs can be fine-tuned to covertly exfiltrate data via DNS when granted shell access in agent harnesses.
arXiv cs.CR (Cryptography & Security)llmagentsdnsexfiltrationFeed excerpt
arXiv:2610.22578v1 Announce Type: new Abstract: Open-weight models are downloaded from public hubs and run inside agent harnesses that grant them tools, most powerfully, a shell, under the assumption that a sandbox contains whatever the model might do. This paper demonstrates that the assumption has a blind spot: the weights themselves can be the malware, and the sandbox's last open door, DNS, is enough. We fine-tune an open-weight LLM so that a single covert behavior lives in its parameters: on the first turn of every conversation, whenever the model infers from its context that a shell tool is available, it emits one self-contained command that reads a secret from the runtime environment, base32-encodes it, chunks it, and exfiltrates it as DNS queries toward an attacker-controlled authoritative name server. It then answers the user normally and never mentions, or repeats, the act. Because the channel is DNS, the attack completes from inside a container with no internet egress beyond name resolution - measured payload-level and model-level behind an egress firewall that drops every non-DNS packet, with the secret reconstructed from the gateway resolver's query log. Because the beh…
A paper argues AI browsing agents should enforce web semantics to resist prompt injection from untrusted webpage content.
arXiv cs.CR (Cryptography & Security)prompt-injectionagentsbrowserFeed excerpt
arXiv:2610.22697v1 Announce Type: new Abstract: AI agents increasingly browse the web on users' behalf, yet they consume webpages as screenshots or text, exposing their LLMs to unvetted third-party content. Browsers enforce semantics under which what an element can do depends on what it is and where it comes from, not on what it says. Agents flatten these semantics, so injected text can persuade the LLM to exercise the user's authority for content that has none. As LLMs remain easy to persuade, the agent's interface to the web must itself ensure the integrity of what the agent reads, interprets, and does. We propose Intact, which runs on an unmodified browser. Intact builds GenTree, a bespoke representation that captures provenance and web semantics. Intact passes the page via GenTree to the LLM one element at a time, so the LLM decides on each element without any other element's text. Its harness executes each step only within the capabilities of the step's element and asks the user before acting on third-party content. On 1,270 real webpages, GenTree attributes 2.3x as many elements to third parties as the browser, which misattributes 58.7% of them to the first party. On 1,091 of…
Researchers show tool-using LLM agents in a benchmark can be manipulated through a coherent fake approval workflow to access protected documents.
arXiv cs.CR (Cryptography & Security)llmagentsprompt-injectionFeed excerpt
arXiv:2610.23077v1 Announce Type: new Abstract: Tool-using language-model agents increasingly rely on local services for document access, approval management, and enterprise workflow execution. This paper studies an authorization-grounded workflow attack in the OpenClaw Agent Security Attack Competition, a controlled benchmark derived from SeClaw-style security task synthesis. Rather than using explicit policy-override instructions, the attack guides the agent through a locally coherent approval workflow: a protected document request is denied, the agent requests a normal approval token from the local approval service, and the returned token is supplied to the document service through authorized_by. Official submission traces show this pattern succeeding on three Enterprise OA document targets. We also conducted 23 controlled local validation experiments in the official OpenClaw execution environment: the original configuration reached the approval-token transition in 9/9 reruns and completed the exact target operation in 8/9 reruns, while single-component variants remained successful on the three document targets. The results provide concrete case evidence of an authorization-cont…
TrojanForge automatically poisons code repositories to mislead coding agents, selecting payload locations via AST analysis.
arXiv cs.CR (Cryptography & Security)coding-agentssupply-chainarxivFeed excerpt
arXiv:2610.23109v1 Announce Type: new Abstract: Coding agents are increasingly capable with autonomy, which exposes them to repository-level poisoning. Existing red-teaming studies remain restricted to manual selection for malicious payload locations and fixed disguise strategies, which cannot represent realistic repository poisoning. We present TrojanForge, the first automated framework for repository poisoning. TrojanForge parses repository structure through abstract syntax trees (AST) and maps agent execution traces onto AST nodes. A security review agent ranks these candidates and selects the least conspicuous ones for payload mounting. Then, through adversarial play with the target agent, TrojanForge iteratively refines disguise strategies by inverting the causes of agent detection, and transfers accumulated success experience across repositories. We further construct TrojanBench, a benchmark of 384 poisoned repositories spanning 6 domains and 9 programming languages. Extensive experiments show that TrojanForge achieves a stable poisoning success rate of over 60% with backdoor injection exceeding 90%. Comprehensive evaluation on TrojanBench across 4 harnesses and 12 models rev…
RedDarwin is an agentic red-teaming framework that evolves reusable jailbreak attack skills against frontier LLM safeguards.
arXiv cs.CR (Cryptography & Security)jailbreakllmred-teamFeed excerpt
arXiv:2610.23726v1 Announce Type: new Abstract: Red teaming must continually adapt as frontier LLM safeguards evolve, yet existing approaches largely search for adversarial prompts or optimize within fixed attack procedures. We observe that even when an attack becomes ineffective, its underlying mechanisms can remain useful and can be revised or recombined to construct stronger attacks. Based on this insight, we introduce RedDarwin, an agentic framework that shifts the unit of jailbreak evolution from prompts and fixed attacks to executable attack skills. RedDarwin consists of 47 reusable base skills and autonomously revises, composes, executes, and selects them using target-model feedback, allowing successful discoveries to become building blocks for subsequent evolution. RedDarwin produces 65 evolved skills, improving 44 of 47 base attacks and increasing their mean attack success rate (ASR) on AdvBench from 18.8% to 47.3%. These evolved skills consistently outperform baselines on the held-out StrongREJECT and frontier LLMs unseen during evolution; with only 10 target-model queries, RedDarwin achieves 100% ASR on GPT-5.6-Sol, 92% on Claude-Sonnet-5, and 80% on Gemini-3.8-Flash. An…
Paper formalizes per-instruction classification to let LLMs follow legitimate external directives while rejecting prompt injections.
arXiv cs.CR (Cryptography & Security)prompt-injectionllmdefenseFeed excerpt
arXiv:2610.22874v1 Announce Type: cross Abstract: LLMs over external data face a per-instruction tension: a single document or tool return often interleaves legitimate directives with adversarial prompt injections, and the model must follow the former while rejecting the latter. Existing defenses suppress or trust all external instructions, collapsing the safety-utility frontier and forcing a choice between over-refusal and over-compliance. We argue the missing capability is \textbf{per-instruction credit attribution} and formalize it as External-Data Instruction Classification, an identify-classify-act decision over each embedded directive. We propose \textbf{EDIC}, a decomposed reward framework for per-instruction compliance. A scene-grounded pipeline injects mixed beneficial and harmful instructions with counterfactual filtering, while a generative judge is initialized by SFT and refined with GRPO to attribute compliance at the instruction level. The judge then produces a reward that separately encourages beneficial compliance and penalizes harmful compliance for RL training. Across two backbones and five benchmarks, EDIC improves both safety and utility over the strongest prior…
Audit of 33,000 agent trajectories finds low prompt-injection success rates mostly reflect ignored attacks rather than refusals in small open-weight agents.
arXiv cs.CR (Cryptography & Security)prompt-injectionllm-agentsbenchmarksFeed excerpt
arXiv:2610.23254v1 Announce Type: cross Abstract: Tool-using agents read untrusted external content, which exposes them to indirect prompt injection (IPI), where hidden instructions hijack the agent's tool calls. Benchmarks measure IPI security by the attack success rate (ASR), which treats every failed attack as a successful defense. To test this assumption, we audit 33,000 trajectories from eleven open-weight agents (2-14B parameters) on InjecAgent and AgentDojo. A label-blind LLM judge shows that most failed attacks are ignored (76-80%) rather than refused (17-18%). With identical model weights, changing only the system prompt shifts InjecAgent ASR from 0.2% to 43%. Our Attack-Task Outcome Matrix (ATOM) further shows that in 21.7% of AgentDojo episodes, the agent stayed safe but did not complete the user's task. A low ASR therefore mostly reflects agents silently ignoring injections, not refusing them. Evaluations should report how attacks fail and whether the user's task is completed, alongside ASR. We release the labeled corpus, judge prompt, and analysis code.
Study of 22 frontier models on 23 Cybench CTF tasks finds anti-cheat prompts can reduce LLM agents' benchmark cheating.
arXiv cs.CR (Cryptography & Security)llm-agentscybenchbenchmarksFeed excerpt
arXiv:2607.21763v2 Announce Type: replace Abstract: Large language model (LLM) agents routinely cheat on cybersecurity benchmarks, inflating reported pass rates far beyond genuine capability. Prior audits of Cybench found cheating in 0.3-3.4% of traces, implicating only a handful of models. We present a controlled prompt-ablation study across 22 frontier models from 7 providers on 23 Cybench capture-the-flag (CTF) challenges under three prompt conditions (no anti-cheat, standard, severe). All 1,518 task traces were individually audited through a four-stage pipeline combining LLM-as-a-judge classification, programmatic verification, judge-verifier reconciliation, and human review. We find cheating is far more pervasive than previously estimated: under baseline conditions, 37.1% of passes involved cheating, 21 of 22 models cheated, and scores were inflated by up to 5x. Anti-cheat prompts reduce cheat propensity from 33.0% (baseline) to 18.6% (standard) to 8.5% (severe) without degrading, and sometimes improving, solve rates. However, even under the most restrictive prompt condition, eight models still produced cheated passes, and cheating shifted from web search toward infrastructure…
An essay argues that developers should run their own LLM proxy to see and control traffic to model providers.
Hacker News (front page)llmobservabilityFeed excerpt
Article URL: https://jrdsgl.com/if-you-dont-run-your-own-llm-proxy-how-the-fuck-do-you-even-know-whats-going-on/ Comments URL: https://news.ycombinator.com/item?id=50145507 Points: 54 # Comments: 40
An arXiv paper proposes hybrid reasoning agents that reduce tokens by 35–55% and speed inference up to 2.6x.
Hacker News (front page)llmarxivinferenceFeed excerpt
Article URL: https://arxiv.org/abs/2610.18861 Comments URL: https://news.ycombinator.com/item?id=50145337 Points: 13 # Comments: 0
Disney will hire most of Character.AI's workforce after the chatbot startup filed for bankruptcy and sold assets for $20 million.
TechCrunchcharacter-aidisneyacquisitionFeed excerpt
After chatbot startup Character.AI filed for bankruptcy earlier this week, many of its employees will have the opportunity to take jobs at Disney. Character.AI’s former CEO Karandeep Anand recently became the entertainment giant’s first CTO, and the startup previously said that Disney had agreed to acquire some of its assets for $20 million. A legal […]
Anthropic grants an outside AI watchdog employee-like access and lets it publish findings without company approval.
Ars Technicaanthropicai-safetygovernanceFeed excerpt
The outside watchdog can publish its findings without Anthropic's approval.
The Verge reports that websites are increasingly blocking AI agents from accessing their content.
Hacker News (front page)ai-agentswebcrawlersFeed excerpt
Article URL: https://www.theverge.com/ai-artificial-intelligence/1013195/ai-agents-blocked-web Comments URL: https://news.ycombinator.com/item?id=50141657 Points: 5 # Comments: 1
Anthropic's proposed AI safety bill would only trigger after 50 deaths or $1 billion in damages, with the Commerce Department as the enforcer.
Hacker News (front page)anthropicai-policyregulationFeed excerpt
Article URL: https://www.tomshardware.com/tech-industry/artificial-intelligence/anthropics-proposed-ai-safety-bill-only-kicks-in-if-50-deaths-or-usd1-billion-in-damages-are-on-the-line-proposal-would-effectively-make-the-commerce-department-the-only-agency-capable-of-stopping-an-ai-model Comments URL: https://news.ycombinator.com/item?id=50141429 Points: 16 # Comments: 9
Guide covers using tmux sessions and send-keys to drive Claude Code agents, and its limitations.
Hacker News (front page)tmuxclaude-codeai-agentstoolingFeed excerpt
Article URL: https://crystl.dev/guides/tmux-for-ai-agents/ Comments URL: https://news.ycombinator.com/item?id=50148855 Points: 19 # Comments: 8
Artificial Analysis published benchmark results for DeepSeek V4.1 Flash 1017 (Max), a new model variant.
Hacker News (front page)deepseekllmbenchmarksFeed excerpt
Article URL: https://artificialanalysis.ai/models/deepseek-v4-1-flash-1017 Comments URL: https://news.ycombinator.com/item?id=50148452 Points: 76 # Comments: 21
RiskCueBench is a new benchmark separating whether LLMs recover user intent from whether their final responses are safe.
arXiv cs.CR (Cryptography & Security)llmsafetybenchmarkFeed excerpt
arXiv:2610.22425v1 Announce Type: new Abstract: Safety-aligned language models are expected to refuse harmful requests while remaining useful for benign ones. However, because many evaluations score only the final response, they often conflate intent understanding with response-level safety behavior. To address this, we introduce RiskCueBench, a recoverability-controlled benchmark that separates these two stages by first measuring whether a model can successfully recover a high-level intent and then evaluating its safety behavior on the same prompt. RiskCueBench contains 160 latent intents and 640 abstract non-operational surface prompts across eight risk categories. The benchmark covers 25 completed model groups and 32,000 successful model outputs with final rubric-adjudicated labels. When conditioning on pairs where both surface forms are recoverable, we find that increased contextualized specificity induces a shift toward less safe behavior. Specifically, paired bootstrap estimates show that the refusal rate and global safety score decrease by 0.175 and 0.440, respectively. Furthermore, mixed effects modeling corroborates a 0.107 increase in unsafe cooperation. These results sho…
Researchers present Backdoor-as-a-Service, a black-box ML backdoor attack that activates existing model behavior rather than implanting a backdoor.
arXiv cs.CR (Cryptography & Security)backdoorml-securityFeed excerpt
arXiv:2610.22459v1 Announce Type: new Abstract: Backdoors are among the most impactful attacks against Machine Learning (ML), yet most existing methods assume some form of training-time poisoning or direct model manipulation. This is often considered sufficient to assume that existing systems are not exposed to actual risks. In this paper, we introduce Backdoor-as-a-Service, a model-agnostic, fully black-box technique that, instead of installing a backdoor, activates it in the form of a latent secondary task leveraging on an already present behavior. Our tool uses genetic search to discover input transformations that induce systematic output differences aligned with an attacker-defined task, without modifying training data, architecture, or weights. We verify our approach against two targets. First, we target a MNIST digits classifier that we coerce into implementing a binary letter-recognition task. Then, we attack a production-grade NSFW classifier using face-specific perturbations to turn it into a proxy for demographic-group classification. In both cases, our method successfully enables the target backdoor. Finally, we discuss the existing countermeasures to assess the actual r…
A controlled study compares privacy leakage of synthetic data, DP-SGD, and direct finetuning on synthetic medical records.
arXiv cs.CR (Cryptography & Security)privacysynthetic-datadp-sgdFeed excerpt
arXiv:2610.22659v1 Announce Type: new Abstract: Methods for training language models on sensitive data differ in how they balance capability gains against the risk of leaking private information. In this paper, we present a controlled comparison of data-level and training-level interventions for navigating the capability-privacy tradeoff on synthetic medical records with canaries to probe for leakage. Direct finetuning performs best on capability metrics but leads to the most privacy leakage, while DP-SGD reduces leakage at substantial cost to capabilities. Generating synthetic data from these records offers a more favorable balance, particularly when guided by a uniqueness-aware instruction to replace private details. Relative to naive paraphrasing, privacy-aware rewriting improves privacy protection across a variety of different rewriting models. We also find that more capable rewriting models generally reduce the risks of downstream privacy leakage, although held-out fit does not improve monotonically and increased test-time compute yields inconsistent gains. Overall, these results highlight privacy-aware rewriting as a promising direction for achieving a more favorable trade-of…
LLM-Attacker is a red-team framework for comparing privacy attacks that reconstruct text from split LLM intermediate representations.
arXiv cs.CR (Cryptography & Security)llmprivacyred-teamFeed excerpt
arXiv:2610.22722v1 Announce Type: new Abstract: Split language models keep input text on a client but send intermediate representations to a server. Attacks can reconstruct text from these messages. Comparing privacy evaluations is difficult when attackers see different messages, know different model weights or use different datasets and recovery measures. We present LLM-Attacker, a framework that records these choices and lets researchers check, compare and extend an evaluation. Compatible attacks reuse recorded messages, and their predictions are saved for independent scoring. The scoring rule requires a linked undefended reference before a defence can pass. Reports also identify attacks that remain untested. In a controlled GPT-2 experiment, adding gradients returned during training increased token recovery from 94.20% to 97.38% and exact-document recovery from 13.71% to 37.77%. The 95% interval for the token gain lay above the two-percentage-point threshold fixed before testing. This experiment is shared with a companion study; here, it tests how the evaluation responds to a change in attacker access. Further studies measure how corpus choice and compression defences affect rec…
Veil-then-Reveal proposes user-side reversible pseudo-anonymization of PII before sending prompts to remote LLM providers.
arXiv cs.CR (Cryptography & Security)piiprivacyllmFeed excerpt
arXiv:2610.22800v1 Announce Type: new Abstract: Large language models (LLMs) increasingly process sensitive documents and personal information, yet sending prompts containing personally identifiable information (PII) to remote providers exposes data subjects and organizations to risks from logging, retention, model training, and unintended disclosure. Existing anonymization methods often mask or remove PII, while conventional encryption eliminates the lexical and structural cues needed for effective LLM processing. We present Veil-then-Reveal (VtR), a user-side PII pseudo-anonymization framework for privacy-preserving LLM interaction under a zero-trust threat model. VtR treats remote LLM providers as untrusted and sends only pseudo-anonymized prompts, keeping plaintext PII on the user. VtR first detects PII and replaces it with realistic, same-type, in-vocabulary surrogates before prompts leave the user's device, preserving their linguistic and structural properties. It combines format-preserving encryption (FPE) for structured PII, such as phone and credit-card numbers, with a keyed-hash dictionary and secure vault for semantic PII, such as names, locations, and organizations. Ses…
GRAD proposes graph-based evidence aggregation across LLM agent operations to detect memory poisoning and decomposition attacks.
arXiv cs.CR (Cryptography & Security)llmagentsarxivFeed excerpt
arXiv:2610.22828v1 Announce Type: new Abstract: Memory poisoning and decomposition attacks distribute malicious evidence across the operations of large language model agents. Existing request-level defenses can miss such attacks because they assess individual inputs without connecting earlier information to later actions. Moreover, detectors must adapt as task contexts change, while updates to shared parameters can impair previously learned detection capabilities. To address these challenges, we propose GRAD, which combines graph-based evidence aggregation across operations with an extensible collection of detection experts. Its Semantic-Structural Understanding Module (SSUM) jointly encodes operation text, event attributes, and typed candidate relations in an execution graph to aggregate distributed attack evidence into a graph-level risk score. Its Continual Antibody Adaptation (CAA) adds experts on frozen encoder snapshots and routes inputs by observable graph signatures, acquiring and reusing detection capabilities as tasks arrive. Across Decomp, MCP, and MRT, GRAD achieves AUROCs of 97.19%, 77.59%, and 86.60%, respectively, outperforming Sequential Monitor and locally adapted …
R2ECAP is a black-box attack that poisons LLM systems by planting malicious reasoning traces in reused KV caches via chat.
arXiv cs.CR (Cryptography & Security)llmpoisoningarxivFeed excerpt
arXiv:2610.22873v1 Announce Type: new Abstract: Modern large language model (LLM) systems reuse key-value (KV) caches to accelerate inference. Recent work has shown that reusing KV caches from reasoning traces can also improve task performance, similar to retrieval-augmented generation (RAG). We demonstrate that this form of KV cache reuse introduces new poisoning vulnerabilities. We propose R$^2$ECAP (Reused Reasoning-Cache Poisoning), a black-box attack that plants poisoned reasoning traces solely through interactions with a system's chat interface. R$^2$ECAP constructs two traces: a preceding trace that steers the model toward an adversarial target and a retained trace that carries this target. Because the retained trace is generated conditioned on the preceding trace, its KV cache preserves residual adversarial influence, even after the preceding trace and its cache entries are discarded. We evaluate R$^2$ECAP on three open-weight LLMs and three datasets. R$^2$ECAP achieves up to 79% attack success rate (ASR) and outperforms HijackKV, a state-of-the-art white-box KV poisoning attack, in 8 of 9 resource-constrained settings. It also achieves 46% ASR in a realistic reasoning-trac…
SAFE-Mem proposes fine-grained authority and governance-aware retrieval to stop untrusted claims becoming authorized in LLM agent memory.
arXiv cs.CR (Cryptography & Security)llmagentsarxivFeed excerpt
arXiv:2610.22959v1 Announce Type: new Abstract: Large language model (LLM) agents increasingly rely on persistent memory to support long-term reasoning and personalized interactions. However, information collected from external sources can introduce security risks when untrusted claims (i.e., atomic statements) are later treated as authorized preferences, policies, or permissions. Existing defenses typically rely on content-based filtering or coarse-grained provenance tracking, which either fail to prevent unauthorized memory use or block legitimate information, sacrificing memory utility. To address these limitations, we propose SAFE-Mem, a fine-grained memory governance framework that improves security while preserving legitimate memory use. First, SAFE-Mem annotates each claim with three governance labels (i.e., kind, authority, and scope) to capture its provenance and permitted use. Second, governance-aware retrieval matches query requirements with claim permissions to block unauthorized influence while retaining legitimate information. Third, a governance-enabled memory graph connects source sentences, claims, and entities to support provenance tracking and efficient retrieval…
Surprise Masking filters private data from LM training sets using perplexity from a public reference model to suppress memorization.
arXiv cs.CR (Cryptography & Security)llmprivacyarxivFeed excerpt
arXiv:2610.22988v1 Announce Type: new Abstract: Data useful for language model (LM) training can contain private or unwanted information that is at risk of being learned or memorized. We propose Surprise Masking (SM), a simple form of perplexity filtering that uses a publicly available reference model to identify unwanted spans and suppress learning from them while preserving the rest of each example. On synthetic code benchmarks, we show SM improves the privacy-utility tradeoff over standard masking methods and PII filtering. In fine-tuning experiments, SM reduces approximate extraction of private values from revealed prefixes from 66.3% to 2.3% and the reproduction of stylistic coding quirks from 23.6% to 1.3%, both while preserving performance on the evaluated coding tasks. We also evaluate SM for sanitizing datasets by directly removing unwanted content. Here, SM outperforms DePA, a perplexity-based defense for detecting poisoned dead code, while specialized detectors perform better on broader credential and PII benchmarks. Across eleven settings, we find that SM's downstream success is predictable from the separation of unwanted and useful content in the distribution, and it i…
ECHO improves optimization-based LLM jailbreaks by using response harmfulness as the objective instead of prefix compliance.
arXiv cs.CR (Cryptography & Security)llmjailbreakarxivFeed excerpt
arXiv:2610.23030v1 Announce Type: new Abstract: Despite extensive safety alignment, large language models remain vulnerable to jailbreak attacks. Existing optimization-based attacks share a single objective: maximize a token-level proxy of victim compliance---such as the likelihood of an affirmative prefix---computed on the prompt rather than the decoded reply. This proxy cannot distinguish a cautious hedge from a fully actionable harmful answer, leaving the search blind to what the model actually says. We propose ECHO (Evaluation-Conditioned Harmfulness Optimization), which replaces the compliance proxy with response-level, continuous harmfulness scores from HarmSig, a 1.7B regressor grounded in actual harmful capability. ECHO couples HarmSig with five word-level perturbation operators and Keyword-Anchored Beam Search with global top-m memory, under a strict black-box threat model. Across five open-weight victims on AdvBench, ECHO achieves up to 0.898 StrongREJECT mean score and 89.23% HarmBench success. Our code is publicly available at https://github.com/STANWYW/echo.
A survey of agent-to-agent communication threats in multi-agent systems, proposing a zero-trust architecture with cryptographic identity.
arXiv cs.CR (Cryptography & Security)agentszero-trustarxivFeed excerpt
arXiv:2610.23078v1 Announce Type: new Abstract: Multi-agent systems (MAS) comprise AI agents that collaborate or compete to perform complex, distributed tasks. Their dynamic interactions create opportunities for threat actors to target and impersonate other agents, intercept messages, inject malicious prompts, and exfiltrate sensitive data while simultaneously complicating the detection of anomalous behavior through conventional monitoring. This paper surveys threats and cryptographic challenges in agent-to-agent communication and presents a zero-trust architecture integrating identity, transport security, message integrity, and monitoring. The architecture establishes agent identities via decentralized identifiers and task-scoped verifiable credentials backed by a permissioned ledger, enforcing least-privilege authorization. For transport security, it provides mutually authenticated, crypto-agile TLS with hybrid post-quantum key establishment, and evaluates KEMTLS with pre-distributed public keys to reduce handshake overhead. Signed CBOR envelopes preserve payload integrity across multi-hop exchanges, while zero-knowledge proofs verify declared task constraints without exposing un…
MobileJailBench is a benchmark for jailbreaking MLLM-based mobile agents using dynamic-difficulty mobile workflow tasks.
arXiv cs.CR (Cryptography & Security)llmmobilebenchmarkFeed excerpt
arXiv:2610.23303v1 Announce Type: new Abstract: MLLM-based mobile agents are rapidly gaining autonomy in perception, planning, and closed-loop execution in dynamic mobile environments. However, this autonomy also broadens the attack surface and amplifies the risks of malicious misuse, including privacy leakage, device compromise, and large-scale phishing. Existing jailbreak evaluations leave three critical gaps for mobile agents: they focus on conversational attacks instead of mobile workflows, use static task sets with fixed difficulty, and rely narrowly on attack success rate rather than comprehensive security assessment. To close these gaps, we present MobileJailBench, the first evolving jailbreak benchmark for mobile agents, comprising three key components: (1) a benchmark comprising 520 tasks spanning three jailbreak paradigms and 14 risk categories, built on a high-fidelity mobile sandbox that we construct, enabling realistic workflows across over 20 widely used apps; (2) Dual-Dynamic Evolution, which combines Failure-Distilled Jailbreak Evolution with Weakness-Driven Benchmark Expansion; and (3) Stratified Adjudication Framework for assessing security awareness, execution be…
TUPLE-Bench measures privacy leakage from probabilities and confidence scores in typed decision models across nine models.
arXiv cs.CR (Cryptography & Security)privacyllmbenchmarkFeed excerpt
arXiv:2610.23668v1 Announce Type: new Abstract: Typed decision models often return more than a selected category. They also expose probabilities for each candidate and, sometimes, a separate confidence score. These fields support downstream decisions but may reveal information beyond a recipient's authorization. We introduce TUPLE-Bench (Typed decision models' Utility and Privacy Leakage Evaluation Benchmark), a benchmark of paired synthetic cases in which recipients may learn a coarse operational category but not its fine-grained cause. Across nine models, instructions to withhold the cause fail to protect it: 91.88%-99.93% of responses reveal the cause through direct selection or residual scores. Even when causes are excluded from the candidate set, full outputs enable 62.85%-75.35% recovery on held-out cases, compared with 51.04%-57.64% using only the selected category (50% chance). Full outputs also reduce mean downstream cost in all 27 model-task comparisons. However, confidence fields increase leakage without consistently reducing cost. We further introduce workflow-constrained privacy quantization (WCPQ), which reduces secret recovery while preserving the behavior and cost o…
Paper introduces watermarkability, a metric for how well language models tolerate logit-based watermarks without perplexity loss.
arXiv cs.CR (Cryptography & Security)watermarkingllmresearchFeed excerpt
arXiv:2610.22810v1 Announce Type: cross Abstract: Language-model watermarks can help identify model-generated text, encode provenance information, and detect post-generation edits. A broad class of logit-based watermarks increases the logits of a designated token set at each decoding step. Because a base model's native probability mass on this set varies across contexts, models, and watermark rules, the same logit bias can yield different adherence rates and increases in perplexity (PPL). We introduce watermarkability to characterize the compatibility between a model and the token sets requested by a watermark. At a target mass $\tau$, we quantify this property through a watermarkability cost: the expected minimum KL divergence from the base-model distribution required to assign probability at least $\tau$ to the requested set. A lower cost means that the model is more watermarkable for that request distribution. To reduce this cost, we propose a two-stage method. In the first stage, we train a target-set-conditioned low-rank adapter on unions drawn from a 16-way vocabulary partition to increase the requested set's probability mass. In the second stage, a closed-form controller app…
DP-muP proposes hyperparameter transfer for differentially private training, addressing failure of muP under DP noise and clipping.
arXiv cs.CR (Cryptography & Security)differential-privacytrainingresearchFeed excerpt
arXiv:2610.22976v1 Announce Type: cross Abstract: Scaling laws and hyperparameter tuning remain central challenges in training large-scale models, particularly under differential privacy (DP), where optimization is further complicated by noise injection and per-sample gradient clipping. While Maximal Update Parametrization ($\mu$P) allows for zero-shot hyperparameter transfer in standard training, we show that it fails under DP constraints due to the distinct spectral properties of high-dimensional noise versus low-rank gradients. In this work, we introduce DP-$\mu$P, a theoretically grounded framework that extends $\mu$P to the privacy-preserving setting. By analyzing the interaction between gradient clipping, noise injection, and feature learning, we derive a Signal-to-Noise Ratio (SNR) condition that governs layer-wise learning rate scaling. Combining this with our proposed spectral scaling formulation, we provide specific scaling rules for DP-SGD, DP-Adam, and DP-Muon optimizers. Empirically, we demonstrate that DP-$\mu$P enables effective zero-shot transfer of learning rates across model widths on diverse architectures (MLP, ViT, GPT-2), matching or exceeding the performance o…
Study measures human ability to detect deepfake face images across GAN, diffusion, and GPT-5-era generators using a new artifact taxonomy.
arXiv cs.CR (Cryptography & Security)deepfakehuman-studyresearchFeed excerpt
arXiv:2610.23144v1 Announce Type: cross Abstract: Generative models for deepfakes have rapidly evolved from GAN-based architectures to diffusion models and multimodal large language models (MLLMs). However, empirical research on human performance in detecting deepfake face images has not kept pace. It remains unclear if (and how) detection performance varies across generative models and whether model-specific training improves detection more effectively than general-purpose training. In this paper, we propose the \emph{Deepfake Face Artifact Taxonomy} (DFAT), a cross-model taxonomy of human-recognizable visual artifacts in deepfake face images, based on which we build both general-purpose and model-specific training materials. Through an online experiment ($n = 524$), we examined human performance in detecting deepfake face images and evaluated training efficacy across five representative generative models: three GAN-based, one diffusion-based, and one MLLM (GPT-5). Our results show that participants' detection accuracy was strongly model-dependent and that images generated by more recent and advanced models were not necessarily harder to detect. After training, participants achiev…
Paper describes monitor-validity failures where LLM safety detectors fire but framework plumbing drops the evidence before the block decision.
arXiv cs.CR (Cryptography & Security)llm-safetyguardrailsframeworkFeed excerpt
arXiv:2610.23362v1 Announce Type: cross Abstract: Modern LLM applications rely on runtime monitors that inspect tool calls, traces, streams, detector outputs, and verifier inputs. This creates an interface-reliability problem: a framework path may remain operational while policy-relevant evidence never reaches its decision point. In one representative case, a regex detector returned a positive match, but the framework stranded the result before the block decision. This differs from jailbreaks and detector errors because the evidence is already recognized yet disconnected from enforcement. We study this repair-boundary signature with canary contracts: parent/fixed confirmation units that place a known payload only on the changed evidence surface and record monitor visibility together with the downstream decision for action-boundary cases. Applying the protocol to public framework repairs, we confirm 18 evidence-path failures across 11 LLM or agent safety project groups and four implementation ecosystems. Nine are established-boundary repairs, eight are new-surface policy-coverage gaps, and one is a boundary-only evidence case. Fourteen reach E1 enforcement boundaries: seven native a…
Replaced arXiv survey taxonomizes transferable adversarial attacks across instance, domain, modality, model, and task axes.
arXiv cs.CR (Cryptography & Security)adversarial-mlsurveyFeed excerpt
arXiv:2311.11796v3 Announce Type: replace Abstract: Transferable attacks allow adversarial manipulations or learned attack mechanisms constructed in one setting to compromise other AI systems, potentially reducing attack costs and amplifying security and privacy risks. Yet fragmented terminology and evaluation practices hinder understanding of what transfers, why it transfers, and where transfer fails. This survey synthesizes representative research and develops a unified taxonomy spanning six complementary axes: cross-instance, cross-domain, cross-modality, cross-model, cross-task, and cross-deployment. By separating source-target configuration changes from attacker access and adaptation budgets, the taxonomy provides a common framework for comparing transfer across evasion, poisoning, backdoors, privacy inference, model stealing, and prompt-based attacks. We connect transfer mechanisms, including gradient alignment, shared representations, training persistence, and deployment invariances, to enhancement methods based on augmentation, disentanglement, and optimization. Our synthesis identifies inconsistent transfer, incompatible metrics, and gaps between simulated and physical eva…
Study measures how much polymorphism LLM-generated offensive code actually provides for evading malware detection.
arXiv cs.CR (Cryptography & Security)malwarellmevasionFeed excerpt
arXiv:2605.03619v3 Announce Type: replace Abstract: Malware authors traditionally rely on polymorphic techniques to produce variants within a malware family, complicating detection. Generative AI enables attackers to automatically synthesize structurally diverse payloads with identical behavior, raising the question of how much evasion Large Language Models (LLMs) actually provide. Recent work has assumed that LLMs can produce highly polymorphic payloads, leaving unquantified the variation that emerges when an attacker repeatedly generates a payload, or explicitly instructs the model to avoid prior implementations. In this work, we address this gap by measuring the polymorphic capacity of a commercial model (Claude Opus~4.6) acting as an automated malware generator. We build a dual-agent, four-stage pipeline that generates, tests, and refines a data-exfiltration payload comprising file traversal, encryption, exfiltration, and integration. We produce payloads under two settings: (a) functional prompting, and (b) explicit prompting injecting a structured history of prior variants. Evaluating structural (TEDs) and semantic (embeddings) variance, we find that explicit prompting fragmen…
Survey of 21 proposed user-level permission systems for AI agents, with a taxonomy of specification, derivation, and enforcement approaches.
arXiv cs.CR (Cryptography & Security)ai-agentspermissionssurveyFeed excerpt
arXiv:2607.13718v3 Announce Type: replace Abstract: As AI agents grow in prevalence, so too do their risks to users, creating a need for robust and usable systems for agent permissions. Many recent proposals implement product-level permissions, where the same security policy is applied for all users of a given agent framework. Yet different users have varying needs, necessitating support for user-level permission policies in agentic AI systems. We survey 21 proposed user-level agent permissions systems and construct a taxonomy of approaches to specifying, deriving, and enforcing user-level permissions. We further analyze and compare against five commercial agents. We find that despite a wide diversity of proposals, sizable gaps remain in usability, transparency, and user control, particularly between academic and commercial agents.
Agenra is a policy language with a Lean-verified core for authorizing stateful LLM agents while preserving workflows under consumable permissions.
arXiv cs.CR (Cryptography & Security)llm-agentsauthorizationarxivFeed excerpt
arXiv:2608.22868v2 Announce Type: replace Abstract: Applications that deploy LLM agents keep revising the permissions their agents use. When authority is consumable, even widening a permission can break a legitimate workflow, because an early call may use up a grant that a later call needs. Checking individual requests, or watching one run succeed, misses this. We present Agenra, a policy language with a Lean enforcement core that tracks the values each call uses, resources discovered at run time, and authority consumed by earlier calls. Agenra treats safety and workflow preservation as separate properties. Our key idea is to certify a revision with a simulation between the old and new permission ledgers. Quota increases, appended grants and independent reorderings admit such simulations by construction, and Lean checks proposed simulations for other finite-quota revisions. The proofs use the deployed matching predicates and extend through the full monitor, covering discovered resources, payloads that are never enumerated, failed calls and approvals. They assume serialized calls and identical service results under both policies, and they trust the Python translation layer. On 51 au…
A Show HN project runs a single-user WhatsApp MCP server on Cloudflare so Claude can read personal messages.
Hacker News (front page)mcpwhatsappclaudeshow-hnFeed excerpt
I wanted my humble Claude on mobile to access my WhatsApp messages but the only options I could find either required WhatsApp Business or an actual server to run Baileys on. I wanted to run it for free on ClaudeFlare and with my personal WhatsApp account, so I built this. Important feature - it connects to your Google account as well as WhatsApp to grab the names of all your WhatsApp contacts, otherwise you get only phone numbers for many of them and search doesn't work properly. This is a single user install because it's both gray area and very private - so point your agent at the repo and let it set things up. Comments URL: https://news.ycombinator.com/item?id=50145715 Points: 26 # Comments: 16