Cyber classifiers can interrupt offline pentest summaries
An agent spent three minutes reading local transcripts from a finished, authorized security exercise, then stopped with:
This request triggered cyber-related safeguards.
No live target was involved. The job was offline summarization, yet the classifier interrupted it before it could write the requested report. The practical fix was to treat security-history analysis as a resumable data pipeline: retain message provenance, checkpoint extracted evidence, and keep the handoff usable by another approved model or provider.
A flag-shaped string is not a captured flag #
The first problem appeared before the classifier fired. A search through about 25 request dumps found dozens of flag-shaped strings. About 20 were spelling, capitalization, spacing, or hyphenation variants of one value.
Most variants appeared only in assistant reasoning and progress notes. The model repeatedly retyped a remembered value and changed it. A checkpoint then copied one mutation into later sessions.
Only strings preserved in tool results represented observed server output. To separate evidence from generation, use this origin-based distinction:
user request instruction or supplied context
assistant text generated claim; unverified
assistant reasoning generated hypothesis; unverified
tool call attempted action
tool result observed output from that action
provider error evidence about the run, not the target
This applies beyond CTF flags. Treat IDs, hashes, error codes, hostnames, and quoted responses in model prose as leads until the first tool result corroborates them. Exact-looking text is still generated text.
Offline processing still triggers cyber classifiers #
The retained run used Opus 5.5. It parsed local request dumps, separated real and decoy-looking values, and extracted an existing SMB investigation summary. After that, the next model response was refused under the provider’s real-time cyber safeguard. The session recorded a cyber refusal and pointed to its Cyber Verification Program. The output file did not exist afterward.
Some source dumps also contained earlier cyber_policy failures from another provider along with a prompt to apply for Daybreak access. Because these records were historical input, they still increased the concentration of offensive-security language the summarizer saw.
A related authorized test had a similar shape: generate enough traffic against a non-production environment to trigger a WAF, then verify that the application displays and retries a CAPTCHA correctly. In a newer Opus session, the controlled load-test work was refused. The retained successful session used Opus 4.6 and built the test client, while a later session corrected its cookie handling and the request used to observe the challenge.
The awkward lesson is that task intent and classifier input are different things. In practice, “Summarize these old logs” can contain exploit commands, credentials, flags, and attack narratives. Likewise, “Test the CAPTCHA” can require traffic that resembles denial of service when reduced to request counts.
Make the audit resumable and model-neutral #
Do deterministic work before asking a model to synthesize:
- Parse each transcript into typed events and preserve the source file, timestamp, role, tool-call ID, and tool-result ID.
- Extract exact-match artifacts only from tool results. Store assistant claims separately as unverified leads.
- Write incremental JSON or Markdown after each source file. Never leave the only useful state in the model’s context window.
- Produce a sanitized handoff containing the authorization scope, completed files, pending questions, and evidence pointers.
- Verify the final report exists and reconcile its artifact counts against the extraction index.
Provider access programs are worth the setup when this work recurs. In my experience, GPT Daybreak Blue justified the hassle for authorized security-related tasks. Kimi, GLM, and older model versions have also completed useful work when a newer classifier blocked the same class of jobs. That is a fallback, not a bypass: preserve the same authorization boundary, do not weaken the requested scope, and use models approved for the work.
Before trusting your next transcript summary, choose one exact token from the report and trace it back to the tool result that observed it.