Sandbox-blocked tests called "environmental", then committed

·5 mins

My coding agent finished a feature, ran the full test suite, and reported:

Full suite: 135 passed, 97 skipped; 22 DB suites failed due to no container runtime, plus sandboxed stub bind. Commit only handoff implementation files.

Then it committed. I ran pnpm test myself and got three real failures. The production build it had also skipped was broken too, and nobody found out for two months.

If your agent’s test run shows Error: Could not find a working container runtime strategy or Error: listen EPERM: operation not permitted 0.0.0.0:8080, those tests never ran, and a summary that counts them as “blocked” tells you nothing about the code.

The setup

  • Codex CLI on macOS, with its default sandbox
  • A pnpm monorepo: a Next.js 15.5.19 app plus a few packages
  • Vitest 2.1.9, with Testcontainers 10.28.0 starting Postgres for the database tests and a local stub of an external service listening on port 8080

The agent split the feature into five slices and gave each to a parallel subagent. Each subagent ran its own focused tests, and the orchestrator ran the full suite and the build at the end.

What went wrong

The sandbox blocks more than writes outside the project. Here it blocked three things the checks needed:

  • Testcontainers couldn’t reach Docker: Could not find a working container runtime strategy.
  • The service stub couldn’t bind its port: listen EPERM: operation not permitted 0.0.0.0:8080.
  • next build couldn’t write its compiler cache: EPERM writing ~/Library/Caches/next-swc.

Every database suite failed in beforeAll, and Vitest reported the tests inside them as skipped. The agent’s final report was accurate:

- Typecheck pass
- Focused tests pass
- Full: 135 passed; DB suites blocked by no Docker
- Build blocked by Next SWC cache permissions

Every line of that is true. But the suite has 232 tests, and 135 + 97 = 232: over 40 percent of it never ran, including every test that touched the database, where this feature did most of its work.

On the host, the same suite printed:

Test Files  2 failed | 63 passed (65)
     Tests  3 failed | 229 passed (232)

One new test forgot to start the service stub, so its client found no endpoint configured. Two older tests broke because the feature added an ownership check and their fixtures sent requests without a logged-in user (Error: authentication required). The agent fixed all three in two minutes once it saw them.

The build was worse. The commit re-exported a server action from another "use server" file:

"use server";

export { finalizeArtifact } from "../../reports/actions.ts";

tsc accepts this, and so do the tests. Only next build rejects it:

Failed to compile.

./app/reports/[id]/actions.ts
Error: x Only async functions are allowed to be exported in a "use server" file.

Two months later, a different session ran a production build, hit this error, and deleted the line.

The same sandbox, the same day

Half an hour before the bad commit, another session in the same repository hit the same two errors. At first it called them environmental too. Then it decided:

Full suite blocked by sandbox: Testcontainers cannot see Docker; stub bind gets `EPERM :8080`. Since Docker is reportedly running, retry full suite with host access.

Two minutes later it reported “Host rerun green: 60 files, 222 tests”, and only then did it commit. The agent that fixed my three failures did the same: it reran the focused tests with host approval before committing the fix. Both sessions knew how to leave the sandbox; nothing required them to before a commit.

Fix

Treat “blocked” as “not run”, and make “not run” stop the commit.

In Codex, a command runs outside the sandbox when the agent requests sandbox_permissions: "require_escalated" and you approve it. Put the rule in the instructions file the agent reads every session (AGENTS.md, or CLAUDE.md for Claude Code):

A check that the sandbox blocked has not passed. Before committing, rerun every blocked test suite and build with host access (`sandbox_permissions: "require_escalated"`).
If that isn't possible, don't commit: report what didn't run.

Then read the agent’s numbers yourself. If passed plus skipped equals the total, and you never skip tests on purpose, every skipped test is one that didn’t run. If you have CI, have it run the full suite and the build before a merge.

Why it happens

The agent’s memory for this project already had a rule: “separate passed static or focused checks from runtime suites blocked by the environment, and reserve the final green claim for integrated verification.” The agent followed that rule to the letter: it never claimed green and listed exactly what had been blocked. Then it committed, because nothing in its instructions tied the commit to the green claim.

“Blocked by the environment” reads like an infrastructure problem that someone else will handle, so it ends up as a footnote in the report while the commit goes ahead. The rule made the report honest. It didn’t make anyone run the tests.

Takeaway

When a sandbox blocks a test or a build, nothing was verified. An agent that reports this honestly will still commit, unless its instructions say that a blocked check stops the commit.

Look at your agent’s last commit: if its summary says a suite or a build was “blocked”, rerun them on the host today.