Lessons learned

Offline login in an iOS or Android app: design it as offline unlock

Tue, 03 Nov 2026 18:00:00 +0100

A product request for “offline login” in a mobile app with a cache of sensitive data (customer invoices, HR records, patient notes, credentials) sounds like one feature. On a phone with no network, nothing can check the password against the server, apply MFA, or learn that the device was revoked yesterday. What the app can do is prove a local factor to a device that was enrolled earlier and release keys for ciphertext already on it:

offline login: authenticate the user, apply policy, authorize data (impossible offline)
offline unlock: prove a local factor, release keys for cached data (what you can build)

I researched how to build the second one on iOS and Android. This post collects the parts that are easy to get wrong: where the password path needs a hardware key, why a biometric prompt without a CryptoObject is only a boolean check, which clock keeps counting while the phone sleeps, and what changes for users who have no password at all. It is longer than a usual post here, because the traps sit in different layers.

What an offline unlock can and cannot prove

A connected sign-in lets the server authenticate the user and the device, require MFA, evaluate current policy and revocation, and send newly authorized data. An offline unlock does none of that. So it must never renew a server session, refresh authorization, or make cached data look current. Label the action “Unlock” in the UI and keep “Sign in” for the server. Show “Last checked” (policy) and “Last synchronized” (data) separately, because a recent local unlock is neither.

The trade-off between availability and revocation is unavoidable. While the device is disconnected, a revocation event cannot reach it, and plaintext it already showed cannot be recalled. A long offline window helps users during outages and travel. It also lets a lost or deauthorized device keep unlocking its cache until the window expires or the device reconnects. Pick a maximum offline age per threat model, make it visible, and apply revocation before any sync or key release at the next connection. Managed users should get an administrator-set finite limit. If you offer an unlimited consumer mode, tell the user that remote revocation and policy changes wait for the next reconnect.

First use of a device is always a connected sign-in. It enrolls the device, downloads the encrypted state, and creates the local unlock material. A device that was never provisioned online must never create offline capability, and a network error must never let you skip a required connected step.

A state machine where a stale device never prompts

Write the states down, so a missing transition shows up in review:

UNENROLLED
 └─ connected sign-in + MFA + policy + initial sync ─▶ LOCKED_CURRENT

LOCKED_CURRENT
 ├─ freshness valid, then valid local factor ────────▶ UNLOCKED
 ├─ successful check-in ─────────────────────────────▶ LOCKED_CURRENT (freshness renewed)
 ├─ freshness expires ───────────────────────────────▶ LOCKED_STALE
 ├─ check-in reports device/user revocation ─────────▶ UNENROLLED
 └─ logout/user removal ─────────────────────────────▶ UNENROLLED

UNLOCKED
 ├─ timeout/background/device lock/manual lock ──────▶ LOCKED_CURRENT
 ├─ successful check-in ─────────────────────────────▶ UNLOCKED (freshness renewed)
 ├─ freshness expires ───────────────────────────────▶ LOCKED_STALE
 ├─ check-in reports device/user revocation ─────────▶ UNENROLLED
 └─ logout/user removal ─────────────────────────────▶ UNENROLLED

LOCKED_STALE
 ├─ successful connected reauthentication/check-in ──▶ LOCKED_CURRENT
 ├─ check-in reports device/user revocation ─────────▶ UNENROLLED
 └─ logout/user removal ─────────────────────────────▶ UNENROLLED

A check-in is any authenticated server call that returns current policy, revocation state, and server time. A background call with a refresh token is enough. If policy also wants full reauthentication with MFA after some maximum age, put that in the policy record, not in app logic.

Evaluate freshness before asking for a factor. A stale device shows the reconnect screen and never runs the password KDF, a PIN check, or a biometric prompt. When the window has expired, a biometric prompt that is bound to fail, followed by a generic “wrong password” error, tells the user nothing about the fix.

Losing one local factor does not change state. When a biometric wrapper is invalidated, or the PIN hits its threshold, delete that wrapper and continue with the others.

Wrap keys; never encrypt data with the password

Use envelope encryption:

Generate a random 256-bit account data key (ADK).
Generate a random key per record and per file and wrap each under the ADK. Do not derive them from the ADK. A derived key can always be derived again, so you cannot destroy or rotate it for one object without rotating the ADK.
Encrypt record fields, search indexes, file metadata, thumbnails, and file content with an AEAD: AES-256-GCM or ChaCha20-Poly1305.
Wrap the ADK once per unlock method: password, biometric, optional PIN, recovery key.
Encrypt the password and PIN wrappers a second time under a non-exportable hardware key that needs no user authentication: Secure Enclave on iOS, Android Keystore on Android.
Put version, algorithm, KDF parameters, key ID, account, object ID, field ID, and purpose into the authenticated data.

The field ID in step 6 looks like overkill until a field moves. If an attacker can swap the bank-account field of supplier record A with that of record B and the AEAD still verifies, the app shows a payment destination nobody entered for that supplier. In an autofill provider, the same swap between URL fields offers entry A’s password to site B.

A whole-database encryption layer can hide metadata and WAL contents. It does not replace per-field AEAD, because it usually protects only while the database is closed.

Why the password path needs the device too

The threat model that changes the design is the attacker with a copied sandbox or backup. Assume they hold the database, salts, KDF parameters, wrapped keys, and preferences. If the password wrapper is only KDF(password), they can run the KDF on GPUs, away from the device, for as long as they like.

Step 5 fixes that. With the wrapper sealed by a device key that never leaves the Secure Enclave or Keystore, every guess has to run on the original phone, at its speed. The KDF becomes the second barrier, not the only one. A secret stored in plaintext next to the copied files is not a second factor and does not count as one.

For the KDF itself:

Prefer Argon2id with a random salt and versioned parameters, benchmarked on the weakest supported device. It is not part of CryptoKit, CommonCrypto, or the standard Android crypto APIs, so it is a third-party dependency with its own review, pinning, and update process.
If you have to use PBKDF2, both platforms ship it: CCKeyDerivationPBKDF and SecretKeyFactory.getInstance("PBKDF2WithHmacSHA256"). Calibrate the iteration count on real devices instead of copying one from another project.
Store the parameters with the envelope and migrate them after a successful unlock.
Derive separate keys for local wrapping and for any protocol authentication, with domain separation.
On iOS, hold keys in CryptoKit SymmetricKey, which zeroes its storage on release, not in Data that the runtime may copy. A password captured as a Swift String cannot be wiped; turn it into bytes for the KDF right away.

KDF cost does not stop a patient person typing guesses into the unmodified app, so add an increasing delay and a persisted failure counter, and require a connected sign-in after a threshold.

A six-digit PIN is a million candidates

An app PIN is a convenience secret in front of the password. It unlocks its own envelope, and that envelope has to be bound to a non-exportable platform key. Six digits are 10^6 candidates. An unbound PIN envelope falls to offline search, whatever the KDF costs.

The rest of the PIN rules:

At least six digits; off by default for high-assurance or managed profiles.
Never store a PIN verifier that is cheaper to attack than the envelope.
A failure counter scoped to the user that survives process death and reboot, with increasing delay and at most 10 consecutive failures, the upper bound NIST SP 800-63B sets for activation secrets.
At the threshold, delete the PIN wrapper and ask for the password. Wiping the local data would destroy the only copy of anything not yet synced.

The catch: neither iOS nor Android enforces a retry limit in hardware for an app-defined PIN. NIST’s limit assumes the authenticator enforces it, and here the app does. The counter stops the user and a thief using the unmodified app. On a rooted or jailbroken device, an attacker can query the platform key directly at hardware speed, and only PIN length helps against that.

The biometric prompt must unlock a key

Biometric success has to authorize the cryptographic operation that unwraps the ADK. If it only flips an isAuthenticated flag, a hooked callback on a rooted device skips it.

Android: pass a `CryptoObject`

Generate the key inside Keystore and require authentication for every use:

val spec = KeyGenParameterSpec.Builder(
 "offline-data-wrap",
 KeyProperties.PURPOSE_ENCRYPT or KeyProperties.PURPOSE_DECRYPT,
)
 .setKeySize(256)
 .setBlockModes(KeyProperties.BLOCK_MODE_GCM)
 .setEncryptionPaddings(KeyProperties.ENCRYPTION_PADDING_NONE)
 .setUserAuthenticationRequired(true)
 .setUserAuthenticationParameters(0, KeyProperties.AUTH_BIOMETRIC_STRONG) // API 30+: every use prompts
 .setInvalidatedByBiometricEnrollment(true)
 .setUnlockedDeviceRequired(true) // API 28+
 .setIsStrongBoxBacked(true) // API 28+; on StrongBoxUnavailableException, retry in the TEE
 .build()

KeyGenerator.getInstance(KeyProperties.KEY_ALGORITHM_AES, "AndroidKeyStore")
 .apply { init(spec) }
 .generateKey()

Then hand the initialized cipher to BiometricPrompt, and decrypt only with the cipher you get back:

val cipher = Cipher.getInstance("AES/GCM/NoPadding")
try {
 cipher.init(Cipher.DECRYPT_MODE, wrappingKey, GCMParameterSpec(128, envelope.iv))
} catch (e: KeyPermanentlyInvalidatedException) {
 deleteBiometricWrapper() // enrolled biometrics changed: fall back to the password
 return
}

val prompt = BiometricPrompt(activity, executor, object : BiometricPrompt.AuthenticationCallback() {
 override fun onAuthenticationSucceeded(result: BiometricPrompt.AuthenticationResult) {
 val unlocked = result.cryptoObject!!.cipher!!
 unlocked.updateAAD(envelope.aad)
 val adk = unlocked.doFinal(envelope.ciphertext)
 // ...
 }
})

prompt.authenticate(
 BiometricPrompt.PromptInfo.Builder()
 .setTitle("Unlock offline data")
 .setAllowedAuthenticators(BiometricManager.Authenticators.BIOMETRIC_STRONG)
 .setNegativeButtonText("Use password")
 .build(),
 BiometricPrompt.CryptoObject(cipher),
)

Set the accepted authenticators explicitly. A default that includes DEVICE_CREDENTIAL means the device PIN opens the offline data, and so does anyone who watched the owner type it. If you want that fallback, make it a written product decision. Since this is a symmetric key with authentication required, creating the wrapper at enrollment also needs a prompt; ask for the password first, as with any new factor.

iOS: a Secure Enclave key and ECIES

Generate the wrapping key in the Secure Enclave with biometric access control, and wrap the ADK with ECIES:

var error: Unmanaged<CFError>?
let access = SecAccessControlCreateWithFlags(
 nil,
 kSecAttrAccessibleWhenPasscodeSetThisDeviceOnly, // the OS deletes it if the passcode is removed
 [.privateKeyUsage, .biometryCurrentSet], // no passcode fallback
 &error
)!

let attributes: [String: Any] = [
 kSecAttrKeyType as String: kSecAttrKeyTypeECSECPrimeRandom,
 kSecAttrKeySizeInBits as String: 256,
 kSecAttrTokenID as String: kSecAttrTokenIDSecureEnclave,
 kSecPrivateKeyAttrs as String: [
 kSecAttrIsPermanent as String: true,
 kSecAttrApplicationTag as String: Data("offline-data-wrap".utf8),
 kSecAttrAccessControl as String: access,
 ],
]
let privateKey = SecKeyCreateRandomKey(attributes as CFDictionary, &error)!

let wrapped = SecKeyCreateEncryptedData(
 SecKeyCopyPublicKey(privateKey)!,
 .eciesEncryptionCofactorX963SHA256AESGCM,
 adk as CFData,
 &error
)

To unwrap, fetch the key with a fresh LAContext passed as kSecUseAuthenticationContext, with touchIDAuthenticationAllowableReuseDuration left at 0, and call SecKeyCreateDecryptedData. The prompt the user sees then authorizes this key use. That is the iOS counterpart of Android’s per-use authentication.

Two details to get right:

The accessibility class goes into SecAccessControlCreateWithFlags. It cannot be combined with a separate kSecAttrAccessible attribute on the same item.
.biometryCurrentSet invalidates the key when Face ID or Touch ID enrollment changes. Handle that by deleting the wrapper and requiring the password. A .devicePasscode or .userPresence fallback lets anyone who knows the device passcode into the offline data.

Data Protection classes that fail after lock

Set a Data Protection class per store instead of relying on container defaults. The complete class key is discarded about 10 seconds after the device locks, which bites in three places:

Plaintext temporaries should use complete.
Background downloads of offline files need completeUnlessOpen.
A SQLite database in WAL mode under complete fails when anything touches it after lock. Either stop database access on protectedDataWillBecomeUnavailable, or use completeUntilFirstUserAuthentication for the database and rely on field-level AEAD.

Files in tmp/ and elsewhere in the container default to completeUntilFirstUserAuthentication unless you set otherwise.

Offline expiry needs a clock that counts sleep

Enforcing maximum offline age with wall-clock time alone fails the moment someone sets the clock back. I covered that half in An offline cache TTL needs a clock-rollback policy . The research added three things.

The freshness record has two parts. A server-signed part (check-in time, policy version, expiry duration), verified with the server’s public key. A local part (wall time at check-in, monotonic time at check-in, boot reference, accumulated elapsed time), authenticated with a keystore device key. Do not protect it with the ADK: freshness is evaluated before accepting any factor, so the record must verify while the app is locked.

The monotonic clock must keep counting while the phone sleeps, because a phone is asleep most of the time:

Platform	Use	Stops during sleep
iOS	`mach_continuous_time()`, `clock_gettime_nsec_np(CLOCK_MONOTONIC_RAW)`	`mach_absolute_time()`, `ProcessInfo.systemUptime`
Android	`SystemClock.elapsedRealtime()`	`SystemClock.uptimeMillis()`

Verify the chosen clock on a device by letting it sleep.

Reboots reset the monotonic clock. Android has Settings.Global.BOOT_COUNT. iOS has no documented boot identifier: kern.boottime moves when the user changes the wall clock, so it detects clock changes, not boots. On iOS, detect a reboot by a monotonic reading lower than the stored one. Then:

update accumulated elapsed time on every foreground, unlock, and lock

same boot: elapsed from the monotonic clock, cross-checked against wall time
after reboot: elapsed = max(accumulated + monotonic_since_boot,
 wall_now - wall_at_checkin)
wall_now < last stored wall time → rollback → stale

A rollback across a reboot can then hide at most the time between the last update and the reboot.

Fail closed on a missing, rolled-back, or inconsistent record, and exclude it from backup. A restore, to another device or to the same one, then leaves the record missing and forces a check-in. Otherwise, a same-device restore brings back an older valid record together with the ThisDeviceOnly Keychain items it belongs to.

These controls resist clock rollback, not state rollback. Nothing stops someone from replacing the current record with an older valid copy, and the expiry is application logic. Like the PIN counter, it binds the unmodified app, not an attacker running their own code against the device keys.

Documents: chunked AEAD and plaintext that outlives the app

Offline documents, such as invoice PDFs or scanned contracts, need more than per-record encryption. CryptoKit’s AES.GCM and ChaChaPoly seal a whole message in memory. Streaming playback and seeking viewers need chunks, and naive chunking lets an attacker truncate, reorder, or drop chunks without failing authentication. Use a format in the style of the STREAM construction:

fixed-size chunks (say 64 KiB), each sealed separately;
a nonce built from a random per-file prefix and the chunk index;
file ID, format version, chunk index, and a final-chunk flag in each chunk’s authenticated data;
an authenticated header with the prefix, chunk size and version;
a new per-file key or nonce prefix whenever the file changes, so no key and nonce pair is ever reused.

Prefer decrypting into memory or a pipe for an in-app viewer. QLPreviewController wants a file URL, so some formats force a plaintext file. When one is unavoidable, put it in a dedicated private directory under complete, with an unpredictable name, excluded from backup and indexing, shared only through a narrow, non-exported FileProvider grant on Android.

The trap is cleanup. A suspended iOS app runs no code, so no timer deletes the file on schedule, and cancellation, render errors, crashes, process death, and reboot all skip view-level cleanup. Make the launch path enumerate and delete expired temporaries before it allows an unlock.

Lock, logout, and Keychain items that survive uninstall

Lock keeps the encrypted state and destroys the usable keys: release the ADK and derived keys, close plaintext streams, blank the app-switcher snapshot, invalidate decrypted caches, and stop background jobs from quietly recreating an unlocked session. A suspended app cannot clear the clipboard later, so set the expiry when you copy:

UIPasteboard.general.setItems(
 [[UTType.utf8PlainText.identifier: secret]],
 options: [.localOnly: true, .expirationDate: Date().addingTimeInterval(60)]
)

.localOnly keeps it off Universal Clipboard. On Android, mark the clip with ClipDescription.EXTRA_IS_SENSITIVE (API 33+) and clear it with ClipboardManager.clearPrimaryClip().

Logout makes the state permanently unusable. Physical overwrite on flash is unreliable, so the primary boundary is cryptographic erasure: destroy every usable wrapping key, then delete tokens, databases with their WAL and journal files, indexes, cached files, temporaries, pending operations, and shared-container state.

On iOS, uninstalling the app deletes its container but leaves its Keychain items, and no app code runs during uninstall. Keep a marker file in the container. When it is missing at launch, delete every Keychain item the app owns, shared access groups included, before doing anything else. Otherwise, a reinstall can find the previous install’s wrappers.

Autofill is offline unlock under another name

If the cached data includes credentials and the app offers an autofill provider, that provider releases secrets from the local cache with no server call, so every rule above applies to it. The platforms constrain it differently.

On iOS, the Credential Provider extension is a separate process with its own unlock session. It sees the app’s state only through a shared Keychain access group and an App Group container, so every key and wrapper it needs, including the Secure Enclave device key and the freshness record, must live in the shared group. It runs under a memory limit Apple describes as only slightly lower than an app’s. That limit caps the Argon2id memory parameter, so benchmark inside the extension on the weakest device. Do not give the extension a cheaper copy of the password envelope: the weakest wrapper sets the strength of all the data it protects. If parameters that fit are too weak, let the extension offer biometrics only and send password users to the app.

An LAContext evaluated in the app does not carry over to the extension. Within one request, the extension can pass one evaluated context to several Keychain reads, so a single prompt covers a list. The system may reuse the extension process for the next request, so nothing decrypted may stay in a static.

On Android, the AutofillService runs in the app’s own process unless the manifest says otherwise. Its authentication activity still needs a CryptoObject, and it should accept the same authenticators as the app.

Rules for both:

Give the provider its own key that wraps only what a fill needs (username, password, URL), so a faulty extension cannot decrypt documents, notes, or indexes.
Share the failure counter between app and provider. Otherwise, an attacker doubles the allowed attempts by alternating between them.
The provider reads freshness and never checks it itself. When stale, it fills nothing and says which app to open.
On iOS, the identity store shows registered usernames next to their domains in the QuickType bar to anyone holding the unlocked phone, with the app locked. Replace the whole identity set after each sync instead of removing and re-adding the current ones, so deleted entries disappear, and consider a managed policy that registers none.

Verify who is asking

The requester check is where autofill phishing happens (Aonzo et al., Phishing Attacks on Modern Android, CCS 2018 ). iOS identifies websites and apps through associated domains. On Android, the provider has to do it:

Verify the requesting app’s signing certificate against the domain’s Digital Asset Links , not its package name. An app named com.examplebank.evil is not Example Bank.
Trust browsers by package name and certificate digest together. On a device without a listed browser, a sideloaded app can take that browser’s package name and report any web domain.
Derive registrable domains from the Public Suffix List , not a hand-written list of suffixes. Otherwise, an entry saved for victim.github.io collapses to github.io and fills in attacker.github.io.

Users without a password

Everything above assumes every user has a password. Users who sign in through enterprise SSO or a partner’s authenticator app do not. In this design, the password has five jobs:

the baseline wrapper of the ADK;
the fallback when a biometric wrapper is invalidated;
the authorization for a new biometric wrapper;
the replacement when the PIN hits its threshold;
the recent password entry that sensitive actions require.

Each job needs a new owner. The options I found:

Connected sign-in takes every fallback role. This one always applies. A user who loses their last local factor while offline has no offline access until they reconnect.
A mandatory app passcode becomes the baseline factor. Set it during connected provisioning and wrap the ADK with it like the password (KDF, then device key). Three PIN rules change. Six digits stop being enough, because nothing stands behind the passcode: at 0.5 seconds per guess on a compromised device, 10^6 candidates take under six days. Allow alphanumeric passcodes with a real minimum length. The threshold sends the user to connected sign-in, and choosing a new passcode needs one too. The user gets a second secret to remember; that is the price of an offline knowledge factor.
The device passcode can back up the biometric wrapper. It avoids losing offline access at the first biometric change and reopens the “watched the owner type it” risk. Make it explicit, let administrators forbid it, and check on a device whether iOS still releases a biometryCurrentSet or devicePasscode item through the passcode after enrollment changes.
A recovery key can have enough entropy that a copied wrapper is safe even without device binding. Users lose these, and nobody can reset one, so offer it only where a lost factor can cost data.

What does not work is anything the identity provider issued. A cached refresh token or ID token proves the device holds a token, not that the user is holding the device. Don’t wrap the ADK under it, and delete the identity provider’s token cache after the server exchange.

Check one thing in your own protocol. If the server opens a passwordless user’s data without any secret from the user, it can provision the ADK again at the next sign-in. A lost factor then costs offline access, not data, as long as offline records stay read-only. Unsynced offline edits are the exception, because they exist only on the device.

Decide the factor set at provisioning from the login method, and store it with the policy. If the same user later provisions through a password login, re-enroll; don’t bolt a password wrapper onto a passwordless enrollment.

Caveats

The ADK’s scope is a protocol decision. If the ADK only protects this device’s cache, logout and revocation destroy it. If it is the user’s end-to-end key shared with the server and other devices, a device revoked while offline keeps a copy that decrypts future data, and revoking it means rotating the key.
Offline edits are a separate project. Ship read-only offline records first. Editable mode needs an encrypted operation log separate from server-confirmed state, conflict rules that keep both versions when a merge could lose data, and an authorization recheck for each operation on reconnect.
On rooted and jailbroken devices, hardware-backed keys and integrity signals raise the cost. Retry counters and expiry are app logic and don’t bind an attacker who runs their own code against the keys. Don’t claim otherwise.

Test the failure paths

Happy-path testing says nothing about whether keys and plaintext stay contained. The release gates I would not skip:

Area	Test
Provisioning	A never-enrolled device fails offline; restoring a backup to another device, or an older backup to the same one, forces a check-in
Crypto	Copied password and PIN wrappers cannot be opened off the device; ciphertext swapped between records or between fields of one record fails authentication
Biometrics	Add and remove enrolled biometrics, change and remove the passcode, reboot, process death
Retries	Counters survive relaunch and reboot; the PIN threshold removes only the PIN wrapper
Freshness	Clock forward and back, time zone change, reboot, a long sleep counted toward expiry, a replayed older record, the exact boundary; a stale device never prompts
Documents	Truncated, reordered, and dropped chunks fail; kill the app mid-preview and confirm the temporary is gone before the next unlock
Deletion	Uninstall and reinstall leave no usable Keychain state; removing one user leaves another intact
Autofill origin	A look-alike package name gets no match; `victim.github.io` is not offered on `attacker.github.io`

OWASP MASVS and MASTG , NIST SP 800-63B , NIST SP 800-132 for PBKDF2, and RFC 9106 for Argon2id set the baseline. They don’t decide where your keys live or what happens when a preview crashes.

Takeaway

Offline, the app can only answer “is this the enrolled user, on the enrolled device, within the allowed window?” Every unlock path, including the password, should require the device’s hardware key; every biometric prompt should gate a key operation; and the window should be measured by a clock that keeps counting while the phone sleeps.

If your app already has an offline mode, copy its sandbox to another machine and try to open the password wrapper there. If that attempt gets as far as a KDF, the password path is missing its device key.

Backing up ~/.claude and ~/.codex without archiving your AI logins

Sat, 31 Oct 2026 22:29:00 +0100

If you back up the whole ~/.claude or ~/.codex folder, the restore works, and the backup now holds everything an infostealer wants. The same folder holds your settings, every session transcript in plaintext, and, on Linux, the login itself:

~/.claude/.credentials.json # Claude Code login, mode 0600
~/.codex/auth.json # Codex login when file storage is in use
~/.claude/projects/*/*.jsonl # transcripts: prompts, tool output, file contents

Split the folder into two lists. Back up what you wrote: settings, instructions, skills, hooks, and memory. Keep transcripts as optional encrypted data. Leave login files out of routine backups and log in again after a restore.

Where AI client state lives on macOS and Linux

I checked these paths against Claude Code 2.1.316, Claude Desktop for Linux 4.4212.0, ChatGPT for macOS 1.2026.183, the Codex desktop app for macOS 26.1027.30824, and Codex source at commit 67f0994. “Documented” means the vendor docs name the path. “Observed” means I found it by inspecting the official build so that it can change in any release.

Client	State root	Login material	Evidence
Claude Code	`~/.claude/` and `~/.claude.json`, both under `CLAUDE_CONFIG_DIR` if set	Keychain on macOS, or `~/.claude/.credentials.json` when the Keychain rejects the write (locked in an SSH session, for example). On Linux, always that file	Documented
Claude Desktop	macOS: `~/Library/Application Support/Claude/`. Linux (beta): `${XDG_CONFIG_HOME:-~/.config}/Claude/`	Electron cookies and Local Storage, extension secrets encrypted with `safeStorage`	MCP config documented, rest observed
ChatGPT desktop, macOS only	`~/Library/Application Support/com.openai.chat/`, `~/Library/Group Containers/2DC432GLL2.group.com.openai.chat/`, `~/Library/Preferences/com.openai.chat.plist`	Keychain items in OpenAI’s access group	Observed
Codex desktop, macOS only	`~/Library/Application Support/Codex/`, plus `~/.codex`	Shared with Codex CLI	Observed
Codex CLI	`${CODEX_HOME:-~/.codex}`. `CODEX_SQLITE_HOME`, `sqlite_home` and `log_dir` can move parts of it	`auth.json` in `file` mode or as the `auto` fallback; `.credentials.json` for MCP OAuth without a keyring	Documented and source

Desktop sessions that run Claude Code write to ~/.claude/, not to the Electron folder, so include both.

What to back up and what to leave out

Treatment	Claude Code	Codex
Back up, versioned	`settings.json`, `CLAUDE.md`, `rules/`, `skills/`, `commands/`, `agents/`, `workflows/`, `output-styles/`, `keybindings.json`, `projects/*/memory/`, `agent-memory/`, `plugins/installed_plugins.json` as a manifest	`config.toml`, `*.config.toml`, `hooks.json`, `rules/`, `skills/`, agent-role files
Back up after a secret check	`~/.claude.json` (personal MCP servers, trust decisions)	MCP `env` blocks in `config.toml`
Optional, encrypted, short retention	`projects//.jsonl`, `subagents/`, `tool-results/`, `history.jsonl`	`sessions/YYYY/MM/DD/rollout-*.jsonl`, `archived_sessions/`, `history.jsonl`
Exclude	`.credentials.json`, `file-history/`, `paste-cache/`, `debug/`, `shell-snapshots/`, `session-env/`, `backups/`	`auth.json`, `.credentials.json`, `log/`, `logs_2.sqlite`, `shell_snapshots/`, `packages/`, caches

For Claude Desktop, keep claude_desktop_config.json after checking its MCP env blocks, plus a list of installed extensions and where each came from. Reinstall extensions instead of restoring their executable caches. The ChatGPT app’s local databases are caches of a server-side account; use the account’s data export for conversations you want to keep.

Why copying the whole folder is the wrong default

Transcripts are plaintext. Anthropic’s docs say that anything passing through a tool is written to the session .jsonl, so a .env file a tool reads or a token a command prints is in there. Codex rollouts record tool activity the same way.

Your backup also keeps transcripts after the client has deleted them. Claude Code deletes transcripts after cleanupPeriodDays (default 30), but keeps sessions started, or last continued in Claude Desktop or Cowork at any age unless you set desktopSessionCleanupPeriodDays. A backup with a year of history still holds a transcript eleven months after the client deleted it.

A restore can also run code. Hooks, MCP server commands, skills, plugins, desktop extensions, and shell snapshots all execute something. If malware edited them, a full restore puts it back on the clean machine.

The Keychain protects less than it seems to. It stops someone who copies the disk offline, but malware running as your user can still request the item. The Linux desktop build can also fall back to Chromium’s basic password store when no wallet is running.

Dead ends worth skipping

Guessing Linux paths from macOS ones. Replacing Library/Application Support with .config works for Claude Desktop, which has a Linux beta. OpenAI ships no Linux desktop app for ChatGPT or Codex. The Codex source has Linux branches, but there is no supported Linux build. On Linux, your ChatGPT state lives in the browser profile.
Copying only the .sqlite files. Codex keeps state in state_5.sqlite, thread_history_1.sqlite, memories_1.sqlite and others . A live copy without the -wal and -shm sidecars can be incomplete or corrupt. Stop the client and copy all three.
Treating the desktop database as the archive. The Codex desktop app can rebuild its index from the rollout JSONL files, but metadata that lives only in the database is lost, back up the JSONL files.
Assuming claude purge cleaned everything. It doesn’t touch shell-snapshots/, backups/ (older copies of ~/.claude.json) or files written by /heapdump. According to Anthropic, a heap snapshot contains the full conversation and any credentials the process held.

Restore into a staging profile

Reinstall the signed apps, plugins, and extensions from their sources.
Restore your config, then read every hook, MCP command, and executable path before the first launch.
Restore transcripts or memory only if you need them.
Log in again. Don’t restore auth.json, .credentials.json, cookies, or Keychain exports.
Resume one session and confirm that it uses the right account and repository.

Verify the backup set

To list what exists and how large it is, without printing any contents, run this on the machine you back up:

for p in "${CLAUDE_CONFIG_DIR:-$HOME/.claude}" "$HOME/.claude.json" \
 "${CODEX_HOME:-$HOME/.codex}" \
 "${XDG_CONFIG_HOME:-$HOME/.config}/Claude" \
 "$HOME/Library/Application Support/Claude" \
 "$HOME/Library/Application Support/Codex" \
 "$HOME/Library/Application Support/com.openai.chat" \
 "$HOME/Library/Group Containers/2DC432GLL2.group.com.openai.chat"; do
 [ -e "$p" ] && du -sh "$p"
done

Then check that no login file ended up in the backup:

find /path/to/mounted/backup \( -name .credentials.json -o -name auth.json \) -print

It should print nothing. If it prints anything, that backup is a credential store: encrypt it, shorten its retention, or rotate the logins.

If malware may already have run

Don’t treat a restore as the cleanup. From a clean device, revoke your Claude and ChatGPT sessions, then rotate API keys, MCP and connector credentials, and any secret that appeared in a transcript or in tool output. Any backup the malware could reach counts as exposed. Rebuild the user profile, restore only config you have reviewed, and log in again.

Caveats

Paths and retention rules change between releases. The ChatGPT and desktop paths come from build inspection, not from a published schema, so check them against your installed versions before you write a backup job. Sources: the .claude directory reference , Claude Code credential management , Codex config and state locations , Codex credential storage , Claude Desktop MCP config paths , and Electron’s app.getPath() .

Back up the config you wrote, and leave out the files the app uses for login. Run the find check above against your latest backup tonight.

Swift Testing withKnownIssue can hide snapshot failures

Fri, 30 Oct 2026 19:45:00 +0100

Two snapshot tests passed while reporting:

No reference was found on disk. Automatically recorded snapshot:

Their Swift Testing withKnownIssue wrapper accepted every issue inside the snapshot assertion, including missing references. Adding a matching: closure limited the exception to expected dependency reports. A deliberate wrong-state snapshot then failed, even though those reports remained known issues.

Restrict which issues count as known

The view had two states: a progress indicator before a check completed, and content afterward. Rendering the content also touched an unimplemented test dependency. The snapshot helper wrapped rendering and comparison in withKnownIssue(isIntermittent: true) to tolerate that report.

Without a matcher, the wrapper also accepted issues from the assertion it surrounded. The initial recording run exited successfully: two tests passed with three known issues. Two of those issues said references were missing; only one was the dependency report the wrapper was meant to tolerate.

The historical fix added this predicate:

String(describing: issue).contains("Unimplemented")

Here is the same policy as a generalized synchronous helper. Pass an existing, nonthrowing snapshot assertion as its body:

import Testing

func withExpectedDependencyIssue(_ body: () -> Void) {
 withKnownIssue(isIntermittent: true) {
 body()
 } matching: { issue in
 String(describing: issue).contains("Unimplemented")
 }
}

The public withKnownIssue matching overload takes a predicate over Issue. Returning false leaves an issue outside this exception. The synchronous overload is rethrows; this helper needs no try because its body is nonthrowing. A throwing assertion needs corresponding throwing handling.

isIntermittent: true allows the body to record no matching issue. That was useful for the progress state, which did not touch the dependency. It does not restrict which issues match. Without matching:, an intermittent wrapper still accepts everything recorded in its scope.

A mutation must change the rendered output

The first attempted verification changed a prefilled field. Both snapshot tests still passed. That was an ineffective mutation for this check; it did not establish that the assertion could detect the intended visual regression.

The stronger mutation completed the check before constructing the view in the test for the before-check reference. It deliberately compared the content state against the progress-state baseline.

That run exited with code 65. The before-check test failed, while the after-check test passed with its expected known issue. Across the two tests, the runner reported three issues, including two known issues. Restoring the original setup produced a successful run of 239 tests in 24 suites with one known issue.

This shows that the narrowed helper rejected an incorrectly rendered state. The session did not run that same wrong-state mutation against the original broad wrapper, so it does not provide an identical before-and-after mutation comparison. The missing-reference reports already demonstrate the broad wrapper’s excessive scope.

Keep the exception smaller than the assertion

The repair used here matched a description substring. It remains brittle: another report containing Unimplemented would also match, and diagnostic wording can change. Where the reporting library exposes a stable public discriminator, prefer that over a broad text search. Better still, provide the missing test dependency so the assertion needs no exception.

Keep that follow-up separate from what was verified. Other snapshot helpers retained broad wrappers; this change did not repair them. A passing suite using this helper says nothing about their failure detection.

These results came from the original Xcode simulator runs. The generalized helper follows Swift Testing’s public matching overload ; it has not been compiled separately.

Before trusting a snapshot helper with a known-issue scope, force a visibly different state against an existing reference and require a failing run. Restore the state and require green again.

A shared Swift timeout left an earlier task without a deadline

Fri, 30 Oct 2026 17:37:00 +0100

A Swift service had a timeout, but waiting for its in-flight work could still hang. A held-fetch regression test reached Swift Testing’s outer limit:

Time limit was exceeded: 60.000 seconds

Starting another check replaced the shared timer while the earlier fetch kept running. The fix was to give each tracked operation its own cancellation deadline and cancel that deadline when the operation ended. That still depends on the fetch cooperating with cancellation.

One timer could not cover overlapping fetches

The service could start another asynchronous check before the previous one finished. A separate tracker retained the in-flight tasks so a caller could wait for their completion. Its waiter took a snapshot of those tasks and awaited their results.

The mistaken assumption was that the service’s existing timeout covered everything in that snapshot. Review found that starting a check reset the shared timer without canceling the previous fetch. Finishing a check also reset the timer so that an older completion could interfere with a newer check’s deadline.

An illustrative sequence makes the ownership mismatch visible:

start A tracked: A shared timer: A
start B tracked: A, B shared timer: B; A's timer reset
finish B tracked: A shared timer: reset
wait for A A still running no cancellation deadline for A

The tracker remembered more work than the timer covered. Making that timer longer would leave the same mismatch.

Put the deadline beside the operation

The correction belonged where the tracker created each fetch task. That scope already knew which task it owned and when it finished.

For each operation, create a separate deadline task that sleeps for the configured duration, then calls cancel() on that operation’s task handle. Keep the deadline handle local. On exit, cancel the deadline and remove the operation from tracking. Forward cancellation from the caller to the fetch as well.

The resulting ownership is:

operation A owns deadline A; completion A cancels deadline A
operation B owns deadline B; completion B cancels deadline B

Canceling a deadline’s sleep must end the deadline task. Swallowing the sleep’s cancellation error and continuing to cancel() would turn normal cleanup into an unintended cancellation request.

Keeping the handles local also means A’s cleanup cannot accidentally cancel B’s deadline through a shared slot.

Prove that the waiter needs this deadline

The regression test used a fetch held until cancellation, a short injected operation timeout, and a signal confirming that the fetch had started. Nothing outside the tracker canceled it. The test then awaited completion and checked that the fetch ended before the waiting caller continued.

Before the implementation used the injected timeout, the test failed at the one-minute limit. After adding the operation-local deadline, the archived test output reports the held-fetch test passing and the focused test target exiting successfully.

That establishes the missing-deadline behavior for a cooperative fake. It does not reproduce the entire overlapping-check sequence live. For your own tracker, also hold A open while B starts and finishes, then confirm that A still receives its own cancellation request.

A cancellation deadline is not a hard timeout

Swift’s Task.cancel() documentation explicitly says cancellation is cooperative. Calling it flags the task and runs cancellation handlers; arbitrary work does not automatically stop. A fetch that ignores cancellation can outlive this deadline and keep a completion waiter suspended.

Later work therefore separated the operation’s cancellation deadline from the caller’s user-facing wait budget. It also added recovery after an unsuccessful check. Giving every operation a deadline did not settle how long a caller should wait or whether continuing after failure was safe.

Before relying on a service timeout, start two operations and finish the newer one first. Check that the older operation still has its own cancellation deadline.

SwiftUI large title disappears after tapping the selected tab again

Fri, 30 Oct 2026 14:38:00 +0100

The Settings large title disappeared after tapping the already-selected tab again. Its space stayed empty; the small title still appeared when scrolling, and the accessibility check passed:

titleBefore=true titleAfter=true

An opaque navigation-bar background was painting over the title. The eventual repair removed a screen modifier’s writes to UINavigationBar.appearance(). Hiding the affected screen’s bar background was an intermediate workaround that introduced a font regression.

The title still existed

The prototype varied the tab view, loading state, scroll-view wrapper, and swipe-back helper. Its accessibility existence checks stayed green. Adding the app’s opaque navigation-bar appearance also produced green checks, even though the title vanished visually.

Counting dark pixels in the prototype’s title region exposed the difference:

Appearance	After first tap	After re-tap
Default	12955	12955
Opaque	12955	0

Those are diagnostic counts from one fixed light-background rendering, not a portable visibility threshold. They contradicted the existence assertion directly.

Before-and-after hierarchy dumps from the running app showed the same title view moving from the navigation bar into the scroll view. The bar’s opaque background then covered it. Initially, the screen displayed a loader without a scroll view, so UIKit built the title inside the bar. Re-tapping the selected tab triggered the transition that exposed the problem on the investigated simulator.

This establishes a rendering mechanism in that environment. It does not establish identical title placement on every supported runtime.

The offending modifier intended to remove a divider and set one screen’s background. Its essential mistake looked like this generalized fragment inside a SwiftUI .onAppear closure:

let appearance = UINavigationBarAppearance()
appearance.configureWithOpaqueBackground()
appearance.backgroundColor = .secondarySystemBackground
appearance.shadowColor = .clear

UINavigationBar.appearance().standardAppearance = appearance
UINavigationBar.appearance().scrollEdgeAppearance = appearance

The closure belongs to a screen; the appearance proxy configures all instances of that class. UIKit applies appearance changes when a view enters a window. A different tab’s later navigation bar can inherit those settings, so screen order becomes part of the bug’s setup.

The repair deleted this global writer. The calling screens already declared their backgrounds through SwiftUI’s per-screen toolbar styling; the investigated runtimes did not need a replacement divider-hiding implementation. Their existing snapshots passed without re-recording. Appearance save-and-restore workarounds around a system-presented screen went away with the leak.

A focused test supplied stronger evidence than the original title lookup: show a styled screen, then create another navigation bar and check its background. It failed before the deletion because the next bar inherited a background, then passed afterward. Separate checks guarded the calling screen’s color and the next large title’s font.

Why the one-line hotfix was reversed

The initial workaround used .toolbarBackground(.hidden, for: .navigationBar) on Settings. It kept the prototype’s title visible under the opaque appearance.

The attempted app snapshot never reproduced the original disappearance. The real screen reloaded into a spinner first, unlike the prototype’s scroll-view-first case. Capturing the loaded window still showed the title without the fix. We dropped that test; it couldn’t serve as a regression guard.

Later checks found that hiding the background also made the large title use the system font instead of the intended custom font. A snapshot cropped to the bar failed with that modifier present and passed after its removal. Once the global opaque write was gone, the workaround no longer had a reason to stay.

Verify the interaction, not just the label

Builds and broad test plans passed after the cleanup. Final manual checks in the running app remained open, and older supported runtimes were untested. The font snapshot and next-bar test do not close that gap.

For release verification:

Visit the screens that previously wrote global appearance, then open Settings. Repeat with a different screen order.
Tap the selected Settings tab again. Capture the title before and after, then scroll through its large-to-inline transition.
Check actual title pixels and contrast against the background. Use the hierarchy to explain occlusion; accessibility existence, or isHittable alone, is not universal proof that text is painted visibly.
Repeat on supported runtimes, checking pushed screens’ bar styling too.

Audit screen-level UINavigationBar.appearance() writes before adding another visibility workaround.

Xcode Archive can ship the wrong display name despite correct source settings

Thu, 29 Oct 2026 14:22:00 +0100

A production iOS archive had the production bundle ID and backend environment, but its CFBundleDisplayName was the beta name. The tagged source said otherwise, and App Store Connect never shows that name, so only an installed build revealed it. Add a Run Script phase that checks the processed app Info.plist, so a manual Product > Archive also rejects the wrong identity.

Using synthetic names, the mismatch was:

CFBundleIdentifier = com.example.notebook
CFBundleDisplayName = NotebookBeta

Source checks missed the artifact

The investigation initially ruled out the mismatch from project settings and runtime configuration checks. That conclusion was too strong. The runtime check validated the product variant against the backend environment; it did not validate the display name. A production bundle with a beta display name could satisfy it.

Reading the archived app’s plist settled the question. Its name disagreed with the committed production settings. An uncommitted local edit was the likely explanation, but the archive alone could not establish exactly how the value changed.

The release flow had no step that showed the name either. App Store Connect metadata covers the store listing’s app name, not the bundle’s CFBundleDisplayName, which only appears under the home screen icon. The only check is to install the production build from TestFlight, or from the App Store after release, and read that label by hand. It is easy to overlook when the bundle ID and backend already look right.

Source-level regression tests still help, but manual Archive does not run the scheme’s Test action. A check confined to a deployment lane also misses someone archiving directly in Xcode.

Check the processed plist during the build

Add a Run Script phase named Verify App Identity to the app target, after its existing phases. Use /bin/sh. Leave Run script only when installing unchecked.

Declare this Input File:

$(TARGET_BUILD_DIR)/$(INFOPLIST_PATH)

Declare this Output File:

$(DERIVED_FILE_DIR)/verified-app-identity.stamp

Uncheck Based on dependency analysis for this small guard. It should validate every build, including repeated archives. The input declaration still establishes the dependency on plist processing; visual phase order alone is insufficient. Apple’s Run Script documentation explains how declared inputs and outputs determine task ordering.

Paste this script, adapting the explicit configuration allowlist:

set -eu
: "${SCRIPT_OUTPUT_FILE_0:?Declare the stamp Output File}"
stamp="$SCRIPT_OUTPUT_FILE_0"
rm -f "$stamp"
: "${TARGET_BUILD_DIR:?}" "${INFOPLIST_PATH:?}" "${CONFIGURATION:?}"
plist="$TARGET_BUILD_DIR/$INFOPLIST_PATH"
fail() {
 printf 'error: %s\n' "$*" >&2
 exit 1
}
case "$CONFIGURATION" in
 Release)
 expected_id=com.example.notebook
 expected_name=Notebook
 ;;
 Debug)
 expected_id=com.example.notebook.beta
 expected_name=NotebookBeta
 ;;
 *) fail "No identity policy for configuration $CONFIGURATION" ;;
esac
[ -f "$plist" ] || fail "Missing processed Info.plist: $plist"
id=$(/usr/libexec/PlistBuddy -c 'Print :CFBundleIdentifier' "$plist") ||
 fail 'Cannot read CFBundleIdentifier'
name=$(/usr/libexec/PlistBuddy -c 'Print :CFBundleDisplayName' "$plist") ||
 fail 'Cannot read CFBundleDisplayName'
[ "$id" = "$expected_id" ] && [ "$name" = "$expected_name" ] ||
 fail "$CONFIGURATION requires $expected_id / $expected_name; got $id / $name"
printf 'Verified app identity: %s / %s\n' "$id" "$name"
: > "$stamp"

Do not substitute the source INFOPLIST_FILE: generated keys and build-setting expansion can change the packaged values. Keep the expected identity independent of the settings under test.

The declared paths also describe the script’s file access for user-script sandboxing. Keep sandboxing enabled where possible and test these declarations with your Xcode version. The original target had it disabled. The stamp lives outside the signed app and is created only after validation. Removing it first, failing on missing values, and running unconditionally prevent an old stamp from becoming a false success after a failed check.

Prove that wrong builds fail

The historical implementation passed normal builds across all four original configurations. Four deliberate mutations failed: the production ID, production name, beta ID, and beta name, changed separately. Each failure reported the mismatched identity; the project file was restored afterward. These were build checks, not evidence of a fresh Archive after the fix.

For your project, repeat those negative controls, then restore the settings and archive twice without cleaning. Confirm the guard runs both times and inspect the resulting archive’s app plist. Also test a missing key and an unrecognized configuration.

This checks the app’s base plist only. Localized InfoPlist.strings can affect the displayed name, and extensions need separate policies. Check the Archive action’s selected configuration in your scheme, then deliberately break its display name and confirm Archive fails before trusting the guard.

A latest-row query cannot tell you whether every device is current

Thu, 29 Oct 2026 00:00:00 +0000

A dashboard can label an account “current” while that account still has an outdated device. Selecting the latest event per account and application retains one observation; it discards observations from other devices. Describe that selection in the report before claiming that the source telemetry cannot distinguish devices.

The distinction matters during a security review: “latest observed state” and “any observed device needs attention” answer different questions. Correct arithmetic cannot repair a mislabeled metric.

Reproduce the lost device with synthetic data

Run this example with Python 3 and SQLite supporting window functions. The names, dates, and status values are synthetic; current is a supplied classification, not an implementation of version comparison.

import sqlite3

con = sqlite3.connect(":memory:")
con.executescript("""
CREATE TABLE observations (
 event_id INTEGER PRIMARY KEY,
 account TEXT, app TEXT, device TEXT,
 observed_at TEXT, current INTEGER
);
INSERT INTO observations VALUES
 (1, 'account-a', 'demo', 'laptop', '2026-01-01T09:00:00Z', 0),
 (2, 'account-a', 'demo', 'phone', '2026-01-01T10:00:00Z', 1),
 (3, 'account-b', 'demo', 'tablet', '2026-01-01T10:00:00Z', 0),
 (4, 'account-b', 'demo', 'tablet', '2026-01-02T10:00:00Z', 1);
""")

latest_account = """
WITH ranked AS (
 SELECT *, ROW_NUMBER() OVER (
 PARTITION BY account, app
 ORDER BY observed_at DESC, event_id DESC
 ) AS rn
 FROM observations
)
SELECT account, current FROM ranked WHERE rn = 1 ORDER BY account
"""

latest_devices = """
WITH ranked AS (
 SELECT *, ROW_NUMBER() OVER (
 PARTITION BY account, app, device
 ORDER BY observed_at DESC, event_id DESC
 ) AS rn
 FROM observations
)
SELECT account, MAX(CASE WHEN current = 0 THEN 1 ELSE 0 END)
FROM ranked WHERE rn = 1
GROUP BY account, app ORDER BY account
"""

account_rows = con.execute(latest_account).fetchall()
exposure_rows = con.execute(latest_devices).fetchall()
assert account_rows == [('account-a', 1), ('account-b', 1)]
assert exposure_rows == [('account-a', 1), ('account-b', 0)]
print("latest account state:", account_rows)
print("any latest-observed device outdated:", exposure_rows)

Both accounts have a current latest observation. Only account-a has an outdated device among its latest per-device observations. The laptop disappeared from the first query because the phone reported later.

The account-b rows are a negative control. Aggregating every historical outdated event would flag the tablet even after its newer current observation. Rank within each device first, then aggregate the surviving states.

State the measurement contract

An accurate table label might be:

Accounts classified by their latest observed event per application, with ties resolved by event ID.

That supports a claim about selected observations. It does not establish that every device on those accounts is current. Nor does the query establish what the ingestion system could identify before aggregation.

For an exposure report, define the device identity and observation window explicitly. A stable device key is a prerequisite for the second query. An installation identifier that rotates after reinstalling may represent installations rather than physical devices; a shared or missing identifier can merge unrelated observations. Inspect the schema and collection semantics before choosing the partition keys.

The example orders normalized UTC timestamps and breaks ties with a unique event ID. That makes the selection deterministic. It does not prove that a larger ID represents a later real-world event when timestamps tie. SQLite’s window-function documentation describes how ROW_NUMBER() follows the window’s ordering.

Keep unknown devices out of the healthy bucket

Even latest-per-device aggregation describes observed devices. Retired devices, missing telemetry, and a stale last observation need an explicit policy. Define freshness before calling the result a current fleet inventory, and report unknown states separately instead of treating silence as evidence of health.

A reporting correction may leave every displayed count unchanged: the query can stay the same while its title and interpretation become accurate. Calculating a replacement device-level metric is separate work and requires source data that supports it.

Audit the partition keys behind one account-level health table, then check whether its title describes exactly the rows that survive.

Swift Testing: .serialized did not stop another suite deleting test files

Wed, 28 Oct 2026 14:36:00 +0100

A filesystem test failed while writing its fixture, even though its suite used @Suite(.serialized) and each fixture had a UUID directory. The underlying error was:

NSPOSIXErrorDomain Code=2 "No such file or directory"

Another suite was constructing the application core, whose initializer swept the shared cache directory. Injecting that startup operation, with an inert test value, stopped unrelated tests from deleting each other’s fixtures.

A unique child directory still has a shared parent

The failing suite created a directory, then wrote a file into it. Its own cleanup tests ran serially. Elsewhere in the same Swift Testing process, ordinary tests constructed the core as part of their setup. Every construction triggered a sweep of the common cache parent.

The destructive interleaving was small:

cleanup test: create cache/<UUID>/
unrelated test: construct core -> sweep cache children
cleanup test: write cache/<UUID>/fixture.bin -> missing directory

UUIDs prevented name collisions. They could not protect a child from an operation that deleted every child.

Swift Testing’s parallelization trait documentation describes the scope: .serialized orders a suite’s contained tests and sub-suites. It does not order those tests relative to unrelated suites. Putting the attribute on the failing suite left the other constructors free to run alongside it.

Make ordinary test construction inert

The fix moved the startup sweep behind an injected operation. Production kept the real sweep; the test dependency supplied a no-op. Tests that needed to verify sweeping explicitly selected the real operation.

Here is a generalized closure-based adaptation, independent of a dependency-injection library:

import Foundation

struct StartupSweep {
 var run: () throws -> Void

 static var inert: StartupSweep { StartupSweep(run: {}) }

 static func live(root: URL) -> StartupSweep {
 StartupSweep {
 let manager = FileManager.default
 for child in try manager.contentsOfDirectory(
 at: root, includingPropertiesForKeys: nil
 ) {
 try manager.removeItem(at: child)
 }
 }
 }
}

struct CacheCore {
 init(startupSweep: StartupSweep) throws {
 try startupSweep.run()
 }
}

The production composition root passes .live(root: cacheRoot). The ordinary test factory passes .inert; neither caller relies on an initializer default. This example propagates filesystem errors so a cleanup test cannot pass by silently ignoring a failed sweep.

For a test-owned root containing a prepared fixture file, the two assertions look like this. This is a test-body excerpt using Testing:

// Ordinary construction must preserve the existing fixture.
_ = try CacheCore(startupSweep: .inert)
#expect(FileManager.default.fileExists(atPath: fixture.path))

// Cleanup coverage explicitly opts into the real operation.
_ = try CacheCore(startupSweep: .live(root: root))
#expect(!FileManager.default.fileExists(atPath: fixture.path))

Use a disposable root for this check. Never point the illustrative live operation at a directory containing files you need. Injecting a separate root per test is another way to isolate filesystem work when the design permits it.

Prove the constructor contract, then repeat the broader target

The historical investigation used an isolated-suite control: the cleanup suite repeated 30 times with zero issues. The broader core target failed on its first run. That narrowed the search to interference from outside the serialized suite.

A regression test then created a fixture, built the core-under-test dependencies, and asserted that the file still existed. Before the injection fix, that assertion failed. It converted an intermittent write failure into a direct check of the destructive constructor behavior.

After the fix, the archived broader-target output reported:

exit: 0
** TEST SUCCEEDED **
issues:
0
max repetition:
repetition 30
✔ Test run with 165 tests in 28 suites passed after 2.806 seconds.

Those results verify the original fix under Xcode and iOS Simulator. The closure-based adaptation above has not been compiled separately. The reported test count is the runner’s summary, not a total multiplied by the repetition limit.

When a serialized suite loses files, check every initializer that can sweep their shared parent, including constructors called by unrelated suites.

A CVE cross-reference is not an advisory entry

Wed, 28 Oct 2026 00:00:00 +0000

A vulnerability index can have the correct set of CVE identifiers and still assign them to the wrong advisories. A regex that searches every paragraph also captures narrative references such as “related to CVE-…”. Extract identifiers from declaration blocks, and verify the resulting relationships before using the index to make patch or exposure claims.

The trap is easy to miss during deduplication: the referenced identifier may already have a legitimate entry elsewhere. Removing its false assignment changes the relationships without changing the global distinct-ID count.

How the wrong relationship gets created

Consider two synthetic advisories. Every identifier below is fictional:

Advisory A, component Reader, declares CVE-2099-10001.
Its impact paragraph mentions CVE-2099-10002 as a related issue.
Advisory B, component Writer, declares CVE-2099-10002.

An extractor that carries the current advisory and component into every regex match produces three rows. The false row associates Advisory A and Reader with the second identifier. Both the broken and corrected indexes contain two distinct identifiers.

This error can contaminate a patch matrix: an analyst might attach the wrong affected component or release boundary to a real vulnerability. It requires no malicious input. A crafted narrative mention could produce the same misassociation in a pipeline that ingests untrusted documents, but that is a possible abuse of the parser, not evidence of an exploited advisory feed.

Filter by document role before matching identifiers

For one fixed document layout, a practical repair is to collect complete paragraph text and process only blocks beginning with the declaration marker CVE-. Collecting the whole paragraph matters: identifiers can be split across inline elements, and indentation should not change classification.

Here is a runnable Python example using a deliberately small HTML schema. Save it as check_advisories.py and run python3 check_advisories.py:

import re
from html.parser import HTMLParser

ID = re.compile(r"\bCVE-\d{4}-\d{4,}\b")
FIXTURE = """
<article data-advisory="A" data-component="Reader">
 <p> <strong>CVE-2099-10001</strong> </p>
 <p>Related to <em>CVE-2099-10002</em>.</p>
</article>
<article data-advisory="B" data-component="Writer">
 <p>\n CVE-2099-<span>10002</span> </p>
</article>
"""

class Index(HTMLParser):
 def __init__(self, declarations_only):
 super().__init__()
 self.declarations_only = declarations_only
 self.context = None
 self.parts = None
 self.rows = set()

 def handle_starttag(self, tag, attrs):
 if tag == "article":
 attrs = dict(attrs)
 self.context = (
 attrs["data-advisory"], attrs["data-component"]
 )
 elif tag == "p":
 self.parts = []

 def handle_data(self, data):
 if self.parts is not None:
 self.parts.append(data)

 def handle_endtag(self, tag):
 if tag == "p" and self.parts is not None:
 text = "".join(self.parts).strip()
 if not self.declarations_only or text.startswith("CVE-"):
 for identifier in ID.findall(text):
 self.rows.add((*self.context, identifier))
 self.parts = None
 elif tag == "article":
 self.context = None

def extract(declarations_only):
 parser = Index(declarations_only)
 parser.feed(FIXTURE)
 parser.close()
 return parser.rows

broken = extract(False)
fixed = extract(True)
assert fixed == {
 ("A", "Reader", "CVE-2099-10001"),
 ("B", "Writer", "CVE-2099-10002"),
}
assert broken - fixed == {("A", "Reader", "CVE-2099-10002")}
assert {row[2] for row in broken} == {row[2] for row in fixed}
print(f"relationships: {len(broken)} -> {len(fixed)}; distinct IDs unchanged")

Expected output:

relationships: 3 -> 2; distinct IDs unchanged

A prefix guard has a document contract

This parser is a fixture, not a general advisory importer. A narrative paragraph could itself begin with a CVE identifier; another publisher might declare identifiers in a table or list. For production ingestion, select the publisher’s actual declaration element or structured field, record the source location, and reject or flag layouts the adapter does not recognize. The standard library’s HTMLParser documentation describes its event interface; it cannot infer a publisher’s semantic roles for you.

Keep regression fixtures for narrative references, whitespace, and inline markup. Review added and removed (advisory, component, CVE) tuples during a parser change, including declarations the new selector might accidentally discard.

Before trusting your next vulnerability-index refresh, check one known cross-reference against its owning advisory. A stable distinct-ID total cannot validate that assignment.

An offline cache TTL needs a clock-rollback policy

Tue, 27 Oct 2026 15:44:00 +0100

AAn offline cache can pass its age check after the device clock moves backward. A timestamp later than “now” produces a negative age, and a check that only asks whether age is below the TTL accepts it. Reject future timestamps and remember the latest wall-clock time observed, while recognizing that neither check provides trusted elapsed time.

A negative age passed the upper-bound check

The feature reused a biometric-protected Keychain cache shared with a companion extension. Its initial eligibility rule was effectively:

now - lastWritten <= maxAge

That has two problems. Moving the clock before the write makes the age negative. Moving it backward after a previous access can also make an expired cache look young again without crossing the write timestamp.

The cache contained independently written items. Using the newest item’s date would make an older item appear refreshed, so eligibility used the oldest modification date. A separate check rejected any item dated in the future. Checking only the oldest date would miss a newer, future-dated item.

Retain the latest observed time

The repair added a clock mark in a separate Keychain item. Each eligibility read retained max(previousMark, now). If now preceded the previous mark, access was refused. Writing the rolled-back time into the mark would let the next read pass, so repeated reads were part of the regression test.

This Python model isolates the arithmetic from Keychain storage. Inputs are timestamps in the same unit and a nonnegative maximum age. None represents initial mark creation; it does not distinguish first use from a failed storage read.

def cache_status(written_at, now, mark, max_age):
 if not written_at:
 return "unavailable", mark

 previous = now if mark is None else mark
 updated = max(previous, now)
 fresh = (
 now >= previous
 and max(written_at) <= now
 and now - min(written_at) <= max_age
 )
 return ("fresh" if fresh else "expired"), updated

Persist the returned mark reliably. Define separate behavior for a missing mark, failed reads, and failed writes. Serialize the read/advance/write operation across its writers: two callers storing independently calculated maxima can otherwise overwrite a newer mark with an older one. This function does not implement those storage guarantees.

A successful online sign-in reset the mark to the current time, allowing recovery after a fast clock was legitimately corrected. That reset belonged to the authenticated online path, not to an offline retry.

Refusing access did not mean deleting the cache

The main app’s expiry policy could not delete data the companion extension still needed. An expired result blocked that entry point and requested an online sign-in; it left the shared cache intact.

Offline use also had to stay separate from an ordinary signed-in state. Otherwise, the normal cache writer could rewrite cached data, advance its modification dates, and renew its apparent freshness without server contact. A timestamp is useful evidence of synchronization only while that write invariant holds.

Eligibility was checked again when opening the cache. A login screen left visible across the deadline could not reuse the earlier “fresh” result.

Test the rollback twice

The original implementation’s red run reported four failed clock-mark assertions, including rollback refusal, repeated refusal, mark advancement, and online-sign-in reset. After the repair, the recorded core-target run passed 184 tests in 31 suites. Those are historical Xcode results, not a new Keychain integration run.

A separate run of the function above passed 15 synthetic-timestamp tests, covering expiry boundaries, future-dated items, repeated rollback refusal, and rechecking at open without mutating the cache. It did not exercise Keychain persistence. For example, two reads with these synthetic inputs must preserve the higher mark:

result, mark = cache_status([100], 190, 200, 1000)
assert (result, mark) == ("expired", 200)
assert cache_status([100], 190, mark, 1000) == ("expired", 200)

A high-water mark cannot establish time that was never observed

A device can spend real time offline while its wall clock never exceeds the saved mark. These checks cannot prove how much time passed, guarantee remote revocation, or withstand a compromised device that can alter the protected state. Biometric access control does not authenticate the clock.

Test your cache by expiring it, moving the injected clock backward, and reading eligibility twice. Both reads should refuse access, and the second must not inherit a lowered mark.

iOS jailbreak protection in 2026: detection is a risk signal, not a security boundary

Mon, 26 Oct 2026 23:20:00 +0100

An iOS app had jailbreak detection, passed an external pentest, and still spent more than a year accepting every jailbroken device because an asynchronous result was discarded. After that bug was fixed, the checks were still a 2025 list of Cydia-era paths that missed current rootless and hidden jailbreaks.

If the requirement says “the app must not run on a jailbroken phone,” the useful answer in 2026 is not a longer path list. Treat jailbreak detection as a bypassable risk signal, decide what sensitive operation the signal protects, enforce that decision at the server where possible, and test the whole policy against current jailbreak and bypass tools.

The short version

A defensible implementation today has five parts:

Write a threat model that explains why a compromised device materially changes this app’s risk.
Combine independent local signals and run them near sensitive operations, not only at launch.
Apply a proportional policy: observe, step up, restrict sensitive data, or refuse service according to confidence and impact.
Put the final authorization decision on a server and use App Attest to protect app-to-server requests. Do not call App Attest to prove that a device is not jailbroken.
Test the release binary statically, dynamically, and by trying to bypass it on representative clean and jailbroken devices.

This is also the difference between “the code contains a jailbreak check” and an engineering conclusion that can survive review.

How a working check silently stopped working

The first implementation was conventional:

sysctl checked P_TRACED and terminated a release build under a debugger;
known paths such as /Applications/Cydia.app, /bin/bash, /usr/sbin/sshd, and /etc/apt were checked;
the app tried to write /private/jailbreak.txt outside its sandbox;
cydia:// was passed to canOpenURL;
loaded Mach-O images were searched for MobileSubstrate.

It ran only in non-Debug builds on physical devices. That was sensible, and the pentest team verified it.

One probe had never worked in the affected app targets: canOpenURL returns false for schemes that are not declared under LSApplicationQueriesSchemes, and cydia was not declared. An app extension could not use UIApplication.shared at all. Current IOSSecuritySuite also removed the cydia:// check because an App Store app registered the scheme and caused false positives. URL-scheme probes are weak evidence even when configured correctly.

Then the check moved behind a function that started a Task, returned from the function immediately, and ignored the task’s Bool. The user interface always received true. The debugger check still appeared to work because fatalError happened inside the task, masking the control-flow bug. Later refactors moved the broken call without changing its behavior.

The eventual fix was not another jailbreak signature. It was to make validation genuinely asynchronous, await it, and test the production call site as well as the checker.

That history produced three reusable lessons:

Test enforcement, not just detection. A perfect signal whose result is ignored protects nothing.
Test every app entry point and extension. The main app, credential provider, widget, and background path do not necessarily share a process or API surface.
Test the release configuration on hardware. A Debug simulator test proves code structure, not production behavior when the checks are compiled out there.

When OWASP says to implement jailbreak detection

OWASP MASTG 2.0, released in July 2026 , makes the trade-off unusually explicit. MASTG-BEST-0030 is currently tagged Android and says “root detection,” but its risk principle applies directly to the equivalent iOS control: it is a bypassable, cost-raising environment signal. It should be used only when compromised-device risk materially affects the app. The selection and response should follow the threat model and risk tolerance.

That points toward jailbreak protection when a compromised platform makes a concrete abuse case easier, for example:

extracting high-value secrets or regulated data from the app process or local cache;
instrumenting transactions, credential filling, cryptographic operations, or fraud controls;
changing a game, licensing, payment, or content-access decision made locally;
automating a protected API through a modified client;
defeating controls that depend on the sandbox, Keychain access policy, or trusted user interaction.

It does not make jailbreak detection a universal mobile-app baseline. A low-risk public-content app may gain little while inheriting false positives, support work, accessibility problems, and an exclusion policy it cannot reliably enforce.

MASVS-RESILIENCE-1 states the security concern: many other controls assume that the platform’s sandboxing, secure storage, and biometrics can be trusted. But the wider MASVS resilience guidance treats anti-tampering and runtime self-protection as defense in depth, never a replacement for the baseline security architecture.

The current techniques are rootless and designed to hide

The old signature list described rootful Cydia and MobileSubstrate installations. Current jailbreaks changed the filesystem layout and then added explicit concealment.

Dopamine is a rootless jailbreak. Its bootstrap commonly appears below /var/jb, with ElleKit providing tweak injection. Dopamine 1.1 introduced JB_ROOT_PATH in the environment of processes with tweaks enabled, while noting that /var/jb might become optional to impede detection. Palera1n has its own loader and helper artifacts. Frida may appear as frida-server, Frida Gadget, a loaded image, or merely runtime behavior after its obvious names have been changed.

At the time of this review, Dopamine’s current upstream release was 3.0.12, published October 20, 2026 . A signature inventory that stops at Cydia or even Dopamine 2 is already historical.

Dopamine 2.4’s release notes are the more important evidence: “Hide Jailbreak” was changed specifically to hide the jailbreak to the point where it should be undetectable. RootHide’s Dopamine integration randomizes the jailbreak root and selectively keeps jailbreak changes away from blacklisted apps. Its purpose is jailbreak-detection bypass.

A 2026 local detector should therefore look across different classes of evidence, for example:

rootful and rootless files, directories, symlinks, package-manager state, and randomized .jbroot-* roots;
whether the sandbox permits a collision-safe write to a protected path;
debugger state and failure of the debugger-status query itself;
suspicious environment keys such as DYLD_INSERT_LIBRARIES and JB_ROOT_PATH;
loaded images associated with Frida, ElleKit, Substrate, Substitute, libhooker, RootHide, or Dopamine’s system hook;
behavior that should be prohibited by the sandbox, where the API is appropriate for App Store software;
server-side anomalies and app-integrity assertions.

Use more than one API for important filesystem evidence. A FileManager.fileExists check, a POSIX lstat, and directory enumeration do not create certainty, but an attacker now has more hooks to maintain consistently.

Do not equate more signatures with “state of the art.” Public signature lists are useful coverage for unsophisticated and accidentally exposed jailbreaks. RootHide, renamed Frida components, or a patch to the function that aggregates all checks can make every local answer say “clean.” OWASP’s jailbreak detection knowledge article says the same thing: these defenses are not effective by themselves, although distributing them through the app increases the cost of bypassing them.

The response matters more than the Boolean

“Jailbroken: yes or no” is too weak a model for a fallible set of signals. Record an internal risk result with the evidence and confidence, then map it to a product policy.

Situation	Reasonable response
One weak or ambiguous signal	Telemetry, retry, or step-up authentication
Several independent local signals	Block sensitive operations; avoid disclosing the exact failed probe
Debugger or injected runtime tool in a high-risk flow	Refuse the operation and invalidate relevant server state
Unsupported App Attest on an otherwise supported device	Treat as a risk signal according to the documented policy
Enterprise-managed fleet with a strict posture requirement	Evaluate Managed Device Attestation at the relying service

OWASP recommends proportional responses and warns that aggressive lockout can affect legitimate users. For an app that stores documents and credentials, a useful soft block can allow login and folder metadata while refusing document bodies, downloads, offline queues, and credential release.

That restriction must be in application logic, not just a screen with a Close button. Better still, the server should not return the sensitive payload. Otherwise, a hook can dismiss the screen or call the same API directly.

A hard process exit is appropriate only if the threat model justifies denial of all functionality and the product accepts false positives. Crashing also gives poor observability and can turn an operational failure into a permanent lockout. A generic refusal with server-side telemetry is usually easier to operate.

App Attest authenticates app instances, not clean devices

Apple’s WWDC26 App Attest session describes the current protocol:

the server initiates enrollment and issues a challenge;
the app creates a Secure Enclave-bound key and attests it;
the server validates and stores the attestation and public key;
sensitive requests include a fresh server challenge in an assertion over the payload;
the server validates the signature and requires a strictly increasing counter to reject replays.

This is valuable. It lets the server distinguish a valid published app instance, detect unexpected signing or launch categories, and bind authorization to a payload that has not been altered in transit.

It is not a jailbreak oracle. The same 2026 session says that a compromised device can still pass attestations and broker valid attestations for modified clients. Apple provides the fraud metric to help detect that pattern. Use App Attest as another server-verified signal, with retry and onboarding policy for legitimate failures.

For managed organizational devices, Managed Device Attestation provides Secure Enclave-backed evidence about device identity and security posture. That is stronger input for a managed-fleet trust decision, but it is not a general capability that a consumer app can require from every unmanaged phone.

Test implementation, behavior, and bypass resistance

OWASP MASTG 2 separates two questions that teams often collapse into one.

MASTG-TEST-0240 is static: extract the shipped app package and inspect the binaries for jailbreak-detection mechanisms. This catches missing code, build flags that compiled it away, or a check that never reached a target. OWASP warns that automated static analysis is not exhaustive; manual reverse engineering may be necessary.

MASTG-TEST-0241 is dynamic: run the app, identify the checks, and try to bypass them. A test fails when detection is absent, but passing a single predefined bypass tool doesn’t prove resilience because its scripts may be incomplete or outdated.

For a real release, I would keep this matrix with the evidence from each run:

Test	Expected evidence
Clean supported iPhones and iOS versions	No false refusal; all sensitive flows work
Release build on physical hardware	Checks execute; Debug/simulator exclusions do not leak into the archive
Rootful and current rootless jailbreaks	Each applicable signal fires; policy is enforced
Dopamine Hide Jailbreak and RootHide blacklist	Document what remains visible and what becomes undetectable
Frida server and embedded/renamed Gadget	Known artifacts are detected; renamed forms challenge bypass resistance; outcomes are recorded
Hook the aggregate result and individual APIs	Sensitive server operation still requires valid authorization and assertions
Main app, extensions, background entry points	Each entry point validates before releasing its protected capability
Async and error paths	Results are awaited; query failures follow the documented fail-open or fail-closed policy
Pentest bypass configuration	It cannot enter a production archive; CI tests that invariant

Unit-test injected signals, cleanup failures, query failures, and clean-device results. Then add integration tests around the call site: the protected operation must not run after validation fails. Finally, let a tester attack the binary rather than merely demonstrating a known file on a jailbroken test phone.

A reviewable 2026 “lege artis” baseline

“State of the art” is not established by citing one library or by counting checks. For an engineering review, the conclusion should be reproducible, dated, scoped, and explicit about uncertainty. This is the practical baseline I would use when assessing whether an iOS app’s jailbreak protection was implemented with professional care in October 2026:

Define the question and date. State which app version, release configuration, iOS range, device classes, sensitive operations, and attacker capabilities are in scope. Practice changes as jailbreaks and iOS change.
Identify the risk before prescribing the control. Connect compromised-platform capabilities to actual assets and abuse cases. Explain why jailbreak detection materially affects this app.
Select authoritative and current sources. Record versions and access dates for OWASP MASVS/MASTG, Apple platform documentation, relevant jailbreak release notes, and the implementation under review. Treat public detection libraries as examples, not normative standards.
Preserve and identify the evidence. Record the source commit, archive hash, build settings, entitlements, tool versions, device/iOS versions, jailbreak version, and test logs. Work from a reproducible copy.
Separate facts from interpretation. “The production binary contains /var/jb and JB_ROOT_PATH checks” is a fact. “Those checks make bypass harder but cannot establish a clean device” is the interpretation supported by OWASP and current bypass projects.
Analyze the whole control path. Review signal collection, aggregation, async propagation, every entry point, the product response, server authorization, telemetry, and release-lane exceptions. Include negative findings and dead code.
Test both presence and effectiveness. Perform static inspection, clean-device compatibility tests, dynamic execution on relevant jailbreaks, and bypass attempts. A unit test or a pentest on one jailbreak version is not enough.
Consider alternatives and false positives. Explain why a detected artifact could be benign, why a missing artifact does not establish cleanliness, and why the chosen response is proportional.
State limitations. Local checks can be hooked or patched; hidden jailbreaks may expose no artifact; App Attest can succeed on a compromised device; the tested device set is finite.
Make a bounded conclusion. Say whether the examined implementation follows the documented risk policy and raises attacker cost for the tested scenarios. Do not conclude that it guarantees a non-jailbroken device.

This structure mirrors a sound software expert analysis: sources and methods are identified, observations remain distinguishable from interpretation, contrary evidence is not hidden, limitations reduce the stated certainty, and the conclusion answers only the technical question examined. It is guidance for an engineering assessment, not a legal conclusion about liability, contractual compliance, or negligence.

Takeaway

The old implementation failed twice: first because application control flow discarded a valid result, then because its threat model had aged past its Cydia-era signatures. The durable fix is a process—current layered signals, proportional server-enforced policy, and repeated adversarial testing—not a permanent isJailbroken function.

Take one sensitive operation in your app and trace it today from device signal to server authorization. If any step can ignore, hide, or locally override the result, that is the next protection to fix.

References

OWASP MASVS: Resilience against reverse engineering and tampering , accessed October 26, 2026.
OWASP MASTG 2.0.0 release announcement , July 4, 2026.
OWASP MASVS-RESILIENCE-1: Platform integrity , accessed October 26, 2026.
OWASP MASTG-BEST-0030: Implementing Root Detection , accessed October 26, 2026.
OWASP MASTG-KNOW-0084: Jailbreak Detection , accessed October 26, 2026.
OWASP MASTG-TEST-0240: Jailbreak Detection in Code , accessed October 26, 2026.
OWASP MASTG-TEST-0241: Runtime Use of Jailbreak Detection Techniques , accessed October 26, 2026.
Apple: Secure your apps with App Attest, WWDC26 , accessed October 26, 2026.
Apple Platform Security: Managed Device Attestation , accessed October 26, 2026.
Dopamine 2.4 release notes: Hide Jailbreak , February 16, 2025.
Dopamine 1.1 release notes: JB_ROOT_PATH , June 16, 2023.
Dopamine releases , accessed October 26, 2026.
Dopamine 3.0.12 release , October 20, 2026.
RootHide integration for Dopamine 3 , accessed October 26, 2026.
IOSSecuritySuite jailbreak checker , accessed October 26, 2026.

Adapt security skills to your project's risks: from Go to OWASP LLM and iOS

Sun, 25 Oct 2026 21:50:00 +0100

A security skill written for Go covers different questions than one written for an LLM application or an iOS app. I started with golang-security from samber/cc-skills-golang , asked Claude to adapt it using the OWASP Top 10 for LLM Applications, and later carried that approach into production iOS development.

The most useful part of the adaptation was aligning the review questions with the project’s risks and the company’s values.

Start with a review method

I was working in Go, a language I was unfamiliar with. I used my security skills alongside other Go skills to keep the implementation in check.

The security skill provided a repeatable method: separate modes for coding, reviewing changes, and auditing a codebase; questions about trust boundaries and attacker-controlled input; and instructions to trace data through surrounding code before reporting a finding. Its references and checklist gave the agent somewhere to look beyond the main instructions.

Adapt OWASP categories to the application

I asked Claude to create a new skill using the OWASP Top 10 for LLM Applications (2025) , adapted to the project’s LLM and Model Context Protocol (MCP) surfaces.

The resulting llm-security skill kept the review modes and evidence requirements. It mapped the categories to tools, resources, prompts, and content returned to the model. The Go checklist stayed active for the code underneath.

For prompt injection , the reviewer needs to identify whose text reaches the model and what an attacker can control. A returned document body or filename can carry instructions. For excessive agency , the questions concern tool permissions, reachable side effects, and authorization enforced outside the model.

Those questions give the agent a path through the repository. It can follow a tool result back to its data source, then follow a tool call forward to the operation it performs. The skill also asks it to account for existing protections when assessing impact.

Repeat the adaptation for iOS

Later, I applied the same approach to an iOS app using the OWASP Mobile Top 10 (2024) . That mobile security skill became part of production development.

The review method carried over. The mapping changed to platform APIs and the code responsible for them:

Mobile risk	Concrete review question
Improper credential usage	Where are tokens stored, and which Keychain access settings apply?
Insufficient input/output validation	What can a deep link or share-extension input cause the app to do?
Insecure data storage	Which protection classes apply to cached and temporary user files?
Inadequate privacy controls	Where can sensitive data appear outside its intended screen, including logs and notifications?

Repository context matters here. If a shared SDK owns authentication or networking, the skill needs to direct the reviewer into that SDK. Stopping at the changed Swift view would leave the relevant behavior unexamined.

For a mobile adaptation, MASVS supplies verification controls, and MASTG provides testing guidance for checking them.

Turn company values into review rules

Company values need concrete instructions too. For example, a company that prioritizes privacy can require reviewers to trace user data into logging, analytics, and crash reporting. That turns a broad preference into something an agent can inspect.

The mobile skill also records settled product decisions. An intentional tradeoff needs a rationale and the protections that still apply so that each review can assess the implementation against that decision. New evidence or changed assumptions can justify reopening it.

Severity needs the same local reasoning. OWASP provides risk categories; the product’s data, attacker access, and existing controls determine a finding’s impact.

A fresh adaptation request could be:

Use the existing security skill as a template. Keep its review workflow
and evidence requirements. Adapt the relevant OWASP categories to this
repository's entry points, data flows, and platform APIs. Link each check
to the responsible code and a primary reference. Record product decisions
with their rationale. Preserve the language-level checks alongside it.

Tailoring a security skill gives the agent concrete questions about the product it is building. Take one OWASP category in your current skill and map it to the code, data, and product decision the reviewer should examine.

A repeatable agentic development workflow: context, tickets, implementation, review

Sun, 25 Oct 2026 18:25:00 +0100

Giving a coding agent a feature request and accepting the first working diff is easy. Repeating that across projects is where things get messy: the agent has to rediscover the domain, invent missing requirements, decide what “done” means, and review its own assumptions.

Here is the setup I now reuse across stacks. It has supported ten iOS app releases this year, helped clean up an existing app built with loose agent guidance, and helped a Go implementation recover after its first approach fell apart - to name a few examples. The Go work also involved a language stack and MCP technology I had not used before. The process didn’t provide that expertise on its own; it made the gaps visible early enough to address them.

Give the agent durable context

Start with mattpocock/skills and run its repository setup. Keep a short domain vocabulary in CONTEXT.md (or the current upstream convention, GLOSSARY.md). Record consequential design choices as ADRs. The agent can then use the same terms in discussions, tickets, code, and reviews.

For a change whose shape is uncertain, use /grill-with-docs to work through the decisions and write them down. Run /to-spec once the conversation has enough detail. Break the spec into small, ordered tickets with /to-issues in older installations or /to-tickets in the current upstream skills. Then use /implement against those tickets.

The sequence is:

shared vocabulary + ADRs → grill → spec → tickets → implement → review

You can enter the sequence at whichever step the work needs. A small, well-specified fix may already have a ticket. An uncertain feature should be grilled before an agent starts writing code.

Put tickets where the project works

The tracker is an adapter for the process. One project keeps issues in a local docs/TODO.txt; others use Vikunja or Taiga. What matters is that the agent knows where the authoritative spec and tickets live, how to record blockers, and how to update their status. Write those rules into the repository’s agent documentation. A tracker integration skill can handle the API details without changing the development sequence.

This also makes the setup portable. Moving to a new project means teaching the agent that project’s domain, tracker, and checks, rather than rebuilding the workflow.

Make implementation answer to the spec

Implementation includes tests and a diff review. Review the result against two different questions: does the code follow the project’s standards, and does it actually satisfy the originating spec? The latter catches plausible code that solves the wrong problem. A previous review found duplicate-write paths after a large test suite passed because it checked a safety claim against the call path.

Where the project’s rules allow it, use a different model family for review. A typical split is Claude writing the code and GPT reviewing it, since a different family brings different blind spots. Claude reviewing Claude has also proven decent, so one family is still a workable setup when you can only use one provider. For larger changes, a multi-review skill can run distinct axes such as correctness, quality, tests and performance, security, and, for frontend, accessibility. Give each reviewer the same bounded diff and the actual spec; collect actionable findings back into the tracker.

Attach stack-specific skills where they help. Go work can load guidance for concurrency, error handling, testing, and security. iOS work can use Swift and mobile guidance. For exposed web APIs, mobile clients, or MCP tools, apply relevant OWASP-based web, mobile, and LLM security skills during implementation and review. A checklist gives the agent concrete questions to ask; it does not turn a code review into a security certification.

Start with one change: write down the vocabulary and decision, turn the request into a spec and tickets, implement one ticket, and review the diff against the spec and its security boundaries. Adjust the tracker, skills, and review axes to fit the next project.

Moodle 5.3 Classic theme serves an empty stylesheet: 'Undefined variable $mds-spacing-md'

Sat, 24 Oct 2026 14:58:00 +0200

After an upgrade to Moodle 5.3, every page on the Classic theme lost its layout. The theme’s stylesheet came back like this:

$ curl -s -o /dev/null -w '%{http_code} %{size_download}\n' \
 'https://moodle.example.com/theme/styles_debug.php?theme=classic&type=scss'
200 0

That’s a 200 with an empty body, and nothing in the logs. With debugging forced to developer level, the SCSS compiler said:

Error while compiling SCSS: Undefined variable $mds-spacing-md: /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle/course.scss on line 132, at column 17

Classic imports Boost’s SCSS, and in 5.3 that SCSS uses the new design system’s tokens. Classic never imports them. One @import fixes it.

The setup

Moodle 5.3.0, PHP 8.4.27, in a Docker image of our own, upgraded from 5.0
The site’s theme is Classic. It left core in 5.3, so the image installs it from moodlehq/moodle-theme_classic (at commit 363a179, the only one when I checked)
Theme designer mode was on, so pages loaded the theme through styles_debug.php

I rehearsed the upgrade on a copy of the production database. Every URL I tried returned 200, and admin/cli/checks.php reported all checks OK. On production, the pages had no layout.

Two things that looked like the cause

The browser console had two errors:

[Error] Failed to load resource: the server responded with a status of 404 (Not Found) (client.bundle.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 (Not Found) (react.bundle.mjs.map, line 0)

Those are source maps. The browser asks for them only for its developer tools, and Moodle doesn’t ship them. They don’t affect the layout.

The import map had a few 404s as well, for entries like /core/esm/<rev>/bootstrap/. Those entries end in a slash because they map a whole prefix, so the bare URL has nothing to serve. The modules under them loaded.

The problem showed up when I fetched every stylesheet the front page links to and printed its size:

curl -s https://moodle.example.com/ -o index.html
grep -oE '<link[^>]+stylesheet[^>]*>' index.html \
 | grep -oE 'href="[^"]+"' | sed 's/href="//; s/"$//; s/&amp;/\&/g' \
 | while read -r u; do
 printf '%s %s\n' "$(curl -s "$u" | wc -c)" "$u"
 done

The plugin stylesheets had a few kilobytes each. The theme’s own, type=scss, had zero bytes.

Getting the real error

Moodle catches the compiler’s exception and passes the message to debugging() at developer level, so at the usual debug setting you get nothing at all. You don’t have to change the site’s settings to see it. Compile the theme from the CLI with debugging switched on for that one process (run it as the web server’s user and adjust the path to config.php):

sudo -u www-data php -r '
define("CLI_SCRIPT", 1);
require "/opt/moodle/config.php";
$CFG->debug = E_ALL;
$CFG->debugdisplay = 1;
$t = \core\output\theme_config::load("classic");
$m = new ReflectionMethod($t, "get_css_content_from_scss");
$c = $m->invoke($t, false);
var_dump(is_string($c) ? strlen($c) : $c);
'

get_css_content_from_scss isn’t public, hence the reflection. It returns false when the compile fails, and the error comes with the import chain:

++ Error while compiling SCSS: Undefined variable $mds-spacing-md: /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle/course.scss on line 132, at column 17
Call Stack:
#0 import /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle/course.scss /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle.scss on line 13
#1 import /opt/moodle/public/theme/classic/scss/../../boost/scss/moodle.scss /opt/moodle/public/theme/classic/scss/moodle.scss on line 2
#2 import /opt/moodle/public/theme/classic/scss/moodle.scss (unknown file) on line 127 ++

Why

Boost’s default preset, theme/boost/scss/preset/default.scss, starts with:

// Import Moodle Design System tokens.
// Must be imported before Bootstrap so that the design system tokens can be used to override Bootstrap variables.
@import "design-system";

design-system.scss pulls in the $mds-* tokens. Classic’s presets import fontawesome, bootstrap and moodle, but not design-system, and Classic’s moodle.scss imports Boost’s. So Boost’s course.scss uses $mds-spacing-md, and nothing has defined it.

The fix

Add the import to the top of theme/classic/scss/classic/pre.scss. Classic’s lib.php puts that file in front of the preset, so this covers both presets:

pre=/opt/moodle/public/theme/classic/scss/classic/pre.scss
{ printf '@import "../../boost/scss/design-system";\n\n'; cat "$pre"; } > /tmp/pre.scss
mv /tmp/pre.scss "$pre"
sudo -u www-data php /opt/moodle/admin/cli/purge_caches.php --theme

The path resolves from theme/classic/scss/, as the call stack above shows. After the purge, the same CLI compile returned 1,218,070 characters instead of false, and styles_debug.php sent 1,557,703 bytes.

If Classic is part of your build, as here, apply the patch in the build. A fix made inside a running container is gone the next time the container is recreated.

Once the layout was back, I turned theme designer mode off (admin/cli/cfg.php --name=themedesignermode --set=0). It recompiles the theme on every page view: here, about 0.8 s and 112 stylesheet requests per page.

Takeaway

A Moodle theme that fails to compile still answers 200, just with no CSS, so upgrade checks that look only at status codes pass right over it.

Before your next Moodle upgrade, add the theme’s compiled CSS size to your checklist: anything near zero is a failed compile.

A five-second Swift Testing wait timed out after its loop had run five times

Thu, 22 Oct 2026 16:12:00 +0200

Five concurrency tests passed on their own in 0.086 seconds. In the full target, they failed together at about 6.8 seconds:

✘ Test fetch_threeSources_asksForEverySourceAtOnce() recorded an issue at FetcherConcurrencyTests.swift:21:30:
 Expectation failed: ContinuousClock.now < deadline
✘ Test fetch_threeSources_asksForEverySourceAtOnce() failed after 6.816 seconds with 1 issue.

It looked like a deadlock in the code under test. It was a test helper measuring the wrong thing.

Setup

Swift Testing, an Xcode test target that mixes @MainActor unit suites with snapshot suites.
Two test plans run the same target. One sets "parallelizable": false for it, and the other doesn’t. Only the parallel plan failed.
A small polling helper that the concurrent-fetch tests use to wait for a gated fake service to see requests:

static func until(
 sourceLocation: SourceLocation = #_sourceLocation,
 _ condition: () -> Bool
) async throws {
 let deadline = ContinuousClock.now.advanced(by: .seconds(5))
 while !condition() {
 try #require(ContinuousClock.now < deadline, "Timed out waiting for the condition to hold",
 sourceLocation: sourceLocation)
 await Task.yield()
 }
}

The helper runs on the main actor. Each Task.yield() lets the queued work, in this case the fetcher opening its requests, run before the condition is checked again.

Reproduce before guessing

Running the target alone under the parallel plan with -test-iterations and -run-tests-until-failure failed 3 out of 3 times, always in the same tests and always at the wall-clock deadline. So it wasn’t a rare flake. It failed every time under that plan.

To check whether the waits were stuck or just slow, I temporarily replaced the deadline with a counter and printed what each wait used:

let start = ContinuousClock.now
var turns = 0
while !condition() {
 turns += 1
 await Task.yield()
}
print("WAITPROBE turns=\(turns) elapsed=\(ContinuousClock.now - start) at \(sourceLocation.line)")

5 5.766 FetcherConcurrencyTests.swift:54
5 5.712 FetcherConcurrencyTests.swift:21
4 6.411 FetcherConcurrencyTests.swift:37
3 5.704 SyncLoggingTests.swift:25
3 5.338 InFlightTests.swift:20

Every wait finished, but the slowest needed only 3 to 5 turns and took more than 5 seconds of wall time. The condition was met on almost the first chance the fetcher got to run. It just didn’t get that chance for seconds.

The reason showed up when I sorted tests by duration. In the parallel plan, snapshot suites share the process and the main actor, and each render holds the actor synchronously, some of them for over 7 seconds. While a render ran, the waiting test’s next turn just sat in the queue, and the clock kept running.

Dead end: a longer timeout

I tried a 60-second deadline as an experiment. Both runs passed, with the concurrency tests taking about 7.9 seconds each.

That shows the cause, but it isn’t a fix. 60 is no more principled than 5. A heavier snapshot suite brings the failure back, and a test that really hangs now takes a minute to report it. I reverted it.

Fix: spend both budgets before failing

The wait should fail only once it has had enough chances to make progress and enough time has passed:

static func until(
 sourceLocation: SourceLocation = #_sourceLocation,
 timeBudget: Duration = .seconds(5),
 _ condition: () -> Bool
) async throws {
 let deadline = ContinuousClock.now.advanced(by: timeBudget)
 var turns = 0
 while !condition() {
 let hasBudgetLeft = turns < Constants.turnBudget || ContinuousClock.now < deadline
 try #require(hasBudgetLeft, "Timed out waiting for the condition to hold",
 sourceLocation: sourceLocation)
 turns += 1
 await Task.yield()
 }
}

enum Constants {
 /// A wait under a fully loaded run of this target took at most 5 turns.
 static let turnBudget = 100
}

On an idle main actor, 100 turns go by in milliseconds, so a condition that never holds still fails just after the time budget. On a starved actor, the wait keeps going until it has had real chances to run.

Pin it with a controlled starvation test

The full target is a slow, noisy reproduction, so I wrote a test that intentionally causes starvation. It makes timeBudget a parameter, holds the main actor for longer than that budget, and makes the condition true only afterward:

@MainActor
@Suite
struct WaitTests {
 @Test
 func until_whenTheMainActorIsHeldPastTheTimeBudget_waitsForTheCondition() async throws {
 let holds = LockIsolated(false)
 Task { @MainActor in
 Thread.sleep(forTimeInterval: 0.3) // stands in for a snapshot render
 Task { @MainActor in holds.setValue(true) }
 }
 try await Wait.until(timeBudget: .milliseconds(100)) { holds.value }
 }

 @Test
 func until_whenTheConditionNeverHolds_fails() async {
 await withKnownIssue {
 try await Wait.until(timeBudget: .zero) { false }
 }
 }
}

Before the fix, it failed after 0.335 seconds with the same ContinuousClock.now < deadline error. After the fix, it passed in 0.303 seconds. The second test makes sure the helper can still fail.

Verify both directions

Check	Result
Parallel plan, before the fix	0 of 3 passed
Parallel plan, after the fix	5 of 5 passed
Serial plan, whole target	690 tests passed
Mutant: fetcher limited to one request at a time	all 5 concurrency tests failed, each at 5.0 s

The last row matters most. A longer timeout would also have made the parallel runs pass. What it wouldn’t keep is fast failure for a real bug. I changed the fetcher’s concurrency limit from 4 to 1, so the “all three requests are open at once” condition could never be true. All five tests caught it, each at the 5-second time budget, and I reverted the mutation.

One caveat: that mutant ran under the serial plan. Under heavy parallel load, a real hang fails after 100 turns, which takes longer than 5 seconds. That’s the trade-off I chose, and it applies to this helper and this workload. It isn’t a general fairness guarantee for Task.yield().

A second guess hiding in the same tests

Three tests waited for the first request and then gave the scheduler “five settling turns” before asserting that all three were open. That turn count was another guess, and it could fail under load the same way. The fix was to wait for the exact condition:

// before
try await Wait.until { service.pendingCount > 0 }
await Wait.untilSettled()
#expect(service.pendingCount == 3)

// after
try await Wait.until { service.pendingCount == 3 }

Takeaway

A wall-clock timeout in a polling helper measures how busy the process is, not whether your code is stuck. When a wait times out only in the big parallel run, count its turns before you suspect a deadlock. If it got only a handful of turns, the actor was starved. Make the helper fail only after both a time budget and a turn budget run out. Then check the fix with a mutation that should hang and make sure it still fails quickly.

Try it on your own polling helper: add a turn counter and run the full target in parallel once. If the worst wait used single-digit turns but took seconds, the timeout is measuring main-actor contention.

rsync code 23: atomic archives can still form an incomplete release

Thu, 22 Oct 2026 00:00:00 +0000

A dataset transfer ended with:

rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1327) [sender=3.2.5]

The packing job published each finished archive by rename, so final filenames never exposed an archive still being written. The transfer nevertheless began before the collection was complete. Repeating it after packing finished, then checking destination hashes, recovered the release.

This happened in the Claude Code over SSH and OpenPBS workflow : a PBS job produced the archives, while separate remote commands and a human-operated transfer consumed them.

The filename was safe before the collection was ready

The packer used this pattern, with paths generalized:

tar -czf groups_00.tar.gz.part groups/part_00 &&
 mv groups_00.tar.gz.part groups_00.tar.gz

A reader looking for *.tar.gz would only see that archive after its successful creation and rename on the same filesystem. But the producer repeated the operation for several archives, then wrote sidecar files. Nothing prevented a consumer from enumerating the directory between those steps.

Transfer instructions went out while the packer was running. The user’s pasted output ended with to-chk=0/16—the intended release contained seventeen archives plus a checksum manifest and a guide. A later source listing showed the complete set.

The count, transferred size, and packing timeline supported the explanation that the last archive and sidecars didn’t exist when the transfer enumerated its inputs. The earlier detailed rsync errors were not included, so this remained a supported diagnosis rather than a direct readback of each missing-path error.

Code 23 alone cannot identify that cause. The rsync manual defines it as a partial transfer due to error; inspect the preceding errors before retrying mindlessly.

Repeat the transfer after checking the producer

The recovery checked that packing had finished, repeated the transfer, and verified the archives at the destination. With server names and filenames generalized, run from the producer’s archive directory:

rsync -rtv --partial --progress \
 SHA256SUMS guide.txt groups_*.tar.gz metadata.tar.gz \
 researcher@192.168.0.3:incoming/release-01/

Then, on the destination:

cd ~/incoming/release-01 && sha256sum -c SHA256SUMS

The user supplied seventeen OK results. Those checks verified the listed archive bytes, including the archives transferred earlier. They did not verify recipient permissions or the guide’s contents. A separate archive-content check also lacked bc, so its aggregate group-count calculation did not succeed; that check must not be presented as another passing validation.

--partial retains partial transfers. It does not by itself promise efficient offset-based resumption under every transfer configuration.

Add a release-complete barrier

We did not implement a stronger publication protocol during this recovery. For a future run, use a fresh release directory that consumers cannot mistake for an earlier finished release. Publish a completion marker only after validating the expected archive set and sidecars. Consumers should require that marker before enumeration and check hashes after transfer.

A marker needs an immutable release behind it. Reusing a directory with an old marker can announce incomplete new contents as ready; writing the marker early merely moves the race.

Before sending a transfer command, check the producer’s completion evidence and the expected file set. Per-file renames protect readers of each file, but consumers still need a defined point at which the collection is ready.

Unprivileged LXC containers share the host's per-user kernel limits: 'Too many open files' and 'disk quota exceeded'

Wed, 21 Oct 2026 13:13:00 +0200

One day, two containers on the same host failed with errors that seemed unrelated. systemd in one of them:

systemd[4573]: Failed to create control group inotify object: Too many open files
systemd[4573]: Failed to allocate manager object: Too many open files
systemd[1]: Failed to start user@1000.service - User Manager for UID 1000.

Docker in another, where every container sat at Exited (128):

OCI runtime create failed: runc create failed: unable to start container process: error during container init: unable to join session keyring: unable to create session key: disk quota exceeded: unknown

No file descriptors had run out, and no disk was full. Both errors come from per-user kernel quotas counted on the host, and every unprivileged container there was drawing from the same two.

The setup

Debian 13, kernel 6.12, LXC 6.0.4
More unprivileged containers than the hardware comfortably runs, all with the same idmap: container root is host UID 1000000
systemd in every container, Docker nested in some

Too many open files: inotify instances

systemd opens inotify instances to watch cgroups and units, and inotify_init1() fails with EMFILE when the user’s instance limit is reached. strerror turns that into “Too many open files”. The limit is fs.inotify.max_user_instances, 128 by default.

Count the instances per user on the host:

sudo find /proc/[0-9]*/fd -lname 'anon_inode:inotify' 2>/dev/null \
 | cut -d/ -f3 | xargs -I{} ps -o user= -p {} | sort | uniq -c | sort -rn
cat /proc/sys/fs/inotify/max_user_instances

Host root held 23 instances, and uids in the container range (1000000 and up) held 104 between them: 127 of 128. On the first sighting, my coding agent got that far, then dropped it as a side effect of having started too many containers at once. That was half right, since each container’s systemd adds about five instances. A day later it came back.

The lxc package already ships a higher limit, and it wasn’t in effect:

$ cat /etc/sysctl.d/30-lxc-inotify.conf | grep -v '^#'
fs.inotify.max_user_instances = 1024
$ cat /proc/sys/fs/inotify/max_user_instances
128

The file was dated three days after the last boot. /etc/sysctl.d is applied by systemd-sysctl at boot, and the package’s install hadn’t applied it, so the setting sat on disk doing nothing.

Disk quota exceeded: kernel keys

runc gives each container it starts a new session keyring. add_key() fails with EDQUOT when the user’s key quota is full, and strerror calls that “Disk quota exceeded”. The quota is kernel.keys.maxkeys, 200 keys for any uid other than root. The host shows usage per UID:

$ cat /proc/key-users
 0: 127 126/126 107/1000000 2283/25000000
 1000: 12 12/12 12/200 86/20000
1000000: 198 198/198 198/200 4189/20000

198/200 are keys used against the quota. UID 1000000 is root in every container, so all the containers shared those 200 keys, and Docker’s restart attempts had nothing left.

The fix

Raise the limits and apply them right away, not at the next boot. Incus recommends 2000 keys and 2 MB in its server settings ; 1024 inotify instances matches the lxc package.

sudo tee /etc/sysctl.d/60-lxc-limits.conf <<'EOF'
fs.inotify.max_user_instances = 1024
kernel.keys.maxkeys = 2000
kernel.keys.maxbytes = 2000000
EOF
sudo sysctl -p /etc/sysctl.d/60-lxc-limits.conf

With Ansible, ansible.posix.sysctl with sysctl_set: true writes the file and sets the live value in one task.

Then run systemctl reset-failed in the affected container, and docker start for the dead Docker containers. Right after, uid 1000000 held 226 keys, already over the old limit.

Why the two quotas are counted differently

The two quotas look similar, but separate idmaps would fix only one.

Keys are counted per host UID, against global sysctls. Containers that share an idmap share one quota. A separate idmap per container (security.idmap.isolated in LXD and Incus) would give each its own.

inotify instances are counted per user namespace and charged up the chain of parent namespaces, at each level against the uid that created the child namespace. A new namespace’s own limit is INT_MAX, so the limit that bites is the host’s, charged to whoever started the containers. LXC started as root charges host root, which is why the 23 instances of host root’s own daemons counted toward the same 128. Separate idmaps don’t change that.

Takeaway

On a container host, “Too many open files” and “disk quota exceeded” can mean that every container together hit a per-user kernel quota on the host. Check the host’s counters before you debug the container that complained.

Run the find count and cat /proc/key-users on your own container host and compare them with the limits before a container restart finds out for you.

OpenPBS array resume: shard by stable task ID

Tue, 20 Oct 2026 00:00:00 +0000

A resumable OpenPBS array can assign the same task to different workers if each worker slices its own remaining-work list. The unsafe implementation was:

todo = [g for g in groups if not completed(g)]
todo = todo[i::n]

We caught this before the array ran and changed ownership to group_id % worker_count. The corrected workers later started at different times and restarted with fewer tasks remaining; the final audit accounted for every group.

The Claude Code, SSH, and OpenPBS setup explains where commands ran and how jobs were submitted. This failure concerns application-level partitioning, not PBS assigning one array index twice.

A pending list changes between starts

The workload was moving from a multi-GPU job to four independently scheduled one-GPU array elements. Each worker skipped groups with stored completion or rejection markers, then selected a shard.

Suppose the immutable task IDs are zero through five. Worker zero starts first and takes every second pending entry:

pending: 0 1 2 3 4 5
worker 0: 0 2 4

It finishes task zero before worker one starts. Worker one’s view is now:

pending: 1 2 3 4 5
worker 1: 2 4

Tasks two and four overlap, while one, three, and five have no owner in that pair of snapshots. Restarting workers creates the same problem. A completion marker can filter old work without stabilizing the positions of everything that remains.

Preserve ownership while filtering completion

The actual correction retained the immutable numeric group ID:

todo = [g for g in todo if g["group"] % n == i]

Equivalently, make the two decisions explicit:

owned = [g for g in groups if g["group"] % n == i]
todo = [g for g in owned if not completed(g)]

In this application, --shard I/N supplied i and n. The submitted script used #PBS -J 0-3 and passed --shard "$PBS_ARRAY_INDEX/4" to the project-specific driver. Each element requested one GPU. Smaller resource requests did not guarantee immediate starts; later queue snapshots still showed waiting elements.

Keep the shard count fixed for a running generation. Changing n changes ownership even with stable IDs. Also, the same shard launched twice still has overlapping ownership; see the separate concurrent-launch failure .

Test different snapshots, then audit the run

This small Python check reproduces the bad assignment and checks the correction:

all_ids = list(range(6))
first_pending = all_ids
later_pending = all_ids[1:]

bad0 = set(first_pending[0::2])
bad1 = set(later_pending[1::2])
assert bad0 & bad1 == {2, 4}

owned0 = {g for g in first_pending if g % 2 == 0}
owned1 = {g for g in later_pending if g % 2 == 1}
assert not owned0 & owned1
assert owned0 | owned1 == set(all_ids)
print("stable ownership checks passed")

The production evidence was broader than a queue disappearing: logs showed staggered starts, three workers restarting with smaller pending counts, and accepted/rejected outcomes covering the design. A final check reported:

missing (neither): []

That established outcome coverage. It was not a separate duplicate-output audit, and the logs did not establish why those workers restarted. Atomic replacement of a shared CSV also does not prove its concurrent snapshots were consistent.

Before your next resumable array, test ownership against two different completion snapshots. Every unfinished task should retain its owner regardless of when that owner starts.

"No route to host" only for the AI agent: macOS Local Network privacy

Sat, 17 Oct 2026 22:30:00 +0200

My coding agent couldn’t reach anything on my home network:

nc: connectx to 192.168.0.2 port 22 (tcp) failed: No route to host
ping: sendto: No route to host

The router answered, and arp -an listed the server’s MAC address. In my own Terminal, the same ssh command connected. If you see this on a Mac and only your agent sees it, your network is fine. macOS is blocking the app the agent runs in because that app doesn’t have the Local Network permission.

The setup

macOS 27 on Wi-Fi
Claude Code in its VS Code extension, so every command the agent runs is a child process of VS Code
Targets on the Mac’s own subnet: a Linux server, a NAS, a second router

What went wrong

The agent was helping me with a server that really was down (that story is here ). It probed the subnet from the Mac, found that only the router answered, and reported with confidence that “the Mac reaching only [the router] looks like client isolation on the Wi-Fi. That’s usually a guest network.” It told me to join a different network.

I wasn’t on a guest network, and I said so. The agent looked again and noticed what rules the network out: ping itself failed with sendto: No route to host. That message comes from the local sendto() call, so the Mac refused to send the packet at all. Client isolation drops packets in the access point, and then ping reports Request timeout, not a send error. The target’s MAC was already in the ARP table, so the kernel knew exactly where to send it.

So the block was at the MAC, and every TCP test the agent ran from there was worthless.

Later that evening, in a new session, it happened again with a different host on the same subnet. This time the agent suspected its own sandbox and retried with the sandbox off. Same error. What settled it was my reply: the same ssh command works in my Terminal.

Fix

Open System Settings → Privacy & Security → Local Network and turn on Visual Studio Code (or whichever app hosts your agent). The agent’s next ssh went through. If yours still fails, restart the app.

The app can be missing from the list if macOS never showed it the permission prompt. Apple’s TN3179 notes that the prompt doesn’t appear when a very short-lived process makes the request, and most agent commands exit right after they fail. Since macOS 27.2, the list includes a + button to add an app yourself.

Why only the agent

Local Network privacy covers traffic to addresses on a Wi-Fi or Ethernet network that goes directly to the host, not through a router. TN3179 lists who gets access without asking:

daemons started by launchd
programs running as root
“command-line tools run from Terminal or over SSH, including any child processes they spawn”

Everything else is charged to its “responsible code”. If an app spawns a helper and the helper connects to a local address, macOS checks the app’s permission. The agent’s shell is a child of VS Code, so ssh, nc, and ping run there under VS Code’s permission. The same binary, run by the same user from Terminal, is allowed.

That explains the other half of the symptom. Internet traffic goes through the router, so it isn’t local and works fine. TN3179 also exempts the device’s DNS server when it sits on the local network, and a home router is usually the DNS server it hands out by DHCP. That is probably why the router was the one host still reachable, and it is exactly how client isolation looks from the inside.

The same trap waits for any agent or tool that an app such as an IDE starts on macOS.

Check before you blame the network

Run ping from the agent’s shell. sendto: No route to host for a host on your subnet whose MAC is in arp -an means the Mac refused to send it.
Run the same command in Terminal. If it works there, check the app’s Local Network permission.
If it fails in Terminal too, the cause is the network or the routing table. Later that same evening, another VPN came up as utun6 and took the default route, which cut the whole Mac off from the subnet. This time the agent caught it in netstat -rn -f inet.

Takeaway

On macOS, an app without the Local Network permission gets No route to host for hosts on your subnet, and so does every tool it spawns. An agent running inside that app will diagnose a working network as broken.

If your agent runs inside an app on macOS, open Privacy & Security → Local Network now and check that the app is on the list and switched on.

The rollback that saved the host, and killed it 24 hours later

Sat, 17 Oct 2026 21:51:00 +0200

The rollback timer fired, the remote host got its address back, and the journal said all was well:

dhclient[1796120]: bound to 192.168.0.2 -- renewal in 34318 seconds.
systemd[1]: lxc-uplink-rollback.service: Deactivated successfully.

Exactly 24 hours later, to the second, the host dropped off the network and took SSH, the port forwards, and the VPN with it.

When a transient systemd-run unit finishes, systemd kills everything its command left running. If your rollback, or any other command run that way, starts a daemon, the daemon dies with the unit. Here it was dhclient, and the damage only showed when the DHCP lease expired.

The setup

Debian 13, systemd 257, ifupdown with ISC dhclient 4.4.3-P1
allow-hotplug enp0s25 / iface enp0s25 inet dhcp, with a 24-hour lease from the router
The only way in is SSH through a port forward to the host’s address; the console is days away

The dead man’s switch worked

Before a risky network change, an Ansible play arms a timer that undoes it, applies the change, and disarms the timer only if a fresh SSH login succeeds:

- name: Arm rollback timer
 ansible.builtin.command: >-
 systemd-run --collect --unit=lxc-uplink-rollback --on-active=5min
 /bin/sh -c 'ifdown -a --exclude=lo; ip link del br0;
 cp /etc/network/interfaces.rollback /etc/network/interfaces;
 ifup -a; ifup -a --allow=hotplug'

# ... apply the change ...

- name: Drop the cached SSH connection
 ansible.builtin.meta: reset_connection

- name: Prove a fresh SSH login still works
 ansible.builtin.wait_for_connection:
 delay: 10
 timeout: 420

- name: Disarm rollback timer
 ansible.builtin.command: systemctl stop lxc-uplink-rollback.timer

Without meta: reset_connection, Ansible reuses its open SSH connection and a broken change looks fine.

The change moved the NIC into a bridge, br0. The bridge got the old address back, but forwarded SSH from the internet stopped reaching the host. (I never found out why, and it isn’t the point here.) Five minutes later the timer restored the old config, the host came back, and I moved on.

What went wrong

Look at the rollback unit again. Its main process is /bin/sh. ifup starts dhclient, which forks into the background, and then sh exits. The unit is now done, and a finished service gets its cgroup cleaned up: by default (KillMode=control-group), systemd kills every process still in it. That includes the dhclient that had just been “bound”.

A killed dhclient doesn’t release the lease, so the address stays on the interface with a lifetime of 86400 seconds. Nothing renews it. When the lifetime runs out, the kernel removes the address, and avahi is the first to notice:

Oct 17 21:08:42 avahi-daemon[936]: Withdrawing address record for 192.168.0.2 on enp0s25.

The clue was in the journal on day one. When the rollback started, its dhclient logged:

dhclient[1796009]: Removed stale PID file

That stale PID file belonged to the dhclient on br0. Ansible had started it five minutes earlier from another systemd-run unit, and systemd had killed it the same way, but the rollback replaced it before anyone noticed.

At boot, ifup@enp0s25.service starts the same dhclient. That unit stays active (exited), so its cgroup is never cleaned up, which is how the boot-time dhclient had kept renewing for about six weeks.

You can see the behavior in two lines:

sudo systemd-run --unit=demo /bin/sh -c 'sleep 600 &'
sleep 1; pgrep -a sleep # nothing: systemd killed it when sh exited

The fix

To get back in, I went through a VM that had its own macvlan leg on the same NIC and ran sudo dhclient enp0s25. With ssh -J, -i applies only to the last hop, so give the jump host its key through ProxyCommand:

ssh -i ~/.ssh/key -o IdentitiesOnly=yes \
 -o ProxyCommand="ssh -i ~/.ssh/key -o IdentitiesOnly=yes -W %h:%p user@jump.example.com" \
 user@target.example.com

The real fix was to stop depending on a long-running DHCP client. The router reserves the address for the host anyway, so the host now has a static stanza. Ansible applies it without restarting networking, because that would cut off the run and the only way in:

ip address replace 192.168.0.2/24 dev enp0s25 valid_lft forever preferred_lft forever
ip route replace default via 192.168.0.1 dev enp0s25
pkill --signal KILL -f '(^|/)dhclient .*enp0s25$' || true

SIGKILL is deliberate: a dhclient that stops cleanly removes the address it holds, the one this very run is connected through.

If the host has to stay on DHCP, keep daemons out of transient units. Start the daemon through the unit that owns it at boot (here ifup@enp0s25.service), or let the transient unit leave its children alone with systemd-run -p KillMode=process. I didn’t retry the bridge, so I haven’t tested either variant in this rollback.

To check which unit a daemon really belongs to:

cat /proc/$(pgrep -o dhclient)/cgroup

Takeaway

Deactivated successfully only says the unit finished; whatever it started may have died with it. A killed DHCP client fails a whole lease later, long after you have stopped looking.

Search your playbooks and scripts for systemd-run calls that run ifup, dhclient or anything else that forks into the background. Then check which cgroup that process ends up in.

fail2ban that never banned anyone: '/bin/sh: 1: nft: not found'

Fri, 16 Oct 2026 15:33:00 +0200

fail2ban said the address was banned:

$ sudo fail2ban-client status nginx-limit-req
...
 `- Banned IP list:	203.0.113.10

From that address, right after:

$ curl -s -o /dev/null -w '%{http_code}\n' https://example.com/
302

The reason was in /var/log/fail2ban.log all along:

2026-03-14 10:21:05,205 fail2ban.utils [1293]: ERROR 7f5e8fbb0d30 -- exec: nft add table inet f2b-table
2026-03-14 10:21:05,206 fail2ban.utils [1293]: ERROR 7f5e8fbb0d30 -- stderr: '/bin/sh: 1: nft: not found'

On Debian, fail2ban bans through the nft command. A host that runs firewalld can easily lack it. fail2ban then counts failures, lists bans, and blocks nothing, including the sshd jail.

The setup

Debian 13, fail2ban 1.1.0-8, firewalld 2.3.1 with FirewallBackend=nftables
nginx as a reverse proxy in front of a few web apps, SSH forwarded from the router
fail2ban set up by hand, with the stock sshd jail

Debian’s /etc/fail2ban/jail.d/defaults-debian.conf picks the ban action:

[DEFAULT]
banaction = nftables
banaction_allports = nftables[type=allports]

How it showed up

I was adding jails for the web logins, so I tested them from a VPS outside every whitelisted network. Testing from home proves nothing when home is in ignoreip. Twenty-five quick POSTs to a login endpoint:

for i in $(seq 25); do
 curl -s -o /dev/null -w '%{http_code} ' -d client_id=probe https://example.com/login
done

The first 11 got 400 from the app, the rest 429 from nginx’s rate limit, and the nginx-limit-req jail banned the VPS as it should. Then the next request got a 302, as if nothing had happened.

sudo nft list table inet f2b-table answered sudo: nft: command not found. The log held ten lines like the ones above. Each failed ban left two ERROR lines in a log nobody reads, while fail2ban-client status still listed the bans.

The fix

Install the nft command:

sudo apt install nftables
sudo systemctl restart fail2ban

Leave nftables.service disabled, as the package installs it. Debian’s /etc/nftables.conf starts with flush ruleset, so enabling the service would wipe firewalld’s rules at the next boot. fail2ban doesn’t need the service, only the binary. It keeps its bans in a table of its own, inet f2b-table, next to firewalld’s inet firewalld.

On restart, fail2ban restores the bans it has on record, so the ones that had silently failed now take effect:

fail2ban.actions [1649343]: NOTICE [nginx-limit-req] Restore Ban 203.0.113.10

Give it a few seconds before you check. Right after the restart, nft list table inet f2b-table still said No such file or directory. A moment later, it showed the set and the rule:

table inet f2b-table {
	set addr-set-nginx-limit-req {
		type ipv4_addr
		elements = { 203.0.113.10 }
	}
	chain f2b-chain {
		type filter hook input priority filter - 1; policy accept;
		tcp dport { 80, 443 } ip saddr @addr-set-nginx-limit-req reject with icmp port-unreachable
	}
}

This time, curl on the VPS couldn’t connect at all (exit code 7). If you manage the host with Ansible or similar, add nftables to the package list alongside fail2ban so a rebuilt host doesn’t lose it again.

Why the binary was missing

firewalld talks to the kernel through the python3-nftables bindings, not the nft command, so it doesn’t pull in the nftables package. fail2ban only recommends it as an alternative:

$ apt-cache show fail2ban | grep ^Recommends
Recommends: nftables | iptables, whois, python3-pyinotify, python3-setuptools

iptables was already installed (firewalld recommends it), so apt considered the recommendation met. The ban action still called nft.

The same thing can happen with any ban action whose command isn’t on the host: the action fails, and the address stays on the ban list anyway.

Takeaway

Banned IP list shows what fail2ban decided, not what the firewall does. The only proof that a ban works is a blocked connection from the banned address, and grep ERROR /var/log/fail2ban.log tells you why one isn’t.

Get one of your own hosts to ban a machine outside its whitelist, then try to connect from that machine.

'Downgrading is unsupported' after a config restore: the update guard was right

Fri, 16 Oct 2026 00:00:00 +0000

After a configuration restore, an updater refused with:

that release predates the gateway runtime these session stores now use. Downgrading is unsupported.

The operator requested an update. The restored configuration selected an older code line than the databases required. Restoring the intended per-install channel corrected target selection; bypassing the guard would have put incompatible code over newer state.

A newer release number can select older capabilities

The checkout followed a development branch. Its session databases already contained runtime-ledger history. The restored configuration backup lacked the install’s channel record, so release selection fell back to the stable channel.

The stable release had a higher patch version, but it came from a release branch without those runtime-ledger tables. Version ordering did not describe the relevant capability ordering.

The historical guard checked two things: whether the proposed target descended from the running commit, and whether a non-descendant target contained the runtime schema marker. It also read the session stores for ledger rows. Its refusal named the affected stores and stopped before moving the checkout.

The implementation explained the consequence: older code could not delete or prune sessions with that runtime history. This was a specific compatibility check, not proof that every forward commit or every future schema change would be safe.

Repair target selection before changing code

The diagnosis followed the configuration lookup:

Identify the actual install path and the channel record associated with it.
Compare that record with the backup that was restored. Check defaults when the record is absent.
Resolve the updater’s proposed target to a commit, rather than relying on its release label.
Compare the target’s ancestry and required state capabilities with the running checkout.
Reassert the intended channel through the updater’s supported configuration command, then run its check-only mode.

The corrected check selected the development branch and reported available forward commits—that verified channel selection. The update itself was deliberately not run, so the record does not establish a completed runtime upgrade.

For a Git-based installation, this read-only check is useful once TARGET is set to the exact resolved commit. Run it inside that installation’s checkout:

git merge-base --is-ancestor HEAD "${TARGET:?Set TARGET to the resolved update commit}"

Exit status 0 means the target descends from the current checkout. Status 1 means it does not; other failures require investigation. A non-descendant can be a valid release-branch target, so this command alone is not a compatibility verdict. Check the particular schema capability named by the refusal.

A config-only restore can create a mixed generation

The database kept its newer state while the restored configuration lost an additive setting. In this implementation, channel records were keyed by install path identity. Restoring an older backup, or moving an install without recreating its record, could therefore change how later updates were selected.

Deployment automation should assert the intended channel after install or restore, then read it back through the supported interface. Do not hard-code a path-derived identifier from another machine. Following a development branch is appropriate only when that is the deliberate operating policy; it is not a universal fix for update errors.

For an intentional rollback, follow the application’s documented procedure for compatible code and pre-upgrade data together. Preserve the current state first, and account for work absent from the older backup. Editing a schema-version value or removing ledger rows merely to silence the guard would discard the evidence it uses to protect the store.

Add an update-target check to your restore drill before allowing the restored installation to update itself.

Rootless Podman backup without app.ini, and a restore that fails with 'Cannot change ownership to uid 100999'

Thu, 15 Oct 2026 17:23:00 +0200

Restoring a Gitea backup to a new host failed on every directory:

tar: data/gitea: Cannot change ownership to uid 100999, gid 100999: Invalid argument
tar: data/gitea/queues: Cannot change ownership to uid 100999, gid 100999: Invalid argument

That one is easy to fix. The worse problem made no noise at all, and I found it just before: the nightly backup didn’t contain app.ini. If you back up a rootless Podman volume with a plain tar as the host user, yours may be missing files too.

The setup

Gitea 1.27 and PostgreSQL ran in rootless Podman on a VPS, with Gitea’s data in a bind mount under ~/gitea/data. The nightly backup ran as the user who owns the containers:

podman exec gitea_db_1 pg_dump -U gitea gitea | bzip2 -z9 > ~/backup/gitea-db-$(date +%y%m%d).sql.bz2
tar c -C ~/gitea/data gitea | bzip2 -z9 > ~/backup/gitea-data-$(date +%y%m%d).tar.bz2

The target was Gitea in Docker, inside an unprivileged LXC container on Debian.

The archive looked fine. It unpacked, and it contained all the repositories.

Compare the archive with the live tree

Before restoring, I compared the archive’s file list with the live data. List the live side from inside the user namespace, so that nothing is hidden from you:

ssh vps.example.com 'cd ~/gitea/data && podman unshare find gitea -type f' | sort > live.txt
tar tjf gitea-data.tar.bz2 | grep -v '/$' | sort > backup.txt
comm -23 live.txt backup.txt > missing.txt
wc -l live.txt backup.txt missing.txt

 12359 live.txt
 12342 backup.txt
 17 missing.txt

Among the 17 were the files Gitea can least do without:

gitea/gitea/conf/app.ini
gitea/gitea/jwt/private.pem
gitea/git/.ssh/authorized_keys
gitea/gitea/avatars/... (3 files)
gitea/gitea/indexers/... (4 files)
gitea/gitea/sessions/... (2 files)

app.ini holds Gitea’s SECRET_KEY, its internal tokens, and the mail settings. private.pem signs its OAuth2 tokens. A restore from this backup would have started with fresh secrets: anything encrypted with the old SECRET_KEY, such as two-factor secrets, would no longer decrypt, and the mailer would be unconfigured.

Why tar skipped them

Rootless Podman maps the container’s UIDs onto the subordinate range in /etc/subuid. With the usual entry:

deploy:100000:65536

container root is you, and container UID 1000 (Gitea’s git user) is host UID 100000 + 999 = 100999. Files that Gitea creates with mode 0600 belong to 100999, and you can’t read them from the host. The repository objects were 0644 in 0755 directories, so tar read those, and the archive looked complete.

tar does complain, on stderr, and exits with status 2. In tar c ... | bzip2 > file, though, the pipeline’s exit status is bzip2’s, which is 0. Without set -o pipefail and someone reading the job’s stderr, nothing tells you.

Fix the backup

Run tar inside the user namespace, where you are root over every mapped UID:

set -o pipefail
podman unshare tar c -C ~/gitea/data gitea | bzip2 -z9 > ~/backup/gitea-data-$(date +%y%m%d).tar.bz2

The archive then records the IDs as the container sees them (1000), not the host’s 100999.

To rescue the files without touching the live host, fetch only the missing ones the same way, then merge them into the old archive with bsdtar’s @archive syntax:

ssh vps.example.com 'cd ~/gitea/data && podman unshare tar -c -T -' < missing.txt > missing.tar
bsdtar -czf gitea-complete.tar.gz @gitea-data.tar.bz2 @missing.tar

Fix the restore

GNU tar run as root restores the owners recorded in the archive (--same-owner is its default for root). Inside an unprivileged LXC container, root is itself a mapped user, and the container’s UIDs usually cover only 0 to 65535. UID 100999 doesn’t exist there, so chown fails with EINVAL, which tar reports as Invalid argument.

Don’t restore owners from an archive made somewhere else. Unpack with --no-same-owner and set the owners the new Gitea expects:

tar -C /var/tmp/gitea-restore --no-same-owner -xzf gitea-complete.tar.gz
cp -a /var/tmp/gitea-restore/gitea/. /srv/gitea/data/
chown -R 1000:1000 /srv/gitea/data/git /srv/gitea/data/gitea
chown -R root:root /srv/gitea/data/ssh

This also covers the merged archive, which carries 100999 for some files and 1000 for others.

After the restore, check against the live instance: the repository count, git fsck on every repository, and the SSH host key fingerprint on the Git SSH port, so existing clones still trust the new host. All three matched.

Takeaway

Under rootless Podman, a host-side tar leaves out the files the container keeps private, which are often its secrets, and the pipeline still exits 0. Comparing the archive’s listing with podman unshare find takes a minute and shows the gap.

Run the podman unshare find and comm check above against your own latest backup and see what comes back.

Decode a MikroTik autosupout.rif yourself: 20 lines of Python found the out-of-memory kernel panic

Wed, 14 Oct 2026 18:17:00 +0200

A MikroTik router kept rebooting every 10 minutes to 2.6 hours, with almost no traffic. After each boot, the log said:

system,error,critical router was rebooted without proper shutdown, probably kernel failure
system,error,critical kernel failure in previous boot
system,error,critical out of memory condition was detected
system,error,critical Automatic supout.rif file generated due to service malfunction, please contact MikroTik support and supply the generated file

You don’t have to wait for support. autosupout.rif is base64 with the bits in the other order, wrapped around zlib, and about 20 lines of Python turn it into plain text: the kernel’s last crash dumps, the log, /proc and the full config. That’s where the cause was:

Out of memory and no killable processes...
Kernel panic - not syncing: System is deadlocked on memory

The setup

MikroTik hAP ac² (IPQ4019, 128 MiB RAM, 16 MiB flash)
RouterOS 6.49.5, then 7.23.8 (long-term) with the legacy wireless package; both crashed the same way
Python 3 on a Mac to decode the file

The file format

Copy the file off the router and look at it:

scp admin@192.168.0.1:/autosupout.rif .
file autosupout.rif
head -2 autosupout.rif

autosupout.rif: ASCII text, with CRLF line terminators
--BEGIN ROUTEROS SUPOUT SECTION
uQWZiV3ZAgHnt3X72tNOSb+7xXFYO5HTy0WSEgfKeG3rdsd643J2xjtT37s5kVHKSIbOWiUNJVs9

It looks like base64, and base64 -d turns it into noise. The alphabet is standard, but each group of four characters is read as a 24-bit little-endian number: the first character holds the lowest 6 bits, and the first output byte is the lowest 8 bits. Decoded, each section is its name, a NUL byte, and a zlib stream.

The decoder

import os, re, sys, zlib

ALPHABET = 'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/'

def decode(text):
 text = re.sub(r'[^A-Za-z0-9+/=]', '', text)
 out = bytearray()
 for i in range(0, len(text), 4):
 group = text[i:i + 4].rstrip('=')
 value = sum(ALPHABET.index(c) << (6 * j) for j, c in enumerate(group))
 out += value.to_bytes(3, 'little')[:len(group) * 6 // 8]
 return bytes(out)

src, outdir = sys.argv[1], sys.argv[2]
os.makedirs(outdir, exist_ok=True)
sections = re.findall(r'--BEGIN ROUTEROS SUPOUT SECTION\r?\n(.*?)--END ROUTEROS SUPOUT SECTION',
 open(src, encoding='ascii').read(), re.S)
for n, section in enumerate(sections):
 name, _, data = decode(section).partition(b'\0')
 text = zlib.decompressobj().decompress(data)
 safe = re.sub(r'[^\w.-]', '_', name.decode(errors='replace'))
 with open(os.path.join(outdir, f'{n:02d}_{safe}.txt'), 'wb') as f:
 f.write(text)
 print(f'{n:02d} {safe} {len(text)}')

python3 supout.py autosupout.rif supout/

The 6.49.5 file had 76 sections and the 7.23.8 one 87. The useful ones:

Section	What’s in it
`oops`	the kernel log of the last few crashes, each with its timestamp
`log`	the RouterOS log since boot
`resource`	`/system resource` plus `/proc/meminfo`
`.proc`	binary-ish dump of `/proc`, including `slabinfo`, `vmallocinfo`, `buddyinfo` (use `LC_ALL=C grep -a`)
`export`	the full configuration
`.debug`	per-service debug output

What it showed

On 7.23.8, every crash in oops ended the same way, after a process table in which every process had the same score:

[ 691.378048][ T142] [ 150] 0 150 1667 153 28672 0 -1000 parser
...
[ 691.378492][ T142] Out of memory and no killable processes...
[ 691.378509][ T142] Kernel panic - not syncing: System is deadlocked on memory

RouterOS runs every process with oom_score_adj -1000, so the OOM killer has nothing to kill and the kernel panics. Any out-of-memory, however brief, is a reboot. 6.49.5 got there by another route, with the kernel set to panic on any OOM: Kernel panic - not syncing: Out of memory: compulsory panic_on_oom is enabled.

The processes weren’t the problem, at about 0.5 MiB each. resource, written a minute after boot, was:

MemTotal: 118248 kB
MemFree: 7368 kB
SUnreclaim: 52756 kB

One minute into the boot, the kernel already held 52 MiB of unreclaimable memory. The caches in slabinfo add up to only about 10 MiB, so the rest is allocated outside them. In two of the panics, other CPUs were inside the Wi-Fi driver (ath_poll, __w_qos_decrypt_mpdu in the wlan and ath modules), which makes it the main suspect, but turning one radio off didn’t settle it.

The same reading works on any embedded Linux box: check oom_score_adj in the OOM dump, then SUnreclaim against what the process table and slabinfo explain.

Read the timestamps with care

The router has no battery-backed clock. After a crash, it restores an old time, and NTP takes minutes to catch up, so log and supout timestamps right after a reboot were about 7 minutes early. Five “crashes” stamped within two seconds of each other were most likely one boot loop.

The supout itself was written a minute after the crash. The e-mail service tried to send the crash notice before PPPoE and DNS were up, died, and that service failure is what produced the file (.debug shows error: DNS resolve failed [code: 21] and then died).

Before you send it to anyone

The decoded sections held, in plain text, the PPPoE password, the WireGuard private key, a TSIG key, and a bearer token from a dynamic DNS script. Rotate those before you hand the file to MikroTik support or attach it to a forum post.

On a router this tight, delete the file after you copy it. The root of the file system is in RAM, and a 480 KiB supout is RAM the kernel needed:

/file remove [find name~"autosupout"]

Takeaway

A supout.rif holds the kernel log, /proc and your config, and you can read it in a minute. Next time your MikroTik writes one, decode it and read oops before you open a support ticket.

No DHCP for an LXC container while iptables counts the packets as ACCEPT: firewalld rejects them in its own nftables table

Tue, 13 Oct 2026 22:56:00 +0200

A new LXC container came up with an IPv6 address and no IPv4 address. According to iptables, the host allowed DHCP from the container bridge, and the counter on that rule kept growing:

$ sudo iptables -L INPUT -v -n | grep dpt:67
 0 0 ACCEPT tcp -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
 26 7956 ACCEPT udp -- lxcbr0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:67

Twenty-six packets accepted, and dnsmasq, listening on that bridge, never logged a single DHCPDISCOVER.

With nftables, accept only ends the chain it sits in. Any other base chain on the same hook still gets the packet. On this host, the next one belonged to firewalld, and firewalld rejected everything arriving on an interface that wasn’t in any of its zones.

The setup

Debian 13 host with the stock lxc-net bridge: lxcbr0, 10.0.3.0/24, dnsmasq at 10.0.3.1
firewalld 2.3.1 with FirewallBackend=nftables, default zone public
iptables 1.8.11, the nf_tables flavor Debian installs as iptables
an unprivileged Debian 13 container running systemd-networkd with DHCP=true

When lxc-net brings up the bridge, it adds its own iptables rules: ACCEPT for DNS and DHCP on lxcbr0, ACCEPT for forwarding, and a MASQUERADE for the subnet. It knows nothing about firewalld.

Following the packet

The first visible symptom was the container’s bootstrap script, which spent nine minutes waiting on DNS lookups that timed out. From there, each check moved the problem one step along the path:

The container sent requests. With debug logging, systemd-networkd showed DHCPv4 client: DISCOVER in the selecting state.
The host received them. The ACCEPT counter went from 20 to 26 to 29 while I made the client retry with networkctl reconfigure eth0.
dnsmasq had the right socket open: 0.0.0.0%lxcbr0:67.
dnsmasq received nothing. strace showed it sitting in poll(). During a retry, it read only an IPv6 router solicitation, which it answered. That is where the IPv6 address came from.

AppArmor (no denials), reverse-path filtering (loose on the bridge), br_netfilter (not loaded), leftover iptables-legacy tables (empty), and socket buffer drops (RcvbufErrors 0) all came up clean. The host had no tcpdump, and no nft either:

$ sudo nft list ruleset
sudo: nft: command not found

That left iptables as the only view of the firewall, and iptables shows only the tables it created. So the packet passed an ACCEPT at the input hook and never reached the socket. Listing the running services found a second firewall:

firewalld.service loaded active running firewalld - dynamic firewall daemon

$ sudo firewall-cmd --get-zone-of-interface=lxcbr0
no zone

Two tables on one hook

After apt install nftables (the binary is enough; leave nftables.service disabled), you can see both chains on the input hook. Trimmed:

$ sudo nft list chains
table ip filter {
	chain INPUT {
		type filter hook input priority filter; policy accept;
	}
}
table inet firewalld {
	chain filter_INPUT {
		type filter hook input priority filter + 10; policy accept;
	}
}

ip filter holds lxc-net’s rules, written through iptables-nft. inet firewalld runs right after it, and an interface that no zone claims falls through to the default zone:

chain filter_INPUT_POLICIES {
	...
	jump filter_IN_policy_allow-host-ipv6
	jump filter_IN_public
	reject with icmpx admin-prohibited
}

public allowed SSH and the DHCPv6 client, but no DHCP server and no DNS, so the DISCOVER hit the final reject. The zone chain also ends with meta l4proto { icmp, ipv6-icmp } accept, which is why the router solicitation got through.

The fix

Put the bridge in a firewalld zone. trusted accepts everything, including forwarded traffic, which the containers need for internet access:

sudo firewall-cmd --zone=trusted --add-interface=lxcbr0
sudo firewall-cmd --permanent --zone=trusted --add-interface=lxcbr0

Or with Ansible:

- name: Trust the container bridge in firewalld
 ansible.posix.firewalld:
 zone: trusted
 interface: lxcbr0
 permanent: true
 immediate: true
 state: enabled

The container got its lease on the client’s next retry, without restarting anything. trusted also lets containers reach every port on the host. If that’s too much, give the bridge a zone of its own that allows only DHCP, DNS, and forwarding. The same applies to any interface that another tool creates, such as a VPN tunnel.

Why the counter lied

The nftables wiki says that “an accept verdict … isn’t necessarily final”, while “drops take immediate effect, with no further rules or chains being evaluated.” Base chains on one hook run in priority order. An accept ends one chain, and a drop or reject in any chain wins. When two firewalls share a host, the stricter one decides, and the permissive one keeps counting packets that never arrive.

Most of the time you won’t see the second table. iptables -L lists only what iptables-nft created, and a host running firewalld may not have the nft command at all. (That missing binary also leaves fail2ban unable to ban anyone .)

Takeaway

A counter on an ACCEPT rule only proves that the packet reached that rule. When an accepted packet still doesn’t arrive, list every base chain on the hook and find the later one that drops it.

If your host runs firewalld alongside anything that writes its own iptables rules, run sudo nft list chains | grep hook and count the chains on input and forward.

"Ansible requires blocking IO on stdin/stdout/stderr" when an AI agent runs Ansible

Tue, 13 Oct 2026 22:34:00 +0200

The first time my coding agent ran Ansible, every command failed before Ansible had read a single file:

ERROR: Ansible requires blocking IO on stdin/stdout/stderr. Non-blocking file handles detected: <stdout>, <stderr>

--syntax-check too, and even ansible --version. If you see this from an agent, a CI job, or a wrapper script, Ansible isn’t broken. The process that started it handed it non-blocking file handles, and a few shell redirections replace them.

The setup

ansible-core 2.20 (Ansible 13.3.0 from Homebrew) on macOS
A coding agent (Claude Code in VS Code) that runs commands in its own shell and captures their output

The check has been in every Ansible CLI entry point since ansible-core 2.14 (2022).

What went wrong

O_NONBLOCK is a flag on the open file, not on the process, so every child that inherits the file descriptor inherits the flag too. The agent’s harness connects the shell’s stdout and stderr to non-blocking pipes, and the shell passes them on unchanged to ansible-playbook.

At startup, Ansible calls os.get_blocking() on each of the three handles and exits if any of them is non-blocking. The list at the end of the message names the handles that failed. That makes it a good diagnostic: the first failure in my session listed only <stderr>, because the command was ansible --version | head -3. The pipe to head had already given stdout a fresh, blocking handle, while stderr still pointed at the agent’s pipe.

Run this in the shell you suspect to see which handles are affected:

python3 -c 'import os; print([os.get_blocking(fd) for fd in (0, 1, 2)])'

False in any position means Ansible will refuse to start there.

Fix

Give Ansible three handles the shell creates itself:

ansible-playbook site.yml </dev/null 2>&1 | cat

| cat gives stdout a new pipe, which is blocking.
2>&1 sends stderr into that same pipe. | cat alone fixes only stdout.
</dev/null replaces stdin. My agent’s stdin was never flagged, but Ansible checks it too, and an agent can’t answer a prompt anyway.

The pipe has a side effect: the command’s exit status is now cat’s, which is 0. A failed playbook looks like a success unless you turn on pipefail (bash and zsh both have it):

set -o pipefail
ansible-playbook site.yml </dev/null 2>&1 | cat

To check the fix without touching your real shell, you can make all three handles non-blocking for a single command and try it:

NB='import os,sys; [os.set_blocking(fd, False) for fd in (0,1,2)]; os.execvp(sys.argv[1], sys.argv[1:])'
(python3 -c "$NB" ansible --version) 2>&1 </dev/zero | cat
# ERROR: ... Non-blocking file handles detected: <stdin>, <stdout>, <stderr>
(python3 -c "$NB" sh -c 'ansible --version </dev/null 2>&1 | cat') 2>&1 </dev/zero | cat
# ansible [core 2.20.1] ...

The outer | cat and </dev/zero keep the flag on throwaway handles, so your terminal stays blocking.

Why Ansible insists

The pull request that added the check explains it. On a non-blocking stdout, a large write can fail partway with [Errno 11] write could not complete without blocking, and Ansible’s output gets cut off, or the run dies with an “Unexpected Exception”. The check turns those intermittent failures into one clear error at startup. The PR names another way to end up there: ssh switches its handles to non-blocking while it runs and doesn’t always switch them back when it exits.

So the same error can turn up wherever a parent process hands down non-blocking pipes, CI runners included, or in a terminal after an ssh that didn’t clean up.

Make the agent remember it

The agent worked this out within two minutes, and later that night it wrote the fix into the project’s skill for provisioning VMs:

Run Ansible from Claude Code with `</dev/null` and pipe its output through `cat`.

The next morning, a session about a different service ran ansible-playbook --syntax-check without the redirections and hit the same error. It never loaded the VM skill, so it never saw the line. A rule about how to run a tool belongs in the file the agent reads in every session, not in the instructions for one task.

Takeaway

“Ansible requires blocking IO” means the process that started Ansible handed it non-blocking file handles. </dev/null 2>&1 | cat gives it blocking ones, and set -o pipefail keeps a failed run from looking like a success.

If your agent runs Ansible, put that line in the instructions file it reads every session (CLAUDE.md or AGENTS.md).

Snapshot precision 0.99 let a wrong text line pass all seven screen tests

Tue, 13 Oct 2026 17:04:00 +0200

A code review claimed that a list screen’s snapshot tests couldn’t fail on the very line they were written to pin: a small gray “Location” line under each search result. To check, I broke that line on purpose. The rows got the container’s short name instead of its full name, so every location line rendered the wrong text. Then I ran the suite:

✔ Test listScreenDeletedItemsShowTheirOrigin() passed after 0.296 seconds.
✔ Test listScreenSearchResultsShowTheirLocation() passed after 0.708 seconds.
...
✔ Test run with 7 tests in 1 suite passed after 0.793 seconds.

All seven passed against the broken code. If your snapshot tests record a whole screen at .device layout with a precision below 1.0, they may be just as blind.

Setup

The project used swift-snapshot-testing 1.19.2, with Swift Testing on the iOS simulator. A shared defaults file sets both tolerances:

public static let perceptualPrecision: Float = 0.90
public static let precision: Float = 0.99

The two gates do different jobs:

perceptualPrecision decides whether a single pixel counts as different. 0.90 absorbs anti-aliasing drift between developer Macs.
precision decides what fraction of pixels must match. 0.99 lets 1% of the image differ by any amount.

The 0.99 was there to unblock CI: one render drifted by about 3,600 pixels between machines. The comment next to it even warned that it was “wide enough to hide a small real regression”. Every screen test used it anyway.

The arithmetic

An iPhone 17 Pro device frame is 1179×2556 pixels.

1179 × 2556 = 3,013,524 px
1% of that = 30,135 px may differ freely
wrong location = ~900 px actually differed

A 1% budget sounds strict. On a full screen it’s about a 58-point square of arbitrary change, more than all the location lines on the screen combined. The test was recorded correctly and passed honestly. It just had no way to fail on the thing in its name.

Measuring the mutation

To size the gate, record the mutated renders, diff them against the committed references, then restore the references. A few lines of Pillow give a rough count (a pixel counts if any channel moves by more than 25):

import sys
from PIL import Image

a = Image.open(sys.argv[1]).convert("RGB")
b = Image.open(sys.argv[2]).convert("RGB")
pa, pb = a.load(), b.load()
w, h = a.size
n = sum(1 for y in range(h) for x in range(w)
 if max(abs(c1 - c2) for c1, c2 in zip(pa[x, y], pb[x, y])) > 25)
print(f"{n / (w * h):.2%} of {w}x{h}")

TEST_RUNNER_SNAPSHOT_TESTING_RECORD=all xcodebuild test ... -only-testing:MyTests/ListScreenSnapshotTests
git show HEAD:path/to/reference.png > /tmp/head.png
python3 pxdiff.py /tmp/head.png path/to/reference.png
git checkout -- path/to/__Snapshots__ # put the real references back

...DarkAppearance...: 0.03% of 1179x2556
...SearchResultsAtTheDynamicTypeCap...: 0.14% of 1179x2556
...SearchResultsShowTheirLocation...: 0.03% of 1179x2556
...DeletedItemsShowTheirOrigin...: 0.03% of 1179x2556

The smallest real defect was 0.03%, about 900 pixels, so the budget had to sit well below that. 0.9999 allows 301.

The fix

Give the screen suite its own precision and write the numbers down where the next person will see them:

as: .image(
 precision: Constants.precision,
 perceptualPrecision: SnapshotDefaults.perceptualPrecision,
 layout: .device(config: config)
)

private enum Constants {
 /// A device frame is 1179×2556 px, so the default 1% budget is 30,135 px: wider than every
 /// Location line on the screen together. The wrong name moves about 900 px; this leaves a
 /// 301 px budget, while `perceptualPrecision` still absorbs anti-aliasing drift per pixel.
 static let precision: Float = 0.9999
}

Then rerun with the mutation still in place. It has to go red:

✘ Test listScreenDeletedItemsShowTheirOrigin() failed after 0.594 seconds with 1 issue.
✔ Test listScreenSortMenuOpen() passed after 0.751 seconds.
✔ Test listScreenBrowsingTheRoot() passed after 0.807 seconds.
✘ Test listScreenSearchResultsAtTheDynamicTypeCap() failed after 1.220 seconds with 1 issue.
✘ Test listScreenSearchResultsShowTheirLocation() failed after 1.618 seconds with 1 issue.
✔ Test listScreenSearchWithNoResults() passed after 1.659 seconds.
✘ Test listScreenDarkAppearance() failed after 2.036 seconds with 1 issue.

The four cases that draw location or origin lines failed. The three cases without them passed, which is also correct. After reverting the mutation, all seven passed again at 0.9999.

A 301-pixel budget is tight, and I verified it on one machine. If cross-machine drift trips it, look at the drift before you loosen the number.

Shrink the frame when you can

For a single component, a tight frame beats a tight precision. The same project measured a toolbar button’s enabled-to-disabled color change at 1,808 pixels, or 0.06% of a device frame. Rendered at 140×60 with layout: .fixed(width:height:), those 1,808 pixels became 2.4% of the image, well past a 1% budget. Keep .device for tests that are really about the whole screen, and size their precision from the smallest change they claim to catch.

Takeaway

precision is a fraction of the image, so a fixed value forgives more pixels as the frame grows. Before you trust a green snapshot, break the thing it’s named after, count the changed pixels, and compare that with (1 - precision) × width × height.

Pick one snapshot test in your suite, break its subject on purpose, and see whether it goes red.

Cyber classifiers can interrupt offline pentest summaries

Fri, 09 Oct 2026 00:00:00 +0000

An agent spent three minutes reading local transcripts from a finished, authorized security exercise, then stopped with:

This request triggered cyber-related safeguards.

No live target was involved. The job was offline summarization, yet the classifier interrupted it before it could write the requested report. The practical fix was to treat security-history analysis as a resumable data pipeline: retain message provenance, checkpoint extracted evidence, and keep the handoff usable by another approved model or provider.

A flag-shaped string is not a captured flag

The first problem appeared before the classifier fired. A search through about 25 request dumps found dozens of flag-shaped strings. About 20 were spelling, capitalization, spacing, or hyphenation variants of one value.

Most variants appeared only in assistant reasoning and progress notes. The model repeatedly retyped a remembered value and changed it. A checkpoint then copied one mutation into later sessions.

Only strings preserved in tool results represented observed server output. To separate evidence from generation, use this origin-based distinction:

user request instruction or supplied context
assistant text generated claim; unverified
assistant reasoning generated hypothesis; unverified
tool call attempted action
tool result observed output from that action
provider error evidence about the run, not the target

This applies beyond CTF flags. Treat IDs, hashes, error codes, hostnames, and quoted responses in model prose as leads until the first tool result corroborates them. Exact-looking text is still generated text.

Offline processing still triggers cyber classifiers

The retained run used Opus 5.5. It parsed local request dumps, separated real and decoy-looking values, and extracted an existing SMB investigation summary. After that, the next model response was refused under the provider’s real-time cyber safeguard. The session recorded a cyber refusal and pointed to its Cyber Verification Program. The output file did not exist afterward.

Some source dumps also contained earlier cyber_policy failures from another provider along with a prompt to apply for Daybreak access. Because these records were historical input, they still increased the concentration of offensive-security language the summarizer saw.

A related authorized test had a similar shape: generate enough traffic against a non-production environment to trigger a WAF, then verify that the application displays and retries a CAPTCHA correctly. In a newer Opus session, the controlled load-test work was refused. The retained successful session used Opus 4.6 and built the test client, while a later session corrected its cookie handling and the request used to observe the challenge.

The awkward lesson is that task intent and classifier input are different things. In practice, “Summarize these old logs” can contain exploit commands, credentials, flags, and attack narratives. Likewise, “Test the CAPTCHA” can require traffic that resembles denial of service when reduced to request counts.

Make the audit resumable and model-neutral

Do deterministic work before asking a model to synthesize:

Parse each transcript into typed events and preserve the source file, timestamp, role, tool-call ID, and tool-result ID.
Extract exact-match artifacts only from tool results. Store assistant claims separately as unverified leads.
Write incremental JSON or Markdown after each source file. Never leave the only useful state in the model’s context window.
Produce a sanitized handoff containing the authorization scope, completed files, pending questions, and evidence pointers.
Verify the final report exists and reconcile its artifact counts against the extraction index.

Provider access programs are worth the setup when this work recurs. In my experience, GPT Daybreak Blue justified the hassle for authorized security-related tasks. Kimi, GLM, and older model versions have also completed useful work when a newer classifier blocked the same class of jobs. That is a fallback, not a bypass: preserve the same authorization boundary, do not weaken the requested scope, and use models approved for the work.

Before trusting your next transcript summary, choose one exact token from the report and trace it back to the tool result that observed it.

Inherited agent notes can send the next agent down the same dead end

Fri, 09 Oct 2026 00:00:00 +0000

A new agent can go through a failed investigation again if its first job is to absorb the conclusions of the previous agent. In the course of an authorised security exercise the user was repeatedly asking for a clean start as the prior progress files could be somewhat misleading. The useful reset was to establish a small verified baseline before using the inherited notes to select the next experiment.

A hint survived longer than its evidence

The conversation history had instructions to go it alone after a number of failed attempts at the task. Another handoff stated the prior agent had got lost and that its notes could be misleading to the replacement.

One concrete example concerned an environment hint favoring IPv6. A later SUMMARY.md excerpt said:

IPv4 and IPv6 behave identically everywhere tested

It was very explicit that the IPv6 hint in AGENTS.md didn’t apply in that case. That’s a useful clarification, with a caveat: this is a recorded summary of the checks, not an independent re-run of every network probe. It does support the removal of that hint for that particular surface. It doesn’t in any way establish equivalence for all services or a successful resolution of the problem. The outcome is still unknown.

Rebuild only enough context to choose a test

On a restart read the current task and the mandatory operating instructions. Observe the authorization boundaries and safety rules. Be very clear in your mind about the difference between the instructions and the factual observations about the environment. A line in AGENTS.md could be a binding rule, it could be a fact about the setup or it could be a hypothesis that you need to go out and check.

Then establish the facts required to inform the next decision. In a network investigation that might involve confirming the target endpoint, checking that it’s reachable and then making a baseline request to it. In a code issue scenario you’d want to reproduce the failure that was reported and then have a look at the current code before accepting any diagnosis from a previous developer.

Only then do you compare the inherited notes against that baseline. That way you don’t re-run the whole investigation and the new agent has its own baseline to work from.

A handoff instruction can be as short as:

Establish a minimal baseline from the current tool output.
Treat inherited diagnoses as hypotheses.
For each claim you make, you need to be able to locate the evidence or at the very least be able to run a very cheap check.
Keep the required operating and safety instructions in force.

Preserve the test behind a negative result

A note such as “IPv6 works better” is completely devoid of the endpoint, the request and the comparison that would actually make it useful. “No difference on these tested endpoints” is much more specific and much easier to verify.

For future handoffs, record:

The claim and its scope (including when it was checked).
The command or test and a pointer to its output.
What remains is an interpretation or an unproven lead.

Keep the failed attempts too and state what each one ruled out. A failed request with one set of credentials doesn’t rule out the whole protocol. The next agent needs to be able to make the judgement as to whether repeating that test would in fact add any evidence.

Copying an assertion into successive summaries doesn’t constitute independent corroboration. If the raw output is missing then unverified is the correct designation rather than promoting it to a fact.

Prior to your next agent handoff, pick the assumption that will drive the next action and attach the relevant tool output in support of that assumption. If you can’t find it then that will be the restart task.

Apache UserDir /home/* can publish private dotfiles

Wed, 07 Oct 2026 00:00:00 +0000

A personal-homepage feature can expose a user’s shell history even when directory listings are disabled. With Apache mod_userdir, a whole-home mapping such as UserDir /home/* maps a request for /~demo/.bash_history directly into that home directory. Restrict publication to a dedicated subtree, and verify that a direct request for a private canary fails while the public page still loads.

This is a filesystem-publication error. It requires neither path traversal nor a dotfile-specific Apache vulnerability: the requested file is already inside the configured web namespace.

How the mapping exposes private files

The Apache UserDir documentation distinguishes relative directory names from absolute-path templates. Compare:

# Unsafe publication boundary if this tree is readable and authorized:
UserDir /home/*

# Publish only the dedicated subtree:
UserDir /home/*/public_html

In the first form, /~demo/.audit-canary maps to /home/demo/.audit-canary. In the second, it maps to /home/demo/public_html/.audit-canary. Filesystem permissions and Apache authorization still determine whether the request succeeds. The mapping alone does not grant read access.

A leading dot is a shell convention, not a general HTTP access-control rule. Likewise, Options -Indexes prevents generated directory listings; it does not prevent fetching a known filename. A deployment may deny some dotfiles explicitly without covering every private file in a home directory.

Exercise the exposure without collecting secrets

In an authorized test, have the administrator create a harmless .audit-canary outside the public subtree and a public index.html inside it. Make the canary readable by the test server so the check tests the mapping, not an accidental permission denial. Do not loosen permissions on real private files.

Against a disposable Apache instance bound to loopback on port 18080, request:

curl --fail --silent --show-error http://127.0.0.1:18080/~demo/.audit-canary
curl --fail --silent --show-error http://127.0.0.1:18080/~demo/public_html/

The vulnerable fixture returned the private marker and the public marker, both with HTTP 200. That establishes unintended file disclosure without downloading anyone’s history or keys.

Real exploitation follows the same direct-file read. Shell history can contain passwords, tokens, or an otpauth:// provisioning URI. That URI contains the shared TOTP seed: anyone who obtains it can generate the same codes as the authenticator. If the same exposure also yields an unencrypted private key accepted by a service requiring key plus TOTP, both factors can be compromised. File disclosure alone does not prove that such credentials exist or that another service accepts them.

Prevent publication at the directory boundary

Use a dedicated public directory with an explicit participant policy:

UserDir disabled
UserDir enabled demo
UserDir /home/*/public_html

<Directory "/home/*/public_html">
 AllowOverride None
 Options -Indexes -FollowSymLinks
 Require all granted
</Directory>

This fragment assumes mod_userdir is loaded. Adapt it to the site’s authorization policy, and check for overlapping directory sections. Remove broad grants on home directories, and keep private files unreadable to the web-server identity. If homepages are unnecessary, turn off the feature entirely. Symlinks and user-controlled overrides deserve separate review because they can expand the effective publication boundary.

If secrets were exposed, closing the route does not revoke copies already obtained. Replace affected keys and tokens, re-enroll affected TOTP credentials, and review access logs. Deleting a provisioning command from history cannot invalidate the seed.

Verify the fix and distrust status-code folklore

I tested the mapping with Debian’s Apache 2.4.69 in a disposable local fixture containing only synthetic files. After adding /public_html to the template, the same private-canary request returned 404, and /~demo/ returned the public marker with 200. No authentication service or real credential was involved.

The fixture’s nonexistent username also returned 404. Do not generalize a particular server’s 403/404 pattern into a reliable account-existence oracle. In Apache 2.4.62’s mapper source , the absolute-template branch substitutes the supplied name without the home-directory lookup used by relative mappings. Later filesystem and authorization processing affects the response.

Before trusting a personal-homepage setup, put a nonsecret canary immediately outside its intended public directory and request it by its exact URL.

A format string in a TRACE logger can read the process environment

Tue, 06 Oct 2026 00:00:00 +0000

A TRACE handler that echoes a request with snprintf(response, size, request_line) does more than log attacker-controlled text. It turns the request line into a printf format string and can leak stack data in an authorized test, including pointers into argv and envp, through positional %p and %s conversions. The one-line fix is snprintf(response, size, "%s", request_line).

This is CWE-134, use of an externally controlled format string . The bug often hides in diagnostics because developers treat the third argument as plain text. For the printf family, it is the format string: a small program that says what to read and, with %n, where to write.

A bounded copy does not make the format safe

The length argument to snprintf bounds the output buffer. It does not stop format directives from being evaluated:

/* Vulnerable: request_line controls the format language. */
int len = snprintf(response, sizeof response, request_line);

/* Fixed: request_line is data for a constant format. */
int len = snprintf(response, sizeof response, "%s", request_line);

The same rule applies to printf(input), fprintf(log, input), and wrappers around them. When no formatting is needed, prefer fputs.

A small local regression fixture makes the difference visible without reading sensitive memory. Save this as format-log.c in a disposable directory:

#include <stdio.h>
#include <string.h>

static void render(char *out, size_t size, const char *input, int fixed) {
 if (fixed)
 snprintf(out, size, "%s", input);
 else
 snprintf(out, size, input);
}

int main(void) {
 char vulnerable[128], fixed[128];
 const char *probe = "TRACE /%p HTTP/1.1";
 render(vulnerable, sizeof vulnerable, probe, 0);
 render(fixed, sizeof fixed, probe, 1);
 printf("vulnerable=%s\nfixed=%s\n", vulnerable, fixed);
 return strcmp(fixed, probe) != 0 || strcmp(vulnerable, probe) == 0;
}

Run it locally:

cc -O0 -Wall -Wextra format-log.c -o format-log
./format-log

The vulnerable line expands %p; the fixed line preserves it literally. The pointer value is intentionally unspecified.

How to exercise the disclosure safely

On an authorized disposable target, start with a normal request and a literal-percent control. Then use %p or positional %N$p to map candidate argument slots. Dereferencing a candidate with %N$s asks the process to treat that value as a string pointer, so expect child crashes and test one slot per fresh connection. Never run a broad %s sweep against production.

Environment strings commonly sit near the initial argument and environment vectors. A pointer leak can therefore become a secret leak when a service receives credentials through environment variables. Slot numbers are build- and runtime-specific; a number from one binary is not a portable exploit.

Fork-per-connection servers make repeated probing easier than expected. Children inherit the parent’s already-randomized mappings so that addresses can remain stable across child connections until the parent restarts. ASLR still applies to the parent; fork() simply copies that layout.

Treat %n as a separate, higher-risk write primitive. A read-only demonstration is enough to prove the logging flaw and should be the limit of most authorized validation.

Fix the sink and reduce the impact

Make every format string a constant and enable compiler diagnostics such as -Wformat=2 -Wformat-security. Add a regression request containing %p, %s, and %n as literal text, then assert that the logger preserves every byte. Static analysis should flag tainted data reaching any printf-family format parameter.

Do not rely on input filtering. Blocking a few percent directives is easy to get wrong and leaves other format sinks exposed. Fix the call boundary.

Finally, keep long-lived credentials out of network-facing process environments where possible. Use narrowly scoped, short-lived credentials and isolate the component. That limits the damage from any memory disclosure, but it does not replace fixing the format string.

Search your diagnostic and TRACE paths for nonliteral printf formats before assuming they are harmless logging code.

Next.js Server Action arguments are Flight data: use error digests carefully

Tue, 06 Oct 2026 00:00:00 +0000

A Next.js Server Action can receive a JSON-looking body whose strings are interpreted as React Flight tokens. If ordinary input works but $-prefixed input changes the response or produces HTTP 500, inspect the decoded arguments before trying more SQL or shell payloads. Differences between error digests can help separate failure paths, but they cannot establish a vulnerable version or remote code execution on their own.

This matters when testing the request behind an action call. The normal client serializer escapes literal strings for the protocol; manually editing the wire body can change what the server decodes.

The application sees the result of decodeReply

In the tagged Next.js 15.0.0 action handler , fetch-action bodies pass through decodeReply or its streaming counterpart before the selected action runs. The React 19.0.0 reply decoder interprets $-prefixed strings, including escaped literals, special numbers, and references.

For a local decoder, these argument values illustrate the distinction:

Wire string inside the arguments array	Decoded argument
`ordinary`	string `ordinary`
`$$literal`	string `$literal`
`$Infinity`	number `Infinity`
`$-Infinity`	number `-Infinity`
`$undefined`	`undefined`

A value changing type before the application function runs explains why an otherwise boring search action can behave strangely. Input validation inside that function does not protect an earlier, vulnerable deserializer.

Reproduce the protocol locally before probing further

Use an isolated scratch directory. The following intentionally installs an old, vulnerable package for an offline decoder experiment. Do not deploy it, expose a listener, or feed it untrusted traffic. Delete the directory afterward.

LAB=$(mktemp -d "${TMPDIR:?Set TMPDIR to your scratch directory}/flight-lab.XXXXXX")
cd "$LAB"
npm install --ignore-scripts --no-audit --no-fund --save-exact \
 react@19.0.0 react-dom@19.0.0 react-server-dom-webpack@19.0.0

Save this as check.cjs in that directory, then run node --conditions=react-server check.cjs:

const assert = require('node:assert/strict');
const { decodeReply } = require('react-server-dom-webpack/server.node');

async function check() {
 const cases = [
 ['ordinary', 'ordinary'],
 ['$$literal', '$literal'],
 ['$Infinity', Infinity],
 ['$-Infinity', -Infinity],
 ['$undefined', undefined],
 ];
 for (const [wire, expected] of cases) {
 const args = await decodeReply(JSON.stringify([wire]), {});
 assert.ok(Array.isArray(args));
 assert.equal(args[0], expected);
 }
 console.log('5 Flight decoding cases passed');
}
check().catch(error => {
 console.error(error);
 process.exitCode = 1;
});

This checks protocol behavior, not exploitability. For an authorized application assessment, reproduce the actual resolved packages and runtime conditions. Next.js may bundle its React implementation; a standalone npm package is not automatically an exact framework replica.

Preserve the decoded root’s expected shape, too. The tagged Next.js handler invokes actionHandler.apply(null, boundActionArguments). A root that resolves to a primitive string fails that call’s array-like argument requirement before the intended action behavior can be tested. Keep transport decoding and action invocation as separate checkpoints.

What an error digest can tell you

The Next.js 15.0.0 error handler generates digests using stringHash over error text and stack information in several paths, unless a digest already exists. One path uses:

err.digest = stringHash(err.message + (err.stack || '')).toString()

Different parser branches can throw from different stack locations and yield different digests even when the visible error message is generic. Record the complete response, status, and digest for each controlled input; repeat baseline requests to check stability. Correlate differences with logs in the local replica.

A digest is a lossy fingerprint. Build paths, package versions, runtime stacks, and existing application digests can alter it. Distinct digests suggest distinct error material; matching digests do not prove identical execution paths. HTTP 500 alone does not even locate the failure in Flight decoding.

Separate protocol reachability from React2Shell

React’s CVE-2025-55182 advisory confirms unauthenticated RCE through a flaw in decoding Server Function payloads. Ordinary Flight tokens also exist in correct protocol handling. Observing the transformations above therefore does not diagnose that CVE.

For exploit validation, first reproduce the affected decoder in a private replica and demonstrate a harmless, deterministic effect there. Keep that result separate from any authorized remote confirmation. The experiment above deliberately stops at benign decoding.

For prevention, follow the current React and framework advisories, update the resolved server-side packages, rebuild, and verify what the running artifact actually contains. Do not use the historical demonstration versions as a patch recommendation. Authenticate and authorize actions independently, validate decoded arguments, and avoid exposing decoded-input debug echoes. Those controls do not replace fixing a vulnerable decoder.

When a $ probe changes behavior, capture the full response and reproduce that transformation locally before assigning a CVE or an impact rating.

Prompt injection: extract tokens, then decode them offline

Tue, 06 Oct 2026 00:00:00 +0000

A model can repeat hidden, encoded prompt fragments and still produce a different decoded secret on each attempt. When an authorized prompt-injection test reaches that point, stop asking the model to solve its own leak. Capture the raw tokens with position and case intact, then decode them with deterministic code offline.

This separates two questions: can the model disclose protected context, and can a decoder reconstruct the encoded value? A fluent final answer is poor evidence for either one.

Lexical refusal filters leave a semantic gap

A filter that rejects phrases such as “show the system prompt” or “reveal the secret” checks wording, not intent. Semantically equivalent requests can ask the model to quote, classify, count, or list text from its instructions without using the blocked nouns. If the model can return those fragments, the control has failed even when the obvious request is refused.

For an authorized assessment, use a synthetic canary in a test system. Ask for a numbered transcription of the relevant tokens, one token per line, with original capitalization. Repeat individual positions and compare exact strings. Keep the transcript as evidence. Do not put a real credential in the prompt merely to test whether it leaks.

Do not ask the model to join, normalize, or decode the result. Those extra transformations invite changed case, missing groups, repaired spelling, and invented separators. Temperature zero may reduce variation, but it does not turn generation into a parser.

Decode a synthetic transcript deterministically

Suppose a test prompt encodes characters as phonetic words. Capitalized words mean uppercase letters, lowercase words mean lowercase letters, and number words mean digits. Save this offline decoder as decode.py:

NUMBERS = {
 "zero": "0", "one": "1", "two": "2", "three": "3",
 "four": "4", "five": "5", "six": "6", "seven": "7",
 "eight": "8", "nine": "9",
}

def decode(tokens):
 result = []
 for token in tokens:
 lower = token.lower()
 if lower in NUMBERS:
 result.append(NUMBERS[lower])
 elif token.isalpha():
 letter = token[0]
 result.append(letter.upper() if token[0].isupper() else letter.lower())
 else:
 raise ValueError(f"unsupported token: {token!r}")
 return "".join(result)

sample = ["Alpha", "bravo", "Charlie", "delta", "seven"]
assert decode(sample) == "AbCd7"
print(decode(sample))

Run python3 decode.py; it prints AbCd7. The assertion is a negative control against accidental lowercasing or regrouping. Adapt the decoder only from a documented encoding rule. If a separator or group length is unknown, report that ambiguity instead of asking the model to guess.

Keep an evidence table with the position, raw token, repeated observation, decoded character, and rule used. Hash or otherwise preserve the raw transcript if chain of custody matters. A model-generated summary is not a substitute for the original response.

Prevent prompt disclosure at the trust boundary

Keyword filters are a weak secondary control. The primary fix is to keep secrets and privileged instructions out of model-visible context. Give tools short-lived, scoped credentials through an execution boundary that does not echo them into prompts or tool results. Treat retrieved documents, web pages, and user content as untrusted data, not instructions.

Constrain outputs structurally when the application expects a fixed schema, and validate them outside the model: test semantic paraphrases, indirect requests, translations, and token-by-token extraction against synthetic canaries. Log enough to identify which context source leaked without recording the secret itself.

A successful refusal to one phrase proves only that one phrase was blocked. Verify that the model cannot disclose the underlying text in pieces.

APFS sparse bundle on SMB: rsync "Operation not supported (45)" and F_FULLFSYNC

Tue, 29 Sep 2026 12:55:00 +0200

A backup job that rsyncs into encrypted disk images on a NAS started printing this for every file (paths elided):

rsync: [generator] failed to set times on …
rsync: [receiver] mkstemp …
rsync: [generator] delete_file: unlink … failed: Operation not supported (45)

Then hdiutil attach for the next image hung for over ten minutes, and the two images whose rsyncs had finished stayed mounted.

If you keep APFS sparse bundles on an SMB share and see error 45 inside them, the files aren’t the problem. The share can’t do F_FULLFSYNC, and APFS can’t commit a single transaction without it.

The setup

macOS 27.0.1 dropped AFP, so an old NAS that speaks only SMB 2.1 got mounted over SMB instead. Each backup source has its own AES-256 sparse bundle on the share, created like this:

hdiutil create -encryption AES-256 -type SPARSEBUNDLE -size 50g \
 -fs APFS -volname app -stdinpass /Volumes/backup/staging/app.sparsebundle

The job attaches a bundle, rsyncs a server’s files into it, and detaches it.

The kernel log

rsync names the files, so the obvious suspects are permissions or the files themselves. The kernel log points at the image instead (timestamps shortened, process column removed):

/usr/bin/log show --last 90m --style compact \
 --predicate 'process == "kernel" AND eventMessage CONTAINS "apfs"'

12:43:53.531 disk7s1 mount-complete volume app, requested by: mount_apfs (pid 82535); parent: mount (pid 82534)
12:43:53.532 disk7s1 Host is remote
...
12:47:45.393 (apfs) tx_flush_thread:1678: disk7 tx_flush(8311) failed, 45, # 297
12:47:45.931 (apfs) tx_flush_thread:1678: disk7 tx_flush(8311) failed, 45, # 298

APFS noticed the image lives on a remote host. From 24 seconds after mounting, it failed to flush a transaction about twice a second, with error 45 (ENOTSUP). The second mounted image did the same. Nothing written inside them was being committed.

The probe

Error 45 from a flush suggested the sync call, not the write. This script tries plain fsync and the two stronger syncs macOS offers. Run it from a directory on the share:

import fcntl, os
p = "_synctest"
fd = os.open(p, os.O_RDWR | os.O_CREAT, 0o600); os.write(fd, b"x" * 4096)
for name, op in (("fsync", None), ("F_FULLFSYNC", 51), ("F_BARRIERFSYNC", 85)):
 try:
 os.fsync(fd) if op is None else fcntl.fcntl(fd, op)
 print(name, "OK")
 except OSError as e:
 print(name, "->", e)
os.close(fd); os.unlink(p)

fsync OK
F_FULLFSYNC -> [Errno 45] Operation not supported
F_BARRIERFSYNC -> [Errno 45] Operation not supported

APFS asks the disk image to flush; the disk image passes that on as F_FULLFSYNC on its band files on the share, and the SMB client says no.

The NAS also had a share called TimeMachineBackup, and Time Machine over SMB needs full sync, so it seemed worth a try. Same result. Its Time Machine backups dated from the AFP days.

What works

Keep the image off the share. The bundles moved to a USB drive, and every healthy one attaches, writes, deletes, and detaches fine there. That’s where the job runs now.

If the image has to stay on the share, use HFS+ inside it. A test bundle on the same share, created with -fs HFS+J instead of -fs APFS, passed everything the job does: rsync with --delete-during, rename, setting times, deletes, detach and reattach.

If the server runs Samba and you control it, fruit:time machine = yes on the share makes vfs_fruit advertise the FULLSYNC volume capability to macOS clients. That should let APFS images commit, but it wasn’t tested here.

Check the images you already have

The failed commits also left damage. When the bundles were copied to the USB drive, one of them wouldn’t mount:

hdiutil: attach failed - no mountable file systems

A read-only check of its volume (fsck_apfs -n) said why:

warning: (oid 0x74e6) om: btn: invalid o_cksum (0xdfbd1ccee26c9006)
** The volume /dev/rdisk8s1 with UUID … was found to be corrupt and cannot be repaired.

Its band files had last been modified during the failed SMB run, and the original on the NAS was identical, so the copy wasn’t to blame. Since it was only a staging mirror, it got recreated empty, and the next run filled it again. After any such episode, attach each image and run diskutil verifyVolume on it.

Why restic didn’t mind

Restic repositories on the same share were unaffected. Restic is written in Go, and Go’s Fsync on macOS tries F_FULLFSYNC and falls back to plain fsync on ENOTSUP. The comment in the Go source names this exact case: “There are scenarios such as SMB mounts where fcntl will fail with ENOTSUP.” APFS inside a disk image has no such fallback: when the full flush fails, it stops committing.

Takeaway

An APFS disk image on an SMB share works only if the share supports F_FULLFSYNC. Without it, every write inside the image fails with error 45, and the failed commits can corrupt the image.

If you’re moving sparse bundles from AFP to SMB, run the probe above on the new share before you mount anything on it.

macOS SMB: OSError: [Errno 9] Bad file descriptor after a silent reconnect

Tue, 29 Sep 2026 12:15:00 +0200

A Python script that had encrypted backup archives on a NAS for months started failing on every run:

Traceback (most recent call last):
 File "/Volumes/backup/./encrypt-one.py", line 74, in <module>
 main()
 File "/Volumes/backup/./encrypt-one.py", line 38, in main
 with open(f, "rb") as src:
OSError: [Errno 9] Bad file descriptor

Then the bash script that called it stopped too. Not intermittently: start the job and something fails sooner or later.

If you run long jobs from an SMB share on macOS and get EBADF on files that are plainly there, the cause is probably a session reconnect you never saw, and the kernel log is the only place that records it.

The setup

macOS 27 dropped AFP, so an old NAS that speaks only SMB 2.1 got mounted over SMB instead. A bash script on the share runs encrypt-one.py on a few archives in parallel with xargs -P. Each run streams a 600 MB archive from the share through openssl enc and writes the result back to the same share.

The dead end

The traceback points at open(), so the first move was to open and read that archive by hand. It worked, and so did a read under xargs -P 3. A file that opens fine and a job that fails anyway tells you the file isn’t the problem. Something happens to the handle while it’s open.

The kernel log

smbutil gave the first hint:

$ smbutil statshares -m /Volumes/backup
...
SMB_VERSION SMB_2.1
SESSION_RECONNECT_TIME 2026-09-29 12:01:17
SESSION_RECONNECT_COUNT 4

Four reconnects; the last one was inside the failed run. The kernel log explains what each one does (timestamps shortened, process column removed):

/usr/bin/log show --last 1h --style compact \
 --predicate 'process == "kernel" AND eventMessage CONTAINS[c] "smb"'

11:58:15.502 (smbfs) smb_iod_sendall: id 0 ts > iod_lastrecv && ts > sr_timesent.
11:58:15.502 (smbfs) smb_iod_reconnect: id 0 Starting reconnect with nas.local
11:58:15.502 (smbfs) smb2_smb_parse_change_notify: smb_rq_reply failed 60
11:58:16.523 (smbfs) smb_iod_reconnect: id 0 The negotiate succeeded to nas.local
11:58:17.065 (smbfs) smb2fs_reconnect_dur_handle: Missing durable handle on <archive.tar.bz2.enc.tmp>
11:58:17.173 (smbfs) smb2fs_reconnect_dur_handle: Missing durable handle on <backup.sh>
11:58:17.186 (smbfs) smb2fs_reconnect_dur_handle: Missing durable handle on <archive.tar.bz2>
11:58:17.205 (smbfs) smb_iod_reconnect: id 0: Reconnect completed successfully.
11:58:17.212 (smbfs) smb2_smb_read_write_async: smb_rq_reply failed 9
11:58:17.212 (smbfs) smbfs_do_strategy: archive.tar.bz2: READ failed with an error of 9
11:58:17.283 (smbfs) smbfs_do_strategy: archive.tar.bz2.enc.tmp: WRITE failed with an error of 9

The NAS stopped answering under sustained I/O (error 60 is ETIMEDOUT). macOS dropped the session and reconnected, and that part worked. Then it tried to reattach each open file; the server had nothing to reattach them to, and every read and write on those files failed with error 9, EBADF.

Look at the middle line: backup.sh is on the list. Bash reads a script from disk as it runs, so the script file is open the whole time and goes bad with everything else. That’s why the shell stopped as well.

Why

A durable handle is how an SMB 2.x client keeps an open file across a dropped connection: the server holds the open for a while, and the client reclaims it after reconnecting. This server granted none. So “Reconnect completed successfully” is true for the session and useless for every file that was open at the time. Nothing reaches the application except EBADF on the next read or write.

What fixed it

Stop running long I/O jobs from the share. The pipeline moved to a USB drive, and only finished files go to the NAS. That ended the failures.

If you can’t move off the share, two workarounds got the job past the failing step:

Make bash read the whole script before it starts, so it never goes back to the share for the next line:
```
#!/usr/bin/env bash
{
set -euo pipefail
# ... the whole script ...
exit
}
```
Bash parses the entire { } block before running any of it, and the exit stops it from reading past the closing brace.
Make the I/O survive a reconnect: process the file in segments (64 MiB worked), open, write and fsync each segment separately, and on EBADF, EIO or EAGAIN reopen and retry only that segment. With AES-CBC you can chain the segments by using the last ciphertext block as the next IV, and the output is byte-identical to one openssl enc pass. In the test run on the share, no reconnect happened at all, probably because the per-segment fsync slowed the writes enough for the NAS to keep up.

If the server is a Samba box you control, check whether it grants durable handles: the durable handles option only takes effect with kernel oplocks = no, kernel share modes = no and posix locking = no. Turning off SMB signing on the client (signing_required=no in /etc/nsmb.conf) might also take enough load off a weak NAS CPU to prevent the stalls. Neither was tried here.

Takeaway

On macOS, an SMB reconnect to a server without durable handles invalidates every open file on the share, including a running shell script, and the only trace is Missing durable handle in the kernel log.

If you’ve seen Errno 9 on an SMB mount, run smbutil statshares -a and look at SESSION_RECONNECT_COUNT before you debug your code.

Zero failures is suspicious: agent checks that silently ran nothing

Tue, 29 Sep 2026 12:05:00 +0200

My coding agent ran each of the two flaky Rust tests 150 times and got this:

streamed_read_right_after_a_rename: 0/150 failed
renames_are_combined_across_changes_they_do_not_depend_on: 0/150 failed

Its next message was “Zero failures is suspicious.” It was right: neither test had run once. In the same week, three of its checks gave a tidy answer (nothing found, all passed, all failed) that came from a broken command, and each broken command hid its own error.

If your agent’s output has (eval):log:1: too many arguments, 0 passed; 0 failed; … filtered out or argument list too long somewhere above the summary, the summary is about a check that never ran.

The setup

Claude Code 2.1.283 on macOS, where the agent’s shell is zsh 5.9
A Rust workspace whose test binary is run directly from target/debug/deps

Trap 1: zsh has a `log` builtin

The agent was debugging an FSKit file system extension and searched the unified log for sandbox denials:

log show --last 30m --style compact --predicate 'process == "MyFS"' 2>/dev/null | tail -40

Nothing came back, and the agent told me: “No log entries so far.” Half a minute later it ran log stream … > /tmp/fs.log 2>&1 & and read the file. Its first line was:

(eval):log:1: too many arguments

zsh has a builtin called log that lists the logins watch reports. Builtins win over $PATH, so log show never reached /usr/bin/log, and 2>/dev/null threw away the only sign of it. You can check your own shell:

$ zsh -c 'whence -wa log'
log: builtin
log: command

With /usr/bin/log, the denial showed up at once, and the agent corrected its earlier claim. The same error turned up in a different project’s session days later.

Trap 2: `cargo test --exact` with the wrong path passes

The 150-run loop called the test binary like this:

$B --exact tests::$t -q > /tmp/out_$t.txt 2>&1 || f=$((f+1))

With --exact, the filter has to equal the full test path. The tests live in vfs::tests, so tests::streamed_read_right_after_a_rename matched nothing, and libtest treats that as success:

test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 93 filtered out; finished in 0.00s

Exit code 0, every time. Without --exact, the same filter would have matched as a substring. The agent listed the real names with $B --list, fixed the prefix, and the races then showed up under load.

Trap 3: an empty variable makes everything fail

After the fixes, the agent stress-ran the suite 400 times and looked up the binary first:

B=$(ls -t target/debug/deps/storage_core-* | grep -v '\.d$' | head -1)

The glob had grown past what ls accepts:

(eval):1: argument list too long: ls

failed runs: 400 / 400

$B was empty. zsh drops an empty unquoted word, so each run executed vfs::tests -q, got command not found into a log file nobody read, and counted as a failure. This time the agent saw the error right above the count and called the result bogus.

Fix

Make every check prove that it ran before you trust what it says.

Call /usr/bin/log by its full path, or put command in front of it, and keep stderr visible while you are still debugging:

/usr/bin/log show --last 30m --style compact --predicate 'process == "MyFS"' | tail -40

Count the tests that ran, not only the failures. One test per run should say 1 passed:

$B --exact vfs::tests::$t -q > /tmp/out_$t.txt 2>&1 || f=$((f+1))
grep -q '1 passed' /tmp/out_$t.txt || { echo "$t did not run"; exit 1; }

Get the binary path from cargo instead of a glob, and stop if it is empty:

B=$(cargo test -p storage-core --lib --no-run 2>&1 | grep -o 'target/debug/deps/storage_core-[0-9a-f]*')
: ${B:?test binary not found}

For the agent, add one line to its instructions file (CLAUDE.md or AGENTS.md):

When a check finds nothing, passes everything, or fails everything, first show that it ran:
how many items it checked, and that a known-bad input fails it.

Why it happens

All three are the same bug: the harness failed, and the failure looked like a result. A loop that counts exit codes can’t tell “no failures” from “no tests”, and a pipeline that ends in 2>/dev/null | tail can’t tell “no matches” from “no command”. Agents write exactly these one-liners, in zsh, many times per session, and they report what the last line printed.

The agent caught two of the three because the number looked wrong: zero out of 150 for known flaky tests, 400 out of 400 for a suite that had just passed. An empty log search looked plausible, so it went to the user as a finding.

Takeaway

A clean result from a check you haven’t seen fail is not evidence. Before you believe “nothing found” or “0 failed”, look for the count of what was checked.

Next time your agent reports zero failures, ask it how many tests actually ran.

xcodebuild says TEST SUCCEEDED, the xcresult says "Issues recorded without an associated test or suite"

Mon, 28 Sep 2026 12:16:00 +0200

A branch passed locally. xcodebuild test exited 0 and printed:

** TEST SUCCEEDED **

The fastlane coverage lane on the same branch failed with Tests have failed and didn’t name a test. Scan’s xcpretty output doesn’t show Swift Testing results, so the log didn’t include any failing tests. The result bundle had the answer:

xcrun xcresulttool get test-results summary --path Test.xcresult

{'result': 'Failed', 'totalTestCount': 2320, 'passedTests': 2317, 'failedTests': 1, ...}
testName: Issues recorded without an associated test or suite
failureText: Issue recorded: failed - Unimplemented: Unimplemented AppVersionClient.currentVersion …
 This issue was emitted from tests running in a host application.
 This can lead to false positives, where failures could have emitted from live application code at launch time, and not from the current test.

If your console is green and fastlane, Xcode Cloud or anything else that reads the .xcresult is red, check the summary for that pseudo-test before you go looking for a flaky test.

Setup

Xcode 26, Swift Testing, an app-hosted unit test target (tests run inside the real app process).
swift-dependencies 1.14, with a client whose testValue calls IssueReporting.unimplemented(...). That’s the usual way to make sure a test sets every dependency it uses.
The app’s root view is a login screen. On appear, it asks a backend for the minimum supported version and compares it with currentVersion().

What happened

Because the app hosts the tests, it launches as it does for a user, and its login screen sends the version request. When the response arrives, the screen reads the version client inside a test process, which resolves to the unimplemented test value and records an issue. No test is running on that code path, so the issue gets attached to no test.

The console run only shows it as a warning:

⚠︎ Test «unknown» recorded a warning at VersionClient.swift:24:1: Issue recorded
↳ failed - Unimplemented: Unimplemented AppVersionClient.currentVersion …

The result bundle counts it as a failure, though, and fastlane reads it.

Main didn’t fail, and the new branch didn’t touch any of this code. Its only addition was a new app-hosted suite that took about seven seconds. That extra time let the response arrive while tests were still running. On main, the run was over before the response came back. This is timing-dependent, so a faster backend or a slower one can hide the failure or bring it back.

Reproduce one target and compare the result bundle summaries:

Run	Total	Failed
Hosted target, as is	21	1
Same, new suite skipped (`-skip-testing`)	16	0
Same, with the fix	20	0

The 21 in the first row is 20 real tests plus the pseudo-test.

Dead end: set the dependency at launch

The obvious fix is to give the app a live value in its entry point:

prepareDependencies {
 $0.appVersion = AppVersionClient { Bundle.main.shortVersion }
}

It built and changed nothing. The same failure showed up again. In a test process, swift-dependencies resets its cached values for each test: it registers an XCTest observer that clears the cache whenever a test case starts. The value you prepare at app launch is gone once the tests start, and the later response reads the test value again.

Fix: don’t show the app UI while hosting tests

isTesting from IssueReporting (re-exported by Dependencies) is true inside a test process. Gate the scene’s content on it:

import Dependencies
import SwiftUI

@main
struct MyApp: App {
 var body: some Scene {
 WindowGroup {
 // Unit tests run inside this app. Its login screen would start a
 // network request whose response reads an unimplemented test dependency.
 if !isTesting {
 RootView()
 }
 }
 }
}

This is the workaround in the library’s Testing gotchas docs. Before applying it, check that no hosted test needs the app’s window or a view in the scene. Here, none did, because the tests only needed a real UIApplication, which still exists with an empty WindowGroup. The shipping app is unchanged, since isTesting is false outside a test run.

With the fix, the hosted target passed 20 of 20, the whole coverage lane reported 0 failures on both schemes, and the test run stopped sending requests to the dev backend.

That last part is a nice bonus. A test host that shows its real UI talks to real servers from your CI, and anything it logs (request and response bodies, anti-forgery tokens) ends up in your test logs.

Takeaway

An app-hosted test run also runs your app. Anything the app starts on its own at launch can record a failure that belongs to no test and shows up only in the .xcresult. Keep the UI out of the test host with if !isTesting, and when the console and the lane disagree, read xcresulttool get test-results summary first.

Try it on your own app-hosted target: grep a recent test log for Test «unknown» recorded. Each hit is an issue recorded outside any test, and the result bundle may already count it as a failure.

unattended-upgrades on Debian that installs nothing: Ubuntu-style Allowed-Origins

Sat, 26 Sep 2026 19:35:00 +0200

A Debian 13 server had automatic security updates. Ansible had installed unattended-upgrades, enabled it in 20auto-upgrades, and apt-daily-upgrade.timer had run that morning. Then a routine check:

$ apt list --upgradable 2>/dev/null | grep -c security
61

Nothing had failed, and nothing had warned. unattended-upgrades had never found anything it was allowed to install.

If your 50unattended-upgrades on Debian has Allowed-Origins with ${distro_codename}-security, yours does the same.

The config

This is what the Ansible role wrote to /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Allowed-Origins {
 "${distro_id}:${distro_codename}";
 "${distro_id}:${distro_codename}-security";
 "${distro_id}ESMApps:${distro_codename}-apps-security";
 "${distro_id}ESM:${distro_codename}-infra-security";
};

The ESM lines give it away: Extended Security Maintenance is Ubuntu Pro. This is Ubuntu’s stock file, copied into a Debian role by hand or by an LLM. On Debian it expands to “Debian:trixie” and “Debian:trixie-security”, which looks right and matches nothing.

Why it matches nothing

Allowed-Origins is the legacy syntax. unattended-upgrades (2.12 on Debian 13) converts each entry to the newer pattern format in get_allowed_origins_legacy():

allowed_origins.append("o=%s,a=%s" % (substitute(distro_id),
 substitute(distro_codename)))

The part after the colon becomes a=, and the matcher compares a with the archive, which is the Suite: field of the Release file:

elif what in ("a", "suite", "archive"):
 match = fnmatch.fnmatch(origin.archive, value)

On Ubuntu, suite and codename are the same string:

# security.ubuntu.com/ubuntu/dists/noble-security/InRelease
Suite: noble-security
Codename: noble

On Debian, they are not:

# security.debian.org/debian-security/dists/trixie-security/InRelease
Origin: Debian
Label: Debian-Security
Suite: stable-security
Codename: trixie-security

apt-cache policy shows both, as a= and n=:

release v=13,o=Debian,a=stable-security,n=trixie-security,l=Debian-Security,c=main,b=amd64

So a=trixie-security matches nothing, and neither does a=trixie, because the main archive’s suite is stable. Run through unattended-upgrades’ own matcher against a trixie host’s package lists, the legacy entries come up empty, and Debian’s patterns each hit their archive:

'o=Debian,a=trixie' -> []
'o=Debian,a=trixie-security' -> []
'origin=Debian,codename=${distro_codename},label=Debian' -> ['stable/trixie']
'origin=Debian,codename=${distro_codename}-security,label=Debian-Security' -> ['stable-security/trixie-security']

With no allowed origin, there is nothing to upgrade, and that’s not an error. Each run logs No packages found that can be upgraded unattended and no pending auto-removals and exits cleanly.

The fix

Use Origins-Pattern with codename=, as Debian’s own file does:

Unattended-Upgrade::Origins-Pattern {
 "origin=Debian,codename=${distro_codename},label=Debian";
 "origin=Debian,codename=${distro_codename}-updates,label=Debian";
 "origin=Debian,codename=${distro_codename},label=Debian-Security";
 "origin=Debian,codename=${distro_codename}-security,label=Debian-Security";
};

Debian’s stock file has the -updates line commented out. Keep it or drop it as you like.

a=stable-security would match too, but don’t use it. On release day, stable moves to the next Debian release, and a suite-based pattern would follow it. A codename stays with the release you installed.

Then check what unattended-upgrades actually picks:

sudo unattended-upgrade --dry-run -d

The Allowed origins are: line shows the patterns after conversion. Below it, the security packages from apt list --upgradable should show up as upgrade candidates. If the run ends with “No packages found” while apt list --upgradable lists stable-security packages, the origins are still wrong.

With Automatic-Reboot "false", a new kernel or libc still waits for you once updates flow again, so check for /var/run/reboot-required.

Takeaway

On Debian, Allowed-Origins "${distro_id}:${distro_codename}-security" is a no-op that looks like a security policy, because it matches the suite (stable-security), not the codename. Use Origins-Pattern with codename=, and confirm it with --dry-run -d.

Run grep -rn Allowed-Origins /etc/apt/apt.conf.d/ on your Debian hosts, and run the same grep over your configuration management repos, where the copy is most likely to live.

FSKit after mount -F: "ls: fts_read: Operation not permitted"

Sat, 26 Sep 2026 16:30:00 +0200

My new FSKit filesystem mounted without complaint, but nothing inside it could be read:

$ ls -la ~/mnt/storage-test
ls: fts_read: Operation not permitted
total 0
$ cat ~/mnt/storage-test/sub/a.txt
cat: /Users/me/mnt/storage-test/sub/a.txt: Operation not permitted

stat on the mount point worked. If your FSKit extension does the same after a mount -F from the command line, check its sandbox before your code: the extension can see the source folder’s metadata but may not read its contents.

The setup

macOS 26, a file system extension built on FSKit’s FSUnaryFileSystem that exposes a local folder through a mount point
A small host app, StorageHost.app, in /Applications; its executable runs mount -F -t storagefs <source> <mount point>
The extension StorageFS.appex with the FSKit entitlement and the app sandbox
Info.plist with FSSupportsPathURLs set to true and FSRequiresSecurityScopedPathURLResources set to false

What went wrong

The first mount was over ~/Downloads, so the obvious suspect was privacy protection (TCC) for that folder. A fresh ~/storage-test with one file in it failed the same way, which ruled TCC out and pushed the search elsewhere.

My coding agent’s first log queries came back empty, and it told me there were no log entries. That was wrong. In zsh, log is a builtin, so log show ... 2>/dev/null runs the builtin, sends its too many arguments error to /dev/null, and prints nothing. Call /usr/bin/log by its full path. With that, the kernel had the answer:

kernel (Sandbox) 3 duplicate reports for Sandbox: StorageFS(83505) deny(1) file-read-data /Users/me/storage-test

The extension’s own log showed the failures from FSKit’s side: readDirectory:...reply:error:1 and readFrom:...reply:error:1. Error 1 is EPERM, which ls turns into fts_read: Operation not permitted.

Fix

mount -F gives the extension a plain path, not a security-scoped URL, so a sandboxed process can’t read an arbitrary path. For development, grant temporary sandbox exceptions in the extension’s .entitlements file:

<key>com.apple.security.temporary-exception.files.home-relative-path.read-write</key>
<array>
	<string>/</string>
</array>
<key>com.apple.security.temporary-exception.files.absolute-path.read-write</key>
<array>
	<string>/Volumes/</string>
</array>

This covers your home folder and external drives. It doesn’t override privacy protection, so ~/Downloads, ~/Documents, and ~/Desktop may still be refused; test with a folder elsewhere. Temporary exceptions are for development on your own machine. A shipping extension should use security-scoped resources instead. FSRequiresSecurityScopedPathURLResources in Info.plist requests that; I haven’t managed a clean test yet (see below).

The same logs flagged two more problems that I fixed along the way:

-[FSModuleVolume(Project) getMaxFileSizeInBits]: Volume implements neither maxFileSizeInBits nor maxFileSize. One of them must be implemented.
-[FSVolumeConnector getStandardItemAttributesForItem:replyHandler:]_block_invoke: Reported attributes are incomplete

The first needs maximumFileSizeInBits on the volume. The second went away once the item attributes set flags, which cleared the log entry.

Reinstalling switches the module off

To test each change, I replaced the app: rm -rf /Applications/StorageHost.app, copied the new build, and launched it once. After that, mount failed outright:

Module com.example.storage.fs is disabled!
mount: Unable to invoke task

pluginkit disagreed. pluginkit -e use -i com.example.storage.fs succeeded, and pluginkit -m listed the extension with a +, which means enabled. FSKit still refused. It keeps its own switch in System Settings → General → Login Items & Extensions → File System Extensions, and I didn’t find a command-line way to flip it. After I turned StorageFS back on there, the next mount listed and read files, and appends, renames, truncation, and an 8 MB write-and-readback all worked.

The disabled module also spoiled my one test of security-scoped resources. With FSRequiresSecurityScopedPathURLResources set to true, mount failed with Unable to invoke task, but the reinstall had just switched the module off, so the failure says nothing about the setting.

Why

An FSKit module is an app extension, and FSKit requires it to run in the app sandbox. When an app hands a folder to the extension, sandbox access can travel with it as a security-scoped URL. mount -F on the command line passes only a string. The sandbox lets the extension look up metadata for that path, which is why the mount and stat worked, and denies file-read-data, which every directory listing and file read needs.

Takeaway

If an FSKit mount works and every read returns Operation not permitted, search the kernel log for deny(1) file-read-data with /usr/bin/log, then give the extension sandbox access to the source path. After every reinstall, check the File System Extensions switch before you trust a test result.

If you’ve got FSRequiresSecurityScopedPathURLResources working with a command-line mount -F, I’d like to hear how.

A failed CUDA build over SSH still launched the old executable

Sat, 26 Sep 2026 00:00:00 +0000

A remote command returned a CUDA compilation error followed by a normal simulation startup banner:

error: calling a __device__ function("__sinf") from a __host__ __device__ function("inflowFluctuation") is not allowed

The simulation started, but it ran an older executable. Semicolons let the launch proceed after nvcc failed. Correcting the source and rebuilding recovered the test; guarding the build-to-launch transition would prevent that misleading result.

The Claude Code and OpenPBS setup explains the local-agent/remote-command boundary. This happened during an early SSH-driven test, before we moved the long-running work into scheduled jobs. The shell mistake also applies inside a PBS script.

Diagnostics did not control execution

The command had this structure, with project paths removed:

nvcc -o solver solver.cu > build.log 2>&1;
grep -A3 error build.log | head;
nohup ./solver case.cfg > run.log 2>&1 < /dev/null &

The build wrote to an executable path that already existed. Compilation failed, leaving a runnable earlier build. Printing matching diagnostics didn’t stop the next command, and backgrounding the program didn’t establish whether the requested source had compiled.

The failed-build result included an ordinary banner. After replacing __sinf with sinf in the host/device function, the rebuilt program’s banner contained the newly added inflowTurb=0.08 parameter and subsequent progress lines. The earlier banner lacked that field. That difference provided evidence of which code path actually ran.

The transcript shows this operational recovery. Its retry still used semicolon-separated build and launch commands, so it did not repair the fail-open shell structure.

Make compiler status a launch prerequisite

For a future build, preserve the compiler’s status explicitly. This generalized Bash pattern builds to a temporary path and only publishes it after success:

#!/bin/bash
set -euo pipefail
cd "${PROJECT_DIR:?Set PROJECT_DIR to the remote project directory}"

candidate=$(mktemp ./solver.build.XXXXXX)
trap 'rm -f "$candidate"' EXIT
if nvcc -o "$candidate" solver.cu > build.log 2>&1; then
 mv -f "$candidate" ./solver
else
 cat build.log >&2
 exit 1
fi
exec ./solver case.cfg

This is a recommended guard, not code installed during the incident. The named source and configuration must exist in the remote project. Use a single build owner: concurrent publishers to ./solver require a separate ownership or versioned-artifact design.

When running remotely, send the whole script through SSH so its directory selection and status checks happen in the same remote shell. Avoid embedding local path expansions accidentally. For batch work, run the payload in the allocated job rather than adding another unscheduled nohup launch.

Reproduce the shell failure without CUDA

On a local machine with a POSIX shell, this harmless fixture leaves an existing command available after a simulated build failure:

sh -c 'solver() { printf "old executable ran\n"; }; false; solver'

It prints old executable ran and exits successfully. The guarded form prevents execution:

sh -c 'solver() { printf "old executable ran\n"; }; false && solver'

That form prints nothing and returns a failure status. For a real compiler test, deliberately introduce an error and verify that no new run starts. Also log a build identifier or other deliberate runtime identity; a healthy-looking startup line only proves that some executable ran.

Check your next remote build command for the exact status dependency between compilation and launch, especially where diagnostic pipelines sit between them.

A second launch of a resumable worker: 'OSError: [Errno 39] Directory not empty'

Sat, 26 Sep 2026 00:00:00 +0000

A resumable remote dataset driver failed while cleaning an unfinished task directory:

OSError: [Errno 39] Directory not empty

Another worker was still writing there. The second launch treated an unfinished task as safe to restart, even though the first process already owned it. Checking the surviving worker and validating its affected output recovered that work unit; no locking fix was installed during the session.

This was an early detached-run failure in the Claude Code, SSH, and OpenPBS workflow . Scheduler-managed execution later improved job tracking, but an application still needs to prevent conflicting owners.

The waiting wrapper had not disappeared

The first launch created a detached wrapper that waited for a pilot, then became the production driver. Its structure was:

until grep -q "finished" pilot.log; do sleep 30; done
exec python -u run_dataset.py dataset >> production.log 2>&1

Before the transition, a search for the production process would not find that driver. After exec, a search for the wrapper’s old command name would no longer identify it. Neither snapshot alone described the complete lifecycle.

A later check did not establish that the waiting launch was gone. Another detached production command started. The resulting log contained two identical pending-work counts followed by a traceback from shutil.rmtree(gdir). Both drivers had selected the same unfinished task, and one attempted cleanup while the other’s solver wrote files.

The resume logic used completion markers to skip finished groups. It offered no exclusive claim on an unfinished directory.

Recovery required checking the survivor

The follow-up inspected running processes, restored the affected task’s configuration, and checked the surviving computation. Its original log file had been lost during cleanup. A later driver result reported that the task was accepted.

A broad grep for success also matched text in the traceback. Tightening the check to a result-line pattern distinguished the actual accepted task from an error message that happened to contain matching characters.

That evidence supports recovery of the affected work unit. It does not prove that no intermediate data was lost or that the session did not add a lock. Eventual production completion checked in a later session also cannot establish that overlapping launches became safe.

Claim ownership before destructive resume

For a single-host driver using a filesystem with working advisory locks, a whole-run lock is one possible improvement. On a system with util-linux flock, run from the project directory:

flock -n .production.lock python -u run_dataset.py dataset

Every launcher must use the same stable lock path. Do not delete and recreate the lock file while work is running. The payload named here is the application’s driver, not a standard tool.

This guard is a new recommendation. A small local check can verify contention without running a dataset:

import fcntl
import tempfile

with tempfile.TemporaryDirectory() as directory:
 with open(directory + "/worker.lock", "w") as first:
 fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB)
 with open(directory + "/worker.lock", "w") as second:
 try:
 fcntl.flock(second, fcntl.LOCK_EX | fcntl.LOCK_NB)
 except BlockingIOError:
 print("second owner rejected")
 else:
 raise AssertionError("overlapping owner accepted")

Cluster filesystems and multi-node workers need an ownership mechanism tested under their actual locking semantics. Per-task claims can allow parallelism that a whole-run lock forbids. Stable PBS shard assignment solves a different problem: preserving each worker’s task set across starts and restarts. It cannot exclude two copies of the same shard.

Before retrying unfinished work, establish whether a live process already owns it. A missing completion marker only says that completion has not been recorded.

Claude Code over SSH: submit long-running work to OpenPBS

Sat, 26 Sep 2026 00:00:00 +0000

Claude Code can prepare a computation on your laptop and operate a remote cluster through ordinary SSH commands. In this setup, Claude Code ran locally on macOS; source uploads, compilation, queue submissions, and checks ran on a Linux server. OpenPBS handled the long-running computation after submission, so the laptop didn’t need to keep an SSH session open.

The operating rule was explicit: work estimated at one hour or more belonged in PBS. That threshold was a local policy, not an OpenPBS default. Use your site’s rules, including its restrictions on short tests and compilation on login nodes.

Establish the SSH connection yourself

Configure authentication and verify the server’s host key before asking an agent to connect. For example, in your local ~/.ssh/config:

Host compute-login
 HostName 192.168.0.2
 User researcher
 BatchMode yes
 ConnectTimeout 10

The address and account are placeholders. With your site’s actual values and authentication configured, run this from the laptop:

ssh compute-login 'hostname; pwd; command -v qsub; command -v python3'

BatchMode makes missing authentication fail instead of waiting for a password. It does not approve the connection or replace host-key verification. Give Claude permission for the intended remote actions through your normal tool-approval workflow; never paste a private key into the conversation.

Make the execution boundary explicit

A useful project instruction is:

Claude runs on the laptop. Use SSH for remote operations.
Keep this project's remote files in ~/batch-demo.
Submit tasks estimated at one hour or more through OpenPBS.
Follow site policy for shorter tests; do not bypass the scheduler.
Record the job ID after submission. Check logs and expected outputs before reporting completion. Ask before cancellation or deletion.

Save this with the project’s agent instructions. A prose rule still needs checking: our early sessions launched substantial work directly before the PBS rule was captured. Also, having a GPU on a login host does not grant permission to use it.

Submit a small smoke test first

These are generalized setup examples, not a copy of the original scientific workload. Create run.py locally:

import socket
import sys

print(f"worker host: {socket.gethostname()}")
print(f"python: {sys.executable}")
print("smoke test complete")

Create run.pbs beside it:

#!/bin/bash
#PBS -N agent-smoke
#PBS -l select=1:ncpus=1
#PBS -l walltime=00:05:00
#PBS -j oe

set -euo pipefail
cd "${PBS_O_WORKDIR:?PBS submission directory missing}"
exec "$PWD/.venv/bin/python" -u run.py

From the laptop, prepare and submit it:

ssh compute-login 'mkdir -p ~/batch-demo && cd ~/batch-demo && python3 -m venv .venv'
scp run.py run.pbs compute-login:batch-demo/
ssh compute-login 'cd ~/batch-demo && qsub run.pbs'

Record the exact job ID that qsub returns. The batch script selects its directory and interpreter without depending on an activated interactive shell. Install real dependencies according to site policy and check that the environment and project directory are accessible on the execution node. Shared paths and compatible Python installations are prerequisites, not promises made by SSH.

For GPU work, request the site’s documented GPU resources. The original array used select=1:ncpus=2:ngpus=1 per element. GPU visibility variables and binding depend on cluster configuration; inspect them inside an allocation rather than choosing an arbitrary device from the login host.

Follow the job through its outputs

On the login host, use qstat -f with the recorded ID for detailed status. Read the output file named in that status. Sites retaining finished-job history may also support qstat -xf for accounting. The upstream qsub manual documents submission options and PBS_O_WORKDIR; the qstat manual covers status queries.

An empty queue is insufficient evidence of success. For this smoke test, check the reported worker and Python path and the completion line. For production, also verify expected task outcomes and output contents.

The subsequent failures were about execution ownership and verification:

A failed remote build launched an old executable .
A second detached launch collided with resumable work .
PBS array resume needed stable task IDs .
Archive transfer started before the release was complete .

Before handing over a long run, submit the smoke test and check its output on the execution node’s actual environment.

max_retries_exhausted in a scheduled agent: alert on overdue outcomes, not request dumps

Fri, 25 Sep 2026 00:00:00 +0000

A scheduled agent produced 67 failed-request dumps over roughly two months, yet the archive contains no successful-turn record. The failure reports ended with:

max_retries_exhausted
non_retryable_client_error

Saving the request made diagnosis possible later, but it did nothing to alert an operator to the failed work. To surface unattended jobs, add an independent watcher that alerts when a verified outcome becomes overdue, even if the agent cannot produce a response.

What the failure archive actually proves

The records span five error shapes: a provider-side 503 circuit breaker, a broken pipe, an access-verification service error, a quota rejection, and an invalid-credential response. Counting the dump metadata gives 66 retry-exhaustion failures and one non-retryable client error.

That is evidence of repeated failed requests. It cannot establish that every scheduled invocation failed: successful work might have been recorded elsewhere, and a request can fail after earlier tool actions succeeded. Nor does the absence of a notification in a dump prove that no external monitor existed.

The operational problem exposed by this archive is narrower and useful: failure artifacts alone cannot answer, “When did the job last finish its work?” Because this corpus contains no success receipt to reconcile the errors, it demonstrates no monitoring repair.

Watch the outcome from outside the agent

For each scheduled job, define a durable success receipt tied to its actual acceptance condition. For a capture job, that might mean the destination readback confirms the captured items. A model response saying “done” is not sufficient.

Record at least a run ID, start time, terminal status, and the last verified success time. Keep partial progress separate so a retry can reconcile completed writes instead of duplicating them.

Then run a watcher independently of the model call:

Alert on a terminal failure, including a normalized error class and a run ID.
Alert when the last verified success exceeds the allowed interval plus a bounded grace period.
Treat a missing first success as overdue too. A watcher that only checks an existing timestamp can overlook a job that never worked.
Send the alert through a channel that does not require the failing model provider.

Deduplicate repeated alerts for the same incident, but keep an overdue-success alert active until someone acknowledges or resolves it. If alert delivery fails, track that failure as its own observable status.

Classify before retrying

The archive’s final credential rejection was classified as non-retryable. That classification is a reason to stop automatic retries and request an authentication/configuration check. It does not prove exactly which configuration change caused the rejection.

A broken pipe or service outage can justify bounded retries. Quota failures may require waiting for a published reset time. Preserve the specific provider error class: an HTTP status alone does not show whether the next attempt can succeed.

Pausing should produce an explicit incident state with a resume condition. Otherwise, “automatically disabled” becomes another silent failure.

Test the monitor with deliberately broken runs

In a disposable job, inject a retry-exhaustion failure and a non-retryable credential failure. Confirm both reach the independent alert channel. Suppress the runner entirely and confirm that the overdue-success check still fires. Finally, make the model report success without completing the destination write; that must not refresh the success timestamp.

Keep raw dumps access-controlled and retention-limited. They can contain the complete outgoing prompt and operational context; send a redacted summary and a restricted evidence reference in alerts.

Before leaving a job unattended, deliberately prevent one run from starting and check whether anyone notices.

QtInfoMsg is numerically above QtFatalMsg, so an INFO filter dropped every warning

Wed, 23 Sep 2026 16:00:00 +0200

A desktop app’s log file had about 20,000 lines in it, all of them INFO. There wasn’t a single WARN or ERROR line, even though the app had plenty of qWarning() and qCritical() calls. This was the filter in the custom message handler (trimmed):

void Logger::logMessage(QtMsgType msgType, const QMessageLogContext &context, const QString &message)
{
 if (msgType < minimumType && minimumType != QtFatalMsg && !message.startsWith("##")) {
 return; // below the configured level
 }
 ...
}

It assumes QtMsgType is ordered by severity. It isn’t:

enum QtMsgType {
 QtDebugMsg, // 0
 QtWarningMsg, // 1
 QtCriticalMsg, // 2
 QtFatalMsg, // 3
 QtInfoMsg, // 4, added in Qt 5.5
 QtSystemMsg = QtCriticalMsg
};

At INFO, minimumType is QtInfoMsg (4). Warning (1) and critical (2) are both below 4, so they were thrown away. Info lines got through, so the log looked healthy.

If your Qt message handler compares QtMsgType values with < or >, it has this bug.

How it got there

No single commit introduced this:

Before Qt 5.5 the enum was in severity order, so msgType < minimumType was correct when the handler was written.
Qt 5.5 added QtInfoMsg at the end, to keep the existing values stable. The app also added INFO to the end of its own level enum, and nobody looked at the comparison again.
Years later, a one-line change moved the fallback level from WARN to INFO. At WARN, the bug had only cost info lines. At INFO it cost warnings and errors, and since INFO was now the default, that became the normal behavior.

Each step looked harmless in its own diff. The cause only showed up when the history was read in order.

Red tests first

Each test sets a level, sends one probe per message type through the real handler to a real log file, and records which probes ended up in the file:

// One letter per probe that reached the file: d(ebug), i(nfo), w(arning), c(ritical)
QString writtenAt(Logger::LogLevel level) {
 log->setLogLevel(level);
 qDebug("probe d");
 qInfo("probe i");
 qWarning("probe w");
 qCritical("probe c");
 const QString content { logContent() };
 QString written;
 for (const QChar probe : QStringLiteral("diwc")) {
 if (content.contains(QStringLiteral("probe ") + probe)) written += probe;
 }
 return written;
}

TEST_F(LoggerTest, logMessage_InfoLevel_WarningWritten) {
 ASSERT_EQ(writtenAt(Logger::LogLevel::INFO), "iwc");
}

Against the old handler, five of the twelve logger tests failed:

Level	Expected	Written
INFO	`iwc`	`i`
WARN	`wc`	`iwc`
ERROR	`c`	`ic`
FATAL	(none)	`diwc`

The two bottom rows are separate bugs. Info (4) is above every other type, so it got through at WARN and ERROR. The minimumType != QtFatalMsg exemption meant FATAL wrote everything.

The fifth failure was a different bug: a level saved as ERROR came back as INFO after a restart. The app’s enum key was Error (an all-caps ERROR collides with a Windows macro), the setting was stored as ERROR, and QMetaEnum::keyToValue is case-sensitive. The lookup failed, so it fell back to the default.

Fix

Map the message types to severity explicitly, and compare those numbers instead:

// Qt numbers its message types in the order they were added, info last, not by severity
int severity(QtMsgType msgType) {
 switch (msgType) {
 case QtDebugMsg: return 0;
 case QtInfoMsg: return 1;
 case QtWarningMsg: return 2;
 case QtCriticalMsg: return 3;
 case QtFatalMsg: return 4;
 }
 return 4;
}

if (severity(msgType) < severity(minimumType) && !message.startsWith("##")) {
 return;
}

The FATAL exemption is gone. Each level now means “this severity and above.” If a future Qt version adds a type, the switch with no default makes the compiler warn about it, and the fallback treats the unknown type as most severe, so it gets logged rather than dropped.

The stored level is now matched without regard to case:

for (int i = 0; i < metaEnum.keyCount(); ++i) {
 if (stored.compare(metaEnum.key(i), Qt::CaseInsensitive) == 0) return LogLevel(metaEnum.value(i));
}
return LogLevel::INFO;

The same twelve tests passed after the fix, and so did the full suite.

The rollout catch

Fixing a filter that has silently dropped messages turns on log statements that nobody has read in years. Here, some of those warnings included user file names. The fix couldn’t ship until we moved those down to DEBUG. Before you fix a filter like this, grep for every qWarning and qCritical call it was dropping, because those are the lines your users will start sending you.

Takeaway

QtMsgType values are not ordered by severity. Never compare them with < or >. Map them first.
A log full of INFO lines with no warnings in it is suspicious, not reassuring.
Test the filter through the real handler with one probe per type, one test per level. The four-letter strings make the whole matrix fit on one screen.
Check enum-to-string round trips (QMetaEnum::keyToValue) with an actual save and reload, not just a set and get.

An UPDATE on encrypted columns wrote nothing for years, and three bugs took turns hiding it

Wed, 23 Sep 2026 15:42:00 +0200

A desktop app maintained an activity history in SQLite, but the UI froze on the first entry into any state. An action could start as ‘in progress’ and later complete, yet the UI would keep showing ‘in progress’ forever.

The update function looked fine at first glance, but nobody had actually run it under test, and it had 0% test coverage. That made the bug easy to miss and hard to trust.

The setup

The app encrypts the path columns as it writes them (column level encryption, not the whole database as with SQLCipher; see why fields rather than the file ). With a random IV per row, you can’t query the encrypted columns at all. To make rows matchable, derive the IV from the plaintext so the same path always encrypts to the same ciphertext. You store the IV alongside the row, base64 encoded:

history(id, run_id, path_enc, type, state, timestamp, iv)

You can’t put a plaintext path in a WHERE clause against path_enc. So the update does three steps:

Check the row’s IV.
Re-encrypt the current path with that IV, then SELECT to check whether the row still matches by id, run, path, and type.
Run an UPDATE with the same match predicate.

Reproduce with the real class

One test that stores, updates, and reloads through the real DAO, with an in-memory schema and a dummy key:

TEST_F(HistoryDaoTest, update_StoredUpload_LoadShowsNewState) {
 HistoryDao dao(&_database);
 Action upload { action(Action::Upload, Action::IN_PROGRESS) };
 upload.rowId = dao.store(&upload, 5);
 upload.status = Action::OK;

 dao.update(&upload, 5);

 EXPECT_EQ(load().value(0).state(), Action::OK);
}

It failed. The reloaded state was still IN_PROGRESS, which was precisely where the problem was.

Three faults, each hiding the next

QByteArray iv = queryIv.value(0).toByteArray(); // 1: still base64
...
query.prepare("SELECT id FROM history" // 2: no space before WHERE
 "WHERE id = :id AND path_enc = :path ...");
bool ok = database->exec(&queryIv); // 2: runs the IV query again
...
QSqlQuery update_query(db);
update_query.prepare("UPDATE history SET state = :state ... WHERE <same predicate>");
...
database->exec(&query); // 3: runs the SELECT, not the UPDATE
database->commit();
return true;

The IV was never decoded. store() decodes the base64 IV before encryption, but update() passed the IV through still encoded. The encrypt helper rejected an IV of the wrong length and threw, so every call stopped there. Before the length check was introduced, a wrong IV would have produced a ciphertext that was impossible to match. In either case, nothing was ever written. The bug had been there since the commit that introduced encryption, because the two sibling functions in that commit disagreed.
The guard SELECT never ran. A later refactor concatenated two string literals with no space between them (historyWHERE), so prepare() failed. The failure was only logged. Then exec() ran on the IV query, which succeeded. A follow-up “fix typo in query” commit fixed half of the SQL and left the missing space.
The UPDATE never ran. A rename to update_query missed the exec() call, which kept executing query instead.

Nothing ever got past fault 1, so faults 2 and 3 were never executed in production. Read the code as a chain of what would have happened if each fault had been fixed one at a time: fix the IV, and the function returns true while writing nothing; fix the SELECT as well, and fault 3 reruns a successful SELECT, commits, and returns true. Each fix would look like progress, and the result would be a function that reports success and writes nothing.

Date the bug before you blame the refactor

The first suspect was the 2022 refactor that added the second SELECT. Running git log -S on the decode call and reading store() and update() side by side in the encryption commit moved the start date back to 2019. That matters for the fix: the refactor wasn’t the thing to revert, and the original rule from before encryption was still the spec.

The tempting shortcut

The obvious simplification is to drop the IV lookup and the re-encryption and run UPDATE … WHERE id = :id. That passes the test above.

It also deletes the rule. The version from before encryption updated a row only while its id, run, paths, and type all still matched. Every later commit kept that predicate. Updating by id alone means a stale action can overwrite whatever row now sits behind that id. So the fix was one small change per fault, keeping the predicate:

QByteArray iv = QByteArray::fromBase64(queryIv.value(0).toByteArray(),
 QByteArray::Base64UrlEncoding);
query.prepare("SELECT id FROM history "
 "WHERE id = :id AND path_enc = :path ...");
bool ok = database->exec(&query);
...
database->exec(&update_query);

A second test pins the rule:

TEST_F(HistoryDaoTest, update_OtherPath_LeavesTheRowAlone) {
 HistoryDao dao(&_database);
 Action upload { action(Action::Upload, Action::IN_PROGRESS) };
 upload.rowId = dao.store(&upload, 5);
 upload.name = "other.jpg"; // same id, different path
 upload.status = Action::OK;

 dao.update(&upload, 5);

 EXPECT_EQ(load().value(0).state(), Action::IN_PROGRESS);
}

Before the fix, this test passed too, trivially, because nothing was ever updated. A negative test proves something only once its positive twin is green. After the fix, both tests passed, and so did the full suite (787 tests).

Side notes

QSqlQuery::size() returns -1 on SQLite, because the driver doesn’t report query sizes. A if (query.size() > 1) guard never fires there. Count with while (query.next()), or use SELECT COUNT(*).
Deriving the IV from the plaintext makes the encryption deterministic: equal paths give equal ciphertexts, which is what makes matching possible. With AES-GCM, never reuse that IV for a different plaintext. If you need equality lookups on encrypted columns, consider a misuse-resistant mode (AES-SIV) or a separate blind-index column (an HMAC of the plaintext) instead.

Takeaway

Test a write path as store → change → reload through the real class against a real database. “Returns true” and “rows changed” are different claims.
Check every prepare() and exec() result and stop on failure. A failure that’s only logged lets the function carry on and commit.
Use one QSqlQuery variable per statement and never pass the wrong one to exec(). After an UPDATE, check numRowsAffected(); zero rows is a valid SQL result, not an error.
When you fix a chain like this, expect the next fault to surface. Keep the reproducing test red until the whole round trip works.
Before simplifying a WHERE clause, find out why each condition is there. Pin the rule with a negative test.

Waiting for logout before login still left a race inside the handshake

Mon, 21 Sep 2026 14:49:00 +0200

Correction 2026-10-31: The earlier version attributed the reported failure to an idle timer. Later source review ruled out that timer for this app. The injected-teardown reproduction and exclusion-gate fix remain valid; the production trigger is unknown.

Users authenticated with Face ID, and the login failed anyway with a generic “login failed” error. The password was right, and the biometric enrollment was fine. The server’s log for the affected user showed the last handshake step arriving on a session in its initial state, with no user bound to it.

The app waited for any teardown already running before starting login. A dedicated test injected a new teardown after that wait and reproduced the same server-state signature. Holding an exclusion gate for the whole handshake fixed that reproduction, but did not establish the cause of the reported incident.

Setup

A stateful, multi-step login (SRP-style): start a session, send the identity, send the proof, finish. The server tracks progress per session cookie.
A session teardown: sign out on the server, then clear the local cookie jar. begin() returns immediately, and concurrent begin() calls join one teardown.
An earlier fix for a related bug: before its handshake, login called waitForCompletion() so a sign-out still in flight couldn’t destroy the new session.
The reproduction explicitly calls begin() during the handshake. It does not rely on an idle timer.

login: { credential in
 await sessionTeardown.waitForCompletion() // once, before the handshake
 return await client.login(credential) // four round trips
}

Reproduce with the real stack

A dedicated live test drove the production login code against a development server and fired begin() at a fixed delay into the handshake. At 500 ms:

start -> {"state":"new"}
send identity -> {"state":"identified", "salt":...}
send proof (M1) -> {"state":"proof-verified", "M2":...}
logout() <- the teardown fires here
POST /logout
finish (keys sent)
Logout success
Session teardown: cookie jar cleared
Error: state "new"
-> failed(.other)

This matched the server-state signature logged for the user. In the reproduction, the one-time wait had nothing to wait for: no teardown was running when login started. The test started one between two round trips.

The trap inside the trap: which effect does the damage

The code comments blamed the cookie-jar clear: the concluding step would go out without a session cookie and land on an anonymous session. That would suggest a small fix: defer only the local cleanup.

In the reproduction’s trace, the cookie jar was cleared after the finish step had already gone out with valid cookies. The server sign-out reset the half-authenticated session, and the concluding step then arrived at a session the server had put back to new. Deferring only the jar clear would still fail this exact trace.

Before you choose what to defer, read the ordering on the wire. The local effect you can see easily may not be the one that broke things.

Fix: hold exclusion for the whole protocol

The teardown gets a gate. Login holds it from the first round trip to the last. A begin() that arrives while the gate is held is honored but deferred—neither its sign-out nor its cookie clear runs until the handshake releases the gate.

private struct TeardownState {
 var inFlight: Task<Void, Never>?
 var handshakes = 0 // a count, so overlapping logins can't release each other's gate
 var isHeldBack = false
}

begin: {
 state.withValue { s in
 guard s.inFlight == nil else { return } // join the running one
 guard s.handshakes == 0 else { s.isHeldBack = true; return }
 s.inFlight = Task { await runTeardown() } // clears inFlight when done
 }
},
beginHandshake: {
 // Wait out a running teardown, then claim the gate. Loop, because the slot
 // is only free on a later pass.
 while let running = state.withValue({ s -> Task<Void, Never>? in
 guard let t = s.inFlight else { s.handshakes += 1; return nil }
 return t
 }) {
 await running.value
 }
},
endHandshake: {
 state.withValue { s in
 s.handshakes -= 1
 guard s.handshakes == 0, s.isHeldBack else { return }
 s.isHeldBack = false
 s.inFlight = Task { await runTeardown() }
 }
}

func excludingTeardown<T>(_ body: () async -> T) async -> T {
 await beginHandshake()
 defer { endHandshake() }
 return await body()
}

login: { credential in
 await sessionTeardown.excludingTeardown {
 await client.login(credential)
 }
}

The other properties stayed the same. begin() still returns immediately. A second begin() still means one teardown, whether that teardown is running or held back. A teardown whose sign-out throws still releases its waiters. Logout still happens: it runs right after login, which then ends in a signed-out state on purpose.

Later source review found that this app created its idle timer only after the full login completed and dropped it when the login screen appeared. The shipped revision had the same wiring. The earlier claim that this timer disproved a signed-in-only assumption was withdrawn. The gate provides exclusion without depending on that reachability argument.

A sibling app did retain its timer after logout and needed a signed-in guard. That was a separate timer-lifetime defect, not evidence that the timer caused this app’s reported failure.

Verification

Unit test for the new direction. A teardown begun inside excludingTeardown runs neither effect until the body returns. The expected order is [handshake, signOut, clearCookieJar]. Inside the body, the test awaits waitForCompletion() rather than Task.yield(), so a buggy teardown is guaranteed a chance to run. A yield would leave that to the scheduler.
Unit test for the old direction. A teardown already running still finishes before the handshake starts. That is the guarantee the earlier fix provided.
Injected-teardown sweep. The reproduction fires the teardown at 100, 200, 300, 400, 500 and 700 ms. It had been marked XCTExpectFailure. After the fix, every delay logged in:

teardown at 100ms -> loggedIn
teardown at 200ms -> loggedIn
...
teardown at 700ms -> loggedIn

The expected-failure marker was removed in the same commit.

What this doesn’t prove

The reproduction establishes a failure mechanism and verifies the exclusion fix. It does not prove that this mechanism caused the reported incident or identify its triggering call. We ruled out the idle timer for this app; the field trigger remained unknown.
The dedicated reproduction and the unit tests verify the fix. The live test class still exited with failures. They came from a separate host-app dependency problem in the test process, and another test in that class still tracks an unrelated second-login bug. Don’t read “the sweep passes” as “the live suite is green”.
The ordinary test plan passed, but live tests are opt-in there, so it says nothing about the server interaction.
Which delays hit the window depends on latency to the server. A sweep is better than a single delay, but it is still a sample.

Takeaways

Waiting once for cleanup before a multi-step protocol protects only that instant. If cleanup can start during the protocol, exclude it for the whole protocol.
Defer cleanup, don’t drop it. Queue the request and run it when the gate is released.
Count holders instead of using a flag, so two overlapping logins can’t release the gate from under each other.
Read the wire ordering before you choose what to defer. Here, the server sign-out did the damage, not the local cookie clear that everyone suspected.
A matching reproduction does not identify the production trigger. Trace the initiating call in the affected build before naming it as the cause of an incident.

Hourly agent cron vs. '429 usage_limit_reached': persist the provider's quota reset

Sat, 19 Sep 2026 00:00:00 +0000

An hourly scheduled agent repeatedly received 429 usage_limit_reached while its provider response already supplied a future reset time. One recorded delay was resets_in_seconds: 202825, about 2.35 days. The scheduler needed a durable not-before deadline checked before making another model call.

A failure archive contained 62 quota-related failed-request dumps, plus five failures of other kinds. Those are dump counts, not counts of individual HTTP attempts: a single invocation could retry several times before saving a dump. The archive does not establish that every scheduled invocation failed, and it contains no demonstrated scheduling repair or successful recovery.

A cron tick should consult shared cooldown state

A retry loop inside one invocation cannot govern the next invocation. When a process exits, an in-memory backoff disappears; the next cron tick can start the same futile sequence.

Persist the cooldown at the scope of the exhausted quota, such as the provider account and quota bucket. Every worker sharing that bucket must consult it before the first request. A different job name does not create a different quota.

The HTTP Retry-After specification permits either an HTTP date or a delay in seconds. Provider JSON fields such as resets_at and resets_in_seconds are provider-specific; normalize them in an adapter rather than assuming their units or presence. Use receipt time for relative delays. If multiple applicable constraints exist, do not schedule before any of them.

An illustrative gate, after parsing and validating those fields:

import math


def next_probe(existing, received_at, *, retry_after_at=None,
 quota_reset_at=None, margin=5):
 deadlines = [x for x in (retry_after_at, quota_reset_at)
 if x is not None]
 if not deadlines:
 raise ValueError("no usable deadline; use bounded fallback policy")
 values = [existing, received_at, margin, *deadlines]
 if not all(math.isfinite(x) and x >= 0 for x in values):
 raise ValueError("invalid deadline")
 return max(existing, max(received_at, *deadlines) + margin)


def may_probe(now, not_before):
 return now >= not_before


# Synthetic clock values; these are not recorded provider timestamps.
deadline = next_probe(0, 1000, quota_reset_at=1000 + 202825)
assert deadline == 203830
assert not may_probe(1000 + 3600, deadline)
assert not may_probe(deadline - 1, deadline)
assert may_probe(deadline, deadline)
# A later response cannot shorten an existing cooldown.
assert next_probe(deadline, 2000, retry_after_at=3000) == deadline
# Both applicable constraints survive normalization.
assert next_probe(0, 1000, retry_after_at=5000,
 quota_reset_at=9000) == 9005
print("cooldown boundary assertions passed")

The gate checks timing only. Production code also needs atomic storage updates, a lock or lease to admit one recovery probe, and clock-skew handling. On restart, read the saved deadline; do not silently treat unreadable state as permission to call. Add bounded jitter after the deadline so workers do not all resume together.

Give long waits an operational policy

A near-term transient rate limit can justify a short bounded retry. A multi-day quota exhaustion should explicitly defer the job, with its next eligible probe time visible to an operator. If a provider advertises an implausibly distant reset, escalate instead of silently sleeping forever or truncating the wait and hammering the endpoint.

At the deadline, admit one probe. A new quota rejection updates the cooldown. An authentication rejection needs intervention rather than the same timed retry. A permitted model response still does not prove the scheduled task completed: reconcile destination writes before replaying work that may have partially succeeded.

Test across process boundaries

The executable example verifies the early, boundary, and monotonic-deadline cases. To verify a real scheduler, inject a quota response, terminate the worker, and start a new invocation before reset. Assert that no provider call occurs. Start competing workers at reset and assert that only one obtains the recovery lease. Then inject another quota rejection and confirm that it persists the next deadline.

Stopping repeated calls can hide an overdue job unless monitoring remains independent of the cooldown. The companion post on unattended failure alerts covers that separate requirement. The disappearance of failure dumps after a reset is not a success receipt.

Before the next scheduled run, check whether a provider reset deadline survives a worker restart.

A read with legal effect: listing ISDS messages counts as delivery

Sun, 13 Sep 2026 14:09:00 +0200

Polling an inbox from a cron job looks harmless, since reading has no side effects. In the Czech Data Box system (ISDS, datové schránky) it does: the moment an authorized person logs in to the box is the moment every message waiting in it is legally delivered (doručení). An application counts as that person when it logs in and fetches the list of received messages. A 3 a.m. retrieval of a tax office’s request starts the deadline to answer it, whether anyone reads the message or not.

If your code talks to ISDS, every scheduled retrieval and every copy of your database that still holds the box’s password is a legal act.

Setup

An Odoo 18 addon files incoming Data Box messages into the company’s document store (the OCA dms addon). It talks to ISDS over SOAP with zeep (the parsing bug from that client is here ). A retrieval calls GetListOfReceivedMessages for the last 100 days and downloads each new message with its signed copy.

A possible next feature was a cron job, so that the documents appear without anyone pressing a button.

Why a read starts a deadline

Act No. 300/2008 Sb. delivers a message in one of two ways. Under § 17(3), it is delivered when a person whose permissions cover it logs in to the box. Under § 17(4), if nobody logs in within 10 days of the message’s arrival, it counts as delivered on the last day of that period, which Czech lawyers call delivery by fiction.

People rely on fiction as a buffer. A box that nobody opens for a week gives you up to 10 extra days before a deadline starts running. A nightly cron throws that buffer away. Every message is delivered the night after it arrives, and its deadline starts then, while the documents sit unread in a folder.

Not every call has this effect. GetOwnerInfoFromLogin checks the credentials and returns the box ID and owner without causing delivery. The addon uses it to validate a login before it creates anything.

The design

Once the domain model had the rule written down (“Every Retrieval causes Delivery of the Messages waiting in the box”), the decisions followed from it:

Each connection has a Retrieval Mode, On demand or Scheduled, and On demand is the default.
The “Check data box” button asks for confirmation: “Checking the data box counts as Delivery (doručení) of every Message waiting in it, which starts legal deadlines. Continue?”
Choosing Scheduled shows a warning, and the form keeps showing it for as long as the connection stays Scheduled. The daily cron job skips On demand connections.
The one-step setup wizard says that its first retrieval counts as delivery.

Copies of the production database

The usual way to get a realistic test environment for Odoo is to restore a production backup. That copy holds the ISDS username and password, and possibly a scheduled connection. A staging server that runs the cron, or a developer who presses “Check data box” to see what it does, delivers the production company’s official mail.

Odoo has a hook for this. When you run odoo-bin neutralize, or restore a database with Neutralize checked in the database manager, it runs each installed module’s data/neutralize.sql. Core modules use it to switch off outgoing mail and crons. The ISDS addon adds its own:

-- a neutralized copy must never cause Delivery: drop the Credential passwords
-- and stop Scheduled Retrievals
UPDATE dms_isds_connection
 SET password = NULL,
 retrieval_mode = 'on_demand';

-- unfinished Quick Starts hold a password too, and connecting retrieves at once
DELETE FROM dms_isds_quick_start;

A retrieval on a neutralized copy fails on the missing password before it contacts ISDS. The test checks exactly that. It runs the module’s neutralization queries, presses the button, triggers the cron, and asserts that the ISDS listing was never called:

self._neutralize()

with self.assertRaisesRegex(UserError, "password"):
 self.connection.action_check_data_box()
self.env.ref(CRON).method_direct_trigger()

listing.assert_not_called()

The development setup restores production data and neutralizes it on every refresh, so nobody has to remember.

Neutralization changes only the copy. The original backup still holds the password in plaintext, so restrict access to backups and encrypt them.

Takeaway

Before you schedule a “read-only” integration, find out what reading does on the other side. In ISDS, it is legal delivery, so the integration retrieves on demand by default, and a restored database can’t log in.

List the integrations in your own staging setup that still hold production credentials, and check whether any of their reads change something for someone else.

Zeep returns an empty list: a repeated xs:sequence lands in _value_1

Sat, 12 Sep 2026 23:59:00 +0200

A SOAP client listed received messages and got an empty list on every call, even though the mailbox had messages. No exception or error status occurred, and the unit tests passed. This was the line that read the response:

records = getattr(response, "dmRecords", None)
page = (records and getattr(records, "dmRecord", None)) or []

The XML on the wire had <dmRecords><dmRecord>…</dmRecord><dmRecord>…</dmRecord></dmRecords>, but zeep never created a dmRecord attribute. It put the records in dmRecords._value_1.

If zeep gives you None or an empty list for a field you can see in the raw response, check whether the schema repeats the sequence rather than the element.

Setup

An Odoo 18 addon pulls messages from the Czech Data Box system (ISDS), a government SOAP service with published WSDL and XSD files, and talks to it through zeep. I reproduced the behavior below with zeep 4.3.3 on Python 3.13. The list type in the ISDS schema looks like this:

<xs:complexType name="tRecordsArray">
 <xs:sequence minOccurs="0" maxOccurs="unbounded">
 <xs:element name="dmRecord" type="tns:tRecord" />
 </xs:sequence>
</xs:complexType>

maxOccurs="unbounded" sits on the xs:sequence, where you would usually expect it on the xs:element.

What went wrong

zeep maps the two forms differently. Here is a minimal schema with both, parsed with zeep’s own types:

<xs:complexType name="tRepeatedSequence">
 <xs:sequence minOccurs="0" maxOccurs="unbounded">
 <xs:element name="record" type="tns:tRecord"/>
 </xs:sequence>
</xs:complexType>
<xs:complexType name="tRepeatedElement">
 <xs:sequence>
 <xs:element name="record" type="tns:tRecord" minOccurs="0" maxOccurs="unbounded"/>
 </xs:sequence>
</xs:complexType>

Feed both the same two <record> children, then call serialize_object(..., dict):

tRepeatedSequence: {'_value_1': [{'record': {'id': '1'}}, {'record': {'id': '2'}}]}
tRepeatedElement: {'record': [{'id': '1'}, {'id': '2'}]}

With a repeated element, you get obj.record as a list. With a repeated sequence, each repetition becomes an item. The sequence has no name, so zeep calls the list _value_1, and each item is a dict keyed by the element names inside the sequence. A sequence with two elements would give you {"a": …, "b": …} per item.

The zeep docs explain _value_1 only under xs:choice and xs:any. Nothing there tells you that a plain repeated sequence ends up there too.

Plain attribute access would have failed loudly on the first real call:

AttributeError: tRepeatedSequence instance has no attribute 'record'

The getattr(..., None) or [] was written so that an empty mailbox wouldn’t crash. It also handled the case where the attribute doesn’t exist at all, and turned it into “no messages”.

The tests didn’t catch it because they replaced the client’s methods with mock.patch. They never sent a SOAP response through zeep, so they never saw how zeep shapes it.

Fix

Read _value_1 and unwrap each item:

records = getattr(response, "dmRecords", None)
page = [item["dmRecord"] for item in (getattr(records, "_value_1", None) or [])]

An empty <dmRecords/> still gives _value_1 == [], so the empty-mailbox case works.

To see the shape before you write the parsing code, dump the WSDL with zeep’s CLI:

python -m zeep dm_info.wsdl

ns0:tRecordsArray([dmRecord: ns0:tRecord])

The square brackets give it away. A repeated sequence prints with brackets around the field list. A repeated element, like tRepeatedElement above, prints with [] after the type instead: ns0:tRepeatedElement(record: ns0:tRecord[]).

Test the parsing with real XML

The test that covers this now runs canned SOAP XML through the real zeep client, with only the HTTP call patched:

def response(xml_body):
 result = requests.Response()
 result.status_code = 200
 result.headers["Content-Type"] = "text/xml; charset=utf-8"
 result._content = soap_envelope(xml_body)
 return result

with mock.patch.object(requests.Session, "post", side_effect=[response(listing)]):
 envelopes = client.list_received_messages(since, until)

self.assertEqual([e["id"] for e in envelopes], ["100001", "100002"])

Zeep still loads the WSDL, parses the envelope, and builds its objects, so a mapping mistake like this fails in the test suite with no network. Write the XML from the schema or from a captured response.

The same commit fixed a second or default with the same problem: return response.dmSignature or b"" would have stored a message without its signed copy. It now raises when the signature is missing.

Takeaway

zeep maps a repeated xs:sequence to _value_1, a list of dicts keyed by the inner element names. A getattr(..., None) or [] on top turns that into a silent empty result. Let missing attributes raise, and test SOAP parsing with real XML through zeep.

Search your zeep code for getattr( with a None default followed by or [], and check each field against the python -m zeep dump of your WSDL.

A background agent Docker build: 'open Dockerfile: no such file or directory'

Sun, 30 Aug 2026 00:00:00 +0000

A background Docker build failed immediately with:

ERROR: failed to solve: failed to read dockerfile: open Dockerfile: no such file or directory

The agent had supplied the project directory through the terminal tool’s workdir argument. The retry put cd inside the command itself. For an agent runner whose background path handling is uncertain, bind the directory in the command or make the Dockerfile and build context absolute.

The failed call already supplied workdir

In the historical runner, the background invocation had this shape, with project and image names generalized:

{
 "background": true,
 "workdir": "/srv/app",
 "command": "docker build --progress=plain -t local/app:dev -f Dockerfile ."
}

Docker could not open Dockerfile. The retry started with an explicit cd to the project before running the build. That corrected the launch form; whether the rest of that session succeeded is outside this lesson.

The error alone does not identify the actual launch directory. The record supports a runner-specific path-handling failure, not a rule that background processes generally discard their parent’s directory. Ordinary shell backgrounding with & inherits the shell’s working directory. An agent tool can launch its jobs through a different execution path.

A later check on the runner used to prepare this post found that background jobs preserved the preceding foreground directory, and an explicit background workdir was honored. The historical failure is therefore not a current universal behavior claim.

Bind both Docker paths deliberately

For an existing project at /srv/app, put the directory change and build in the same command:

cd /srv/app &&
 docker build --progress=plain -t local/app:dev -f Dockerfile .

The && prevents the build from running if the directory change fails. Alternatively, remove the directory dependency from both path arguments:

docker build --progress=plain -t local/app:dev \
 -f /srv/app/Dockerfile /srv/app

Making only -f absolute still leaves . as an accidental build context. Likewise, when using Compose, check its file selection and relative-path behavior rather than assuming a successful Docker build proves the next command has the right context.

Probe the actual execution mode

Before a multi-minute job, use a harmless foreground command to change to a known scratch directory:

cd "${TMPDIR:?Set TMPDIR to your scratch directory}" && pwd

Then submit pwd as a separate background tool call without workdir, and read that job’s captured output. Repeat with workdir set to the project root. Those two checks distinguish remembered foreground state from an explicit launch directory without building an image or starting services.

For a real build, print pwd and check test -f Dockerfile inside the same command that launches it. A foreground check in an earlier call cannot establish the background job’s directory.

Separate service startup from readiness checks

The same historical session hit another runner boundary. A foreground command combining docker compose up -d with a health-check loop was rejected with:

This foreground command appears to start a long-lived server/watch process.

The runner directed the agent to launch it with background=true, verify readiness, and run tests separately. That refusal came from the runner, even though Compose’s -d requests detached containers.

Keep service startup in its own call, following the runner’s execution rules. Check readiness in a separate bounded command, and fail explicitly if the deadline expires. Starting a service and proving it healthy are separate results.

Before your next background build, compare its reported directory with the intended build context using the harmless probe above.

Uninstalling one Odoo module can remove its dependents and their data

Sun, 30 Aug 2026 00:00:00 +0000

An Odoo module replacement passed a rehearsal’s record-count and accounting-total checks, yet the uninstall also removed a dependent addon that owned important business data. The application still loaded. The acceptance checks had missed the field that mattered.

Before replacing a module, inspect its reverse dependencies and data ownership. Rehearse the corrected transition on a restored copy, then compare the affected values by record identity before approving the live change.

Uninstall computes a reverse-dependency set

In Odoo 18’s module implementation , downstream_dependencies() recursively finds modules that depend on the selected modules. button_uninstall() marks both the selection and those dependents as to remove.

The same source describes module_uninstall() as removing database structures created by the module, including tables and columns. Actual deletion depends on the affected records and metadata; do not assume every business record disappears, or that surviving rows retain every field.

Generalize the replacement to a full addon, a smaller replacement, and an extension that depends on the full addon. Installing the smaller addon does not automatically redirect the extension’s dependency. Uninstalling the full addon still includes that extension in the removal set.

The first rehearsal had the wrong acceptance criteria

The historical transition checked surviving records, aggregate totals, and registry health. Those checks passed while the dependent addon was also uninstalled. The user caught the missing preservation requirement and required a restore.

The subsequent investigation found a second problem: changing the extension’s manifest alone would be insufficient. The smaller replacement did not yet implement a field the extension required. That field’s behavior needed to be ported before changing the dependency.

The retained record includes the corrective restore and a later clone’s uninstall/registry-reload output. It ends with the corrected rehearsal and final verification still open. It does not prove a successful final transition or a completed before/after digest comparison.

Rehearse the field handoff explicitly

A safer transition procedure is:

Quiesce writes and capture a consistent database and filestore backup. Verify checksums and prove that the pair restores.
Record installed module states, reverse dependencies, and the fields and records that must survive. Keep the addon source revisions with the backup.
Implement the required field behavior in the replacement, including compute and reversal behavior where applicable. Version-control the dependent addon too.
On an isolated restore, install the replacement and upgrade the dependent addon so the database’s module metadata reflects the new dependency. Inspect the removal set again before uninstalling the old addon.
Complete the uninstall, reload the registry, and compare the required values and behavior. Investigate field metadata and ownership during the handoff; identical field names alone do not establish safe transfer.

Disable scheduled jobs and outbound integrations on the clone. A rehearsal should not send mail, contact payment services, or write to production storage.

Compare identity and value, not only counts

For a small affected dataset, compare canonical exports directly. For larger exports, a digest is convenient if the query scope, ordering, and serialization remain identical.

This synthetic Python check demonstrates why counts and non-null counts are insufficient. It models two records whose field values were exchanged; it is not an Odoo migration test:

import hashlib
import json


def digest(rows):
 payload = json.dumps(sorted(rows), separators=(",", ":"), ensure_ascii=True)
 return hashlib.sha256(payload.encode("utf-8")).hexdigest()


before = [(1, "2026-01-10"), (2, None), (3, "2026-01-20")]
after = [(1, "2026-01-20"), (2, None), (3, "2026-01-10")]
assert len(before) == len(after)
assert sum(value is not None for _, value in before) == 2
assert sum(value is not None for _, value in after) == 2
assert digest(before) == digest(list(reversed(before)))
assert digest(before) != digest(after)
print("counts match; identity/value digest detects the change")

For a real comparison, export the stable record ID alongside each affected field, preserve nulls, and normalize dates and other types explicitly. A missing column must fail the check. Also test compute behavior and important workflows: a value digest cannot prove those work.

Before your next uninstall, list which dependent addons and field values must remain, and make their survival a rehearsal acceptance criterion.

ISDOC.PDF: an XML invoice in a PDF, and the VAT date it can't structure

Sun, 09 Aug 2026 05:06:00 +0200

Czech VAT law lets you put a month of separate supplies on one tax document, as long as each supply states its own date. ISDOC, the national e-invoicing format, has one tax date per document. Move that element onto an invoice line and validation fails:

Element 'TaxPointDate': This element is not expected.
Expected is one of ( Note, VATNote, Item, ... )

A human reading the PDF sees every date. The accounting software importing the XML sees one.

What ISDOC is

ISDOC is an XML invoice format, free to use, and maintained by the Czech Ministry of the Interior as the “national standard for electronic invoicing.” The current version is 6.0.2 from March 2022. Many accounting programs open it directly, so nobody retypes the invoice. A 2018 Ministry of Finance notice also lists ISDOC 5.2 and later among the formats central government bodies must accept.

A document comes in three representations :

.isdoc: plain XML in UTF-8
-isdoc.pdf: the XML embedded in a PDF/A-3, known as ISDOC.PDF
.isdocx: an archive with a manifest and attachments

ISDOC.PDF is the practical choice when you don’t know the recipient’s software. It opens like any PDF, and software that understands the format extracts the XML. German ZUGFeRD and French Factur-X use the same idea.

Making an ISDOC.PDF

The spec’s requirements for the PDF:

The PDF is PDF/A-3a, and the file name ends in -isdoc.pdf.
The embedded XML is named exactly invoice.isdoc, with subtype text/xml.
Its /Filespec is listed both in the catalog’s /AF array and in the /EmbeddedFiles name tree.
The /Filespec has /F, /UF and /EF, and /EF holds both F and UF stream entries.
/AFRelationship is /Source if the PDF was rendered from the XML, or /Alternative if both came from your database.

Generate the XML first, since LaTeX has to find it at compile time. With pdflatex, the preamble is short:

\usepackage[a-3b]{pdfx} % needs \jobname.xmpdata next to the .tex
\usepackage{embedfile}
\embedfile[
 filespec=invoice.isdoc,
 mimetype=text/xml,
 afrelationship=Alternative
]{invoice.isdoc}

If you already have a PDF, the official repository has a Java tool that joins a PDF and an ISDOC file and links to a Ghostscript-based script. Its examples/output folder is the reference to compare against.

The ministry doesn’t run a validator , so check the output yourself:

xmllint --noout --schema isdoc-invoice-6.0.2.xsd invoice.isdoc
pdfdetach -list invoice-isdoc.pdf # 1: invoice.isdoc
pdfdetach -save 1 -o extracted.isdoc invoice-isdoc.pdf
cmp extracted.isdoc invoice.isdoc
pdfinfo -meta invoice-isdoc.pdf | grep pdfaid
pdfinfo invoice-isdoc.pdf | grep Tagged

These commands don’t prove PDF/A conformance. For that, use a validator such as veraPDF. XSD validation also doesn’t run ISDOC’s Schematron rules.

What the checks turned up

Our generator’s XML validated, and we extracted its attachment byte for byte. The PDF still differs from the spec in two ways, as a comparison with an official sample showed:

	Official sample	`pdfx` + `embedfile`
`pdfaid` conformance	A	B
`Tagged`	yes	no
`/EF` keys	`F`, `UF`	`F`

A tolerant reader won’t care, and pdfdetach didn’t. A strict one could. PDF/A-3a needs a fully tagged document, which this pdflatex setup doesn’t produce.

Three other traps:

Optional validation can skip silently. Our --xsd flag printed “validation skipped” and carried on, because lxml was missing. In CI, make a skipped check fail.
TaxPointDate is not the issue date. A summary document can go out up to 15 days after the end of the month, so the two differ by design.
Signing happens before embedding. The ISDOC signature is an enveloped XMLDSig inside the XML. The transform removes only the Signature element, so reformatting the file afterward breaks the digest. A signature on the PDF is a separate step and should be an incremental update, which leaves the embedded bytes alone.

What the law requires and the standard lacks

The summary tax document (souhrnný daňový doklad, § 31b of Act No. 235/2004 Sb.) covers several separate supplies to one customer within a calendar month. Under § 31b(4), it must state separately for each supply its scope, date, unit price, tax base, rate and tax, and for each received payment its date of receipt. Data common to all of them can appear once (§ 31b(3)), so a header date is enough only if every supply happened on the same day.

VAT Act	ISDOC 6.0.2	Gap
§ 31b(4)(a)(2): date of each supply	`TaxPointDate` on `Invoice` only	No structured field per line
§ 31b(4)(b)(2): date of each payment	`TaxedDeposit` has no date	No structured field per payment
§ 31b: a separate kind of document	`DocumentType` 1–7, none for it, and the spec’s definition of “tax document” doesn’t list it	Receiver can’t tell from the standard fields
§ 29(1)(h): date of supply on a tax document	`TaxPointDate` is optional in the XSD, and only the spec’s prose says to fill it	XSD accepts a document without it
§ 26(3): customer consents to electronic form	`ElectronicPossibilityAgreementReference` is mandatory, but empty validates	Presence is checked, consent is not
§ 34: origin and integrity	Optional XML signature	None: a recognized signature is one route among several

I tested the per-line and document-type rows by editing a valid file: TaxPointDate on a line, TaxPointDate on an item, and DocumentType 8 each failed the XSD. The payment row comes from reading TaxedDepositType. This is my reading of the statute and the schema, not a ruling or tax advice. Nor does it make an ISDOC.PDF unlawful. If the visible document prints every date, the data is on the document. The gap is that the machine-readable half can’t carry it in a form a recipient’s software will understand.

The spec says its rules should, where possible, be independent of legislation. That is a fair position for validation rules. It doesn’t help when the structure has nowhere to put a value the law requires.

The implementations have the same shape

Header-only dates aren’t unique to ISDOC. In Odoo 18, account.move.line.date is a read-only related field that mirrors the journal entry’s date, and the Czech localization adds a single taxable_supply_date to the entry, defaulting to today. Modeling this correctly took a dedicated per-line field and a rule that derives the header from it. The upstream issue also notes that many commercial accounting systems lack per-line dates.

Our generator has its own limits: one VAT rate per invoice, CZK only, no deposits, and no reverse charge.

EN 16931, the European e-invoicing norm, offers a partial precedent. It has an invoice line period (BG-26) with start and end dates. A line period is not a tax date, but it shows that per-line dates fit in an invoice model.

What to do today

Print each supply’s date in the visible PDF. That is the part a person, or a tax auditor, reads.
Set TaxPointDate to the last supply date. Section 31b doesn’t define a document-level date, but the 2024 VAT control statement instructions ask for “the last (calendar) date” of the supplies on a summary document, so this keeps the two consistent.
Put each date on its own line in a fixed format. Both of these validate:

<Note>Datum plnění: 2026-07-14</Note>
<Item><Description>Work A</Description></Item>
<Extensions>
 <x:TaxPointDate xmlns:x="urn:example:line-tax-point">2026-07-14</x:TaxPointDate>
</Extensions>

Agree on the format with any recipient who needs to import the dates. A standard importer won’t read your note or your namespace.
If a recipient needs structured dates and can’t parse those, issue one ordinary invoice per date. It costs you the summary document, but the data is exact.

The proper fix looks small: in a new schema version, an optional TaxPointDate on InvoiceLine and a date on payments, with the header kept as it is. Issue #7 reports the missing per-line date and is open.

Takeaway

XSD-valid doesn’t mean legally complete, and legally complete doesn’t mean importable. Before you adopt an exchange format for regulated data, list what the law requires per record and find the element that holds each field. Any requirement that ends up in a free-text note is one your recipient’s software can’t see.

A dedup lookup that fails reads as "not found": urllib.parse.quote() is not query escaping

Thu, 23 Jul 2026 01:05:00 +0200

An OCR’d invoice number with an apostrophe in it, say O'BRIEN-1, went into a duplicate check like this one (trimmed):

flt = urllib.parse.quote(f"firma='{firma}' and cisDosle='{cis_dosle}'", safe="")
try:
 r = self.get(f"faktura-prijata/({flt}).json?detail=custom:id,kod&limit=1")
 entries = r.get("winstrom", {}).get("faktura-prijata", [])
 return entries[0] if entries else None
except FlexiBeeError:
 return None

The server rejected the filter with HTTP 400 (“Špatný formát WQL”, bad WQL format). The except turned that into None, the caller read None as “no such invoice yet”, and pushed it again: a second copy of the invoice in the books, and no error anywhere.

If your client puts a filter expression into a URL and wraps it in quote(), the apostrophe problem is probably yours too.

Setup

The API is the REST interface of ABRA Flexi, a Czech ERP, called from Python’s standard library. Its filters are a small query language (WQL) carried in the URL path: evidence/(field='value').json. The ?filter= query-string form is silently ignored, so the path it is used instead.

The values come from OCR and a language model. The invoice number was normalized for dashes and whitespace only, so an apostrophe made it into the filter intact.

What went wrong

quote() did its job. It turned the expression into something safe to put in a URL:

>>> urllib.parse.quote("cisDosle='O'BRIEN-1'", safe="")
'cisDosle%3D%27O%27BRIEN-1%27'

The server URL-decodes the path before it parses the filter, so the parser gets back cisDosle='O'BRIEN-1'. The literal ends after O; the rest is garbage, and the query fails.

Percent-encoding protects the URL, and the server undoes it before the filter language ever sees the value.

Every client in the codebase built its filters this way, including the MCP server. Two call sites even disagreed on safe= ("'=" versus ""), which made no difference, since the server decodes either form back to the same bytes.

The failure stayed hidden because of the except. A lookup that can’t answer reported “not found”, and for a duplicate check, “not found” means “write it”.

Fix

First, find out what the grammar’s escape actually is. No client library for this API escapes it (the PHP one interpolates values raw), so there was nothing to copy. A wrong guess would cause the very duplicate the lookup is there to prevent, so it was settled by asking the server, read-only, in a sandbox company, with values that can’t match any record:

cisDosle='ZZZTEST' balanced -> 200
cisDosle='ZZZ'TEST' raw (the bug) -> 400 Špatný formát WQL … pozice 15
cisDosle='ZZZ\'TEST' backslash -> 200
cisDosle='ZZZ''TEST' SQL-style doubling -> 400
cisDosle='A\\B' doubled backslash -> 400
cisDosle='A\B' lone backslash -> 400

So this grammar escapes an apostrophe with a backslash, the way C does, not by doubling it, the way SQL and OData do. A backslash escapes nothing else, which means a literal backslash can’t be written at all. That’s acceptable for invoice numbers, codes, company IDs, and dates, which never carry one.

Second, build every filter in one module, so the escape exists exactly once:

import urllib.parse

def escape_literal(value: object) -> str:
 return str(value).replace("'", "\\'")

def eq(field: str, value: object) -> str:
 return f"{field}='{escape_literal(value)}'"

def and_(*clauses: str) -> str:
 return " and ".join(c for c in clauses if c)

def evidence_path(evidence: str, expr: str) -> str:
 return f"{evidence}/({urllib.parse.quote(expr, safe='')}).json"

path = evidence_path("faktura-prijata", and_(eq("firma", firma), eq("cisDosle", cis_dosle)))

escape_literal handles the query language and quote handles the URL, each in one place. Against the sandbox, the real client with this filter now gets a 200 for an invoice number with an apostrophe.

Before the fix, the gap was parked as an expectedFailure test. It checks that the quotes in the generated filter balance once either escape form is removed, so it would pass whichever escape the probes picked, and it flipped to an unexpected pass the moment the fix landed.

Why

Any value that goes into a query language needs that language’s escape, and then the transport’s. Percent-encoding is the transport’s. OData $filter and most search APIs set the same trap with their own escape rules, so probe the server instead of guessing.

The other half is the except. The finder in this codebase still maps an error to None, and that remains on its list of known issues. If you write a duplicate check, give it three outcomes: found, not found, and couldn’t tell, and make the caller stop on the third.

An earlier bug in the same codebase took the other route to the same end: a bank-movement duplicate check used LIKE, which this API doesn’t support. The server didn’t reject it; it returned an empty list, so every movement was pushed a second time. The fix was the API’s BEGINS operator, plus a one-off script to delete the copies.

Takeaway

Escape a value for the query language first and for the URL second; quote() only does the second. A duplicate check that turns errors into “not found” creates a duplicate every time it breaks.

Search your code for quote(f" and check each hit for a quoted literal built from user or OCR input.

HTTP 500 that still commits: never retry an ambiguous write

Wed, 22 Jul 2026 22:29:00 +0200

We ran the client’s real retry loop against a fake server and counted the writes on the wire, before and after the fix:

before: push PUT 500 -> 200 writes_on_wire=2 -> reported ok
after: push PUT 500 -> 200 writes_on_wire=1 -> FlexiBeeError(maybe_committed=True)

The fake server played what the real API does now and then: it answers a PUT with 500 Internal Server Error after it has already saved the invoice. The client retried; the second PUT came back 200 OK, and the batch reported success. On a real API, that invoice is on the books twice. If your API can return a 5xx after committing, a retry that succeeds is the worst outcome you can get: you end up with a duplicate and nothing in the logs.

Setup

The API was the REST interface of ABRA Flexi, an ERP, called from Python through urllib and http.client. It returns 500 now and then after the write has gone through. That much was known; one client already looked the record up by document number after a failure.

Five hand-written clients for that API were merged into one transport layer with a shared RetryPolicy. An AI coding agent did the merge, two commits after writing an ADR that warned about duplicate writes.

What went wrong

The policy didn’t know the method

RetryPolicy decided from the response alone, so it retried a 500 on a PUT the same way it retried one on a GET. The test for that case fed in three 500s and checked only the flag on the final error. It pinned the number of retries and never asked whether a write should be retried in the first place.

The safety nets sat outside the loop

The first fix made the method a required argument and turned off retries of ambiguous writes by default. Two callers opted back in, because each had a safety net. In outline:

# MCP server: an idempotency key per tool call
def write_tool(key, payload):
 if idem_seen(key): # checked once, before the first attempt
 return cached_result(key)
 result = request("PUT", url, payload) # retries happen in here
 idem_record(key, result) # recorded only after success
 return result

# webhook worker: look the record up when the write fails
def upsert(doc):
 try:
 return request("PUT", url, doc) # retries happen in here
 except ApiError:
 if find_by_number(doc.number):
 return "committed-via-verify"
 raise

A review five days later, by a different model, caught it: “_idem_seen runs before this function, and _idem_record runs only after success, so neither prevents this retry from duplicating a write.” Likewise, the GET check runs only when request() raises, and a 500 followed by a successful retry never raises. Both checks were correct; the retry loop inside request() simply never consulted them.

Fix

Writes are never retried on an ambiguous outcome, and no caller can opt back in. The error goes up to the caller and says the write may have committed. The policy, trimmed to the part that matters:

WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}

class RetryPolicy:
 def __init__(self, method: str): # required: a default would let the next client forget it
 self._is_write = method.upper() in WRITE_METHODS

 def next_retry(self, status: int, body: str):
 if is_rate_limited(status, body): # the server refused: nothing committed
 return self._backoff("rate-limit")
 if 500 <= status < 600 and not self._is_write:
 return self._backoff("transient-5xx")
 return None # a 5xx on a write: raise, maybe_committed=True

 def next_retry_for_network_error(self):
 # no response at all: the server may have processed it before the connection died
 return None if self._is_write else self._backoff("network")

The new test counts requests, because a retry that succeeds looks exactly like a first attempt that did:

def test_500_on_a_write_is_never_retried(self):
 fake = FakeConn([FakeResponse(500, b"boom"), FakeResponse(200, b"{}")])
 with patch.object(client, "_get_conn", return_value=fake):
 with self.assertRaises(ApiError) as cm:
 client.put("invoice.json", {"invoice": [{}]})
 self.assertEqual(len(fake.requests_made), 1)
 self.assertTrue(cm.exception.maybe_committed)

Reads still retry, and they make up most of the traffic in a batch. A write that comes back maybe_committed is checked by whoever started it, before anything resends it.

Why

The policy asked, “may I retry a 5xx?” For a write, the question is “may I resend something whose outcome is unknown?” Two outcomes leave a write ambiguous: a 5xx, and no response at all. A 429, or a 400 rejecting a stale user role, is the server refusing the write; nothing committed, so a retry can’t duplicate anything.

The HTTP method doesn’t settle it either. RFC 9110 calls PUT idempotent, and urllib3’s default allowed_methods includes PUT, so a status_forcelist=[500, 502, 503] resends it. This API’s PUT to a collection creates a record each time.

If you do want writes retried automatically, the check has to run between the failure and the resend: before every attempt, look the record up or send a key the server itself deduplicates on. A check before the first call or after the last one can’t see the retries in between.

Takeaway

A 500 on a write means you don’t know what happened, so pass that up the stack instead of resending. Test it by counting requests on the wire, since the final result looks the same either way.

Find the retry loop in your own HTTP client and check whether it knows the method it is retrying.

Standards vs Spec review: a second agent found duplicate writes past 1,572 green tests

Wed, 22 Jul 2026 21:58:00 +0200

This is what the review agent reported, just before it listed its findings:

The test evidence is clean: Ruff passes, Python reports 1,572 passed / 56 skipped /
1 expected failure, and the GUI reports 418 passed / 4 skipped. The review nevertheless
found safety gaps the tests do not exercise — most importantly, two write clients retry
"500-after-commit" before their advertised idempotency/verification guard can run.

A coding agent had built the branch: 53 commits, 147 files. A different model reviewed it and found two paths that could put the same invoice in the ERP twice and report plain success. No test covered them, and no linter could. They came from a reviewer with one job: check the code against what the branch said it does.

The setup

The repository vendors the two-axis code-review skill from mattpocock/skills . It pins a fixed point and diffs against the merge base, git diff master...HEAD, then forks two subagents in parallel:

Standards: does the diff follow the repo’s documented rules (glossary, ADRs, a baseline of Fowler code smells)?
Spec: does the diff do what the originating issue or PRD asked for?

The parent agent runs lint and tests meanwhile, “as supporting evidence, not as a substitute for the diff review”. It then reports the two axes under separate headings and doesn’t rerank them, so a pile of naming nits can’t bury a P1.

The Spec brief is short:

Report: (a) requirements the spec asked for that are missing or partial;
(b) behavior in the diff that wasn't asked for (scope creep);
(c) requirements that look implemented but where the implementation looks wrong.
Quote the spec line for each finding, under 400 words.

What each axis found

There was no PRD for the branch, so the Spec reviewer used the commit messages and the ADRs. One commit claimed that the MCP server’s idempotency check made it safe to retry a write that got a 5xx. The reviewer traced the call: the check ran before _request(), the key was recorded after success, and the retry happened inside _request(), in between. A webhook worker had the same shape, with a verify-by-GET that ran only if the request finally raised. A retry that succeeds never raises. (The full HTTP 500 story is here .)

The Standards reviewer found something else. Two functions called urlopen() directly, past the shared transport that an ADR says owns all retries. It also found an import that went through a module the ADRs forbid as a re-export facade, and an unfinished vocabulary migration.

None of the Standards findings overlapped with the Spec ones. All six were fixed the same evening, in one commit.

Why the tests stayed green

The tests came from the same author as the claim. No test let a 500 commit the write, let the retry succeed, and counted the writes, because the commit said the idempotency check covered that case. A suite tests what its author thought could break.

The Spec reviewer starts from the claim instead. A commit message says the code is safe, so it goes looking for the line that makes it safe, and here that line ran on the wrong side of the retry.

Two more runs, same pattern

In the next review, a few hours later on a six-commit range, a guard was undone downstream. Every write had to carry exactly one confidence label, and create_* checked that. But when label setup had failed, a fallback further down stripped all labels from the payload. The reviewer reproduced a labeled invoice leaving without its label. It also found a write that went through a raw put() and skipped the scope check that limits which record types may be written at all.

The third review had nothing to go on but commit messages. The branch added a gate: if an invoice is addressed to another company, warn and refuse to post it. The reviewer said it was inferring the spec from the commits, so I pasted the original request and the Spec subagent re-checked against that. The request held a second requirement the commit summary had left out, which turned out to be implemented. The review also found that a replay of staged payloads called the push function directly and skipped the new gate. And while company registration numbers were checksum-validated, VAT IDs relied on regex alone, so one misread digit could block a legitimate invoice. Both failed the first test, then got fixed.

How to use it

Run Standards and Spec as separate subagents, so neither sees the other’s findings, and report them under separate headings.
Give the Spec reviewer the authoritative spec. If you only have commit messages, say so, and swap in the real request the moment you have it.
Treat claims in commit messages and ADRs (“X makes this safe”, “Y owns all retries”) as requirements. They are the cheapest spec you have, and the author’s tests already agree with them.
If you can, review with a different model than the one that wrote the code. It is less likely to share the writer’s blind spots.

Takeaway

A green suite tells you the code does what its author expected. A Spec review by a second agent tells you whether that matches what the author promised. The duplicate writes were in the gap between the two.

Take the last branch your agent wrote, and have a second agent check each “this makes X safe” in its commit messages against the code.

Date the bug before you park it: fence, drift or silence

Fri, 17 Jul 2026 01:13:00 +0200

My coding agent was splitting a 738-line Python function into steps and found a real bug along the way:

result.issues = validation.issues

That line rebinds the list instead of extending it, so it dropped every issue recorded before validation, including the note that repair mode had just deleted a record in the accounting system. The push still succeeded, and the operator’s report said nothing about the delete.

The agent didn’t fix it. It wrote a known-issues entry and added two @unittest.expectedFailure tests, because the handoff for the refactor said: “characterize, then collapse, then fix separately”. It was the wrong call, and two git log -S runs showed it.

The setup

Claude Code working in a Python repository of an accounting pipeline, tested with unittest;
A long-lived function where several steps append to result.issues, the operator’s per-file report;
A refactor rule that a split must not change behavior.

What parking costs

An expectedFailure marker looks like a decision. The next reader sees a documented, tested, known gap and assumes someone weighed it and chose to keep it. If nobody did, the marker turns an accident into policy and keeps a live bug in the tree behind it.

So before you park a bug, find out whether anyone ever decided it. I asked the agent to run the fence check (Chesterton’s fence: don’t remove a fence until you know why it was put up).

Date the bug

A bug is a relationship between two things: this line and the code it gets wrong — date both.

git log --oneline -S'result.issues = validation.issues' -- <path>
git log --oneline -S'result.issues.append' -- <path>

The result went into the fix’s commit message:

2026-04-16 rebind written. result.issues had NO prior writers;
 assigning validation's list into it lost nothing.
2026-04-19 + vendor mismatch trail above it +3d
2026-04-23 + REPAIR delete note above it +7d
2026-04-24 + registry retry verdict above it +8d
2026-05-27 + self-billed reroute above it +41d

The rebind was correct when it was written. Over the next 41 days, four commits added writers above it, and none of them touched it. Nobody decided to discard those issues; the code predates the thing it breaks.

That gives one of three verdicts:

Fence: someone decided this, in a commit message, a spec’s non-goals or an ADR. Leave it.
Drift: the code predates what it is wrong about, or its author never touched the site. Fix it now, and put the dates in the commit message so a reviewer can see you didn’t remove a fence.
Silence: the history says nothing either way. Park it, and say plainly that nobody decided it.

This one was drift, so it got fixed the same night. It had stayed hidden because the only test that reached this code patched the whole function body out. The split made the body reachable, and the first end-to-end repair test found the bug.

The obvious fix was wrong

A drift verdict says nobody meant the bug; you still have to find out what depends on it. The one-word fix:

result.issues.extend(validation.issues)

would have broken an alias. A later step appends registry failures to result.issues, and those reach the invoice’s note only because the rebind made result.issues and validation.issues the same list, which the note builder reads:

result.issues = validation.issues # one list, two names
apply_registry_failure(result) # appends to result.issues ...
note = build_note(validation) # ... and the note sees it through the alias

With .extend(), every invoice that failed the business-registry check would have lost that warning from its note: one bug traded for a quieter one. The fix kept the rebind and put the earlier issues back after the later writers ran:

earlier_issues = list(result.issues)
result.issues = validation.issues
apply_registry_failure(result)
# ... the other writers ...
result.issues = earlier_issues + result.issues

The tests were written to catch both mistakes: reverting the fix fails the four tests that check the earlier issues survive, and the naive .extend() fails the one that checks the registry warning still reaches the note. The known-issues entry was deleted, and the expectedFailure markers became ordinary tests.

What silence looks like

That night, four other bugs were dated as drift and fixed. One wasn’t. Debug files were named from the input’s stem, so invoice.docx and invoice.pdf in one batch wrote the same file, and the later one won. That naming shipped on a pipeline that already read several formats, so the code didn’t predate the collision. No commit message or spec mentioned it either: stem is just the obvious way to name a sidecar file. That one stayed a known-issues entry, with the dating written into it.

The skill

The fence check is a Claude Code skill I had written down earlier that evening. The agent had it and didn’t run it until asked, so its memory now says to run it the moment it finds a bug, before choosing between a fix and a known-issues entry. Here it is verbatim. The examples and the docs/superpowers/ paths are from my repository; swap in your own spec directory.

---
name: checking-the-fence
description: Date a bug before fixing it — Chesterton's fence. Use when you have found code that looks wrong and are about to change it, when you are about to delete something that looks dead, or when another skill needs the fence check.
---

# Checking the Fence

A bug you found is a **fence** across a field. Before you take it down, find who put it there. Here that is written down twice — in `git log` and in `docs/superpowers/` — and is usually decisive in a minute.

Two verdicts. They are the whole skill:

- **Fence** — someone decided this. Leave it, and record an ADR if the next reader will reach for the same fix you just did.
- **Drift** — nobody decided it. The code predates the thing it is wrong about, or an author was never told which files the job touched. Fix it.

Fences here are real, not hypothetical: `docs/adr/0002` exists only to stop a reader from deleting emoji handling that has no inbound traffic and looks dead.

## Steps

Stop at the first step that returns a verdict.

### 1. Date the code

```
git log --oneline -S'<symbol>' --all -- <path>
git log -1 --format=%B <sha>
```

Commit messages here carry the rule the author meant to encode, and the rule is often the thing the bug breaks. `_PARSER_PRIORITY`'s own commit says *"process deterministic parsers before PDFs/images"* — while the table shipped the cheapest format last.

**Done when** you can name the commit and quote the rule.

### 2. Date what it is wrong about

A bug is a relationship: a table missing a format, a copy missing an entry, a switch missing a branch. Date the other side too.

Code cannot have decided about something that did not exist yet. `_PARSER_PRIORITY` (2026-04-27) predates isdoc (2026-06-01) by five weeks.

**Done when** you have both dates. Predates → **drift**. Stop.

### 3. Read the spec

```
grep -rli '<feature>' docs/superpowers/specs docs/superpowers/plans
```

Read the **Non-goals** first — that is where a deliberate gap is named. A spec that scopes the gap out is a **fence**. A spec whose own words the bug contradicts is **drift**: the ISDOC spec calls it *"a deterministic fast path before Claude"*.

**Done when** you have quoted the line that scopes it out, or established that no spec names it.

### 4. Read the blast pattern

```
git show --stat <sha>
```

Which files that commit touched, against which it needed to. An author who updated three of five consumers was not deciding about the other two — nothing told them. The isdoc and docx commits landed a day apart and touched different subsets.

**Done when** you can say whether the author addressed the site at all. Never addressed → **drift**.

### 5. Was it born wrong?

For a forked copy, compare it against its source **on the day it was copied**, not today.

```
git show <sha>:<source-path>
```

`coverage_report.py` shipped without `.json` on 2026-04-24, and the map it was copied from carried `.json` that same day. Born forked, then rotted twice more.

**Done when** you know whether the gap is drift-over-time or a bad copy. Either is **drift**.

## Applying the verdict

**Drift** — fix it, and say in the commit message what you found: the dates, and that nothing decided it. That is what stops a reviewer from wondering whether you removed a fence.

**Fence** — leave it. Offer an ADR when a future reader would reach for the same fix; that reader is usually the next architecture review, or you.

**Neither** — the history is silent and no spec mentions it. Say so plainly and let the user decide; silence is not consent, and it is not a fence either.

Save it as .claude/skills/checking-the-fence/SKILL.md in your repository, and tell the agent in CLAUDE.md or AGENTS.md to run it before it records a bug as a known issue.

Takeaway

A parked bug looks like a decision, so check whether anyone made one. If the code predates what it breaks, fix it now, and check the mechanism first: knowing nobody meant the bug doesn’t tell you what depends on it.

Next time your agent adds an expectedFailure or a known-issues entry, ask it for the two dates first.

Every test mocked the helper, and except Exception swallowed its TypeError on every PDF

Thu, 16 Jul 2026 23:42:00 +0200

For seven weeks, a document classifier never read a single PDF. This is the helper that was supposed to give the model some text (trimmed):

def _extract_excerpt(file_path: Path) -> str:
 """First-page text for PDFs, first ~8 kB for everything else."""
 if file_path.suffix.lower() == ".pdf":
 try:
 pages = extract_pdf(file_path)
 return (pages[0] if pages else "")[:EXCERPT_BYTES]
 except Exception:
 return ""
 ...

extract_pdf doesn’t return a list of pages. It returns a dataclass, so every call raised this:

TypeError: 'PDFExtraction' object is not subscriptable

The except caught it and returned "". The prompt then held the filename and an empty excerpt, and the model answered anyway: a document type and a confidence score, for every PDF it was asked about.

If a helper of yours has a catch-all except and your tests patch that helper, check whether anything ever runs it for real.

Setup

A web upload went through a small Python script that sorted the file into one of the document kinds declared for an accounting period (received invoices, bank statements, and so on). With two or more possible kinds, it built a prompt from the filename plus the start of the document and asked a language model, which answered with JSON like {"kind": "...", "confidence": 0.9}. The PDF text came from an in-house extract_pdf() built on pdfplumber. Tests used the standard library’s unittest and unittest.mock.

What went wrong

The helper was wrong from the day it was written. extract_pdf had returned a PDFExtraction dataclass with a pages list since its first commit, six weeks earlier, and its docstring said so. The if pages check didn’t catch the mistake: a dataclass without __len__ is always truthy, so the check passed and pages[0] raised.

The except was there so that a corrupt or encrypted upload wouldn’t break classification. It caught the TypeError just as readily and returned "", which is also the documented answer for an image upload with no text to read.

And every test patched the helper out. The tests that reached the model looked like this:

with patch("scripts.classify_upload.invoke_ai") as m, \
 patch("scripts.classify_upload._extract_excerpt", return_value="Invoice 2026-001 ..."):
 m.return_value = json.dumps({"kind": "received_invoices", "confidence": 0.92})
 result = classify_file(Path("/tmp/x.pdf"), ["issued_invoices", "received_invoices"], "claude")

They covered the prompt plumbing and the JSON parsing, and none of them ever called _extract_excerpt. The implementation plan that specified the helper had the same broken line, and it specified these tests, mocks included. The code, the plan, and the tests agreed with each other, and none of them had been checked against extract_pdf.

The model hid the rest. A filename like invoice-0423.pdf can be enough for a right guess, and a wrong guess came back with a confidence score like any other, so nothing in the output said the excerpt was empty.

Fix

The fix reads the first page, as the docstring promised:

extraction = extract_pdf(file_path)
first_page = extraction.pages[0].text if extraction.pages else ""
return first_page[:EXCERPT_BYTES]

The new test class calls the real function on a real PDF:

class RealPdfExcerptTests(unittest.TestCase):
 def setUp(self):
 self.tmp = tempfile.TemporaryDirectory()
 self.root = Path(self.tmp.name)

 def tearDown(self):
 self.tmp.cleanup()

 def test_a_pdf_yields_its_first_page_text(self):
 path = self.root / "invoice.pdf"
 _write_pdf(path, "INVOICE 2026-001 Supplier ACME")
 self.assertIn("INVOICE 2026-001", _extract_excerpt(path))

 def test_a_corrupt_pdf_yields_no_signal_rather_than_raising(self):
 path = self.root / "broken.pdf"
 path.write_bytes(b"%PDF-1.4\nnot actually a pdf at all")
 self.assertEqual(_extract_excerpt(path), "")

_write_pdf is an existing test helper that writes a one-page PDF with a text layer. Before committing, I ran the first test against the original code to confirm it fails there. The except stayed, because corrupt uploads are real. The second test now says what it’s for.

A day later, the same helper had a second route to the same result. It kept its own list of text suffixes, so Word, Excel, and XML e-invoice uploads fell through to "" and were classified by filename too. Two of those formats joined the pipeline a week after the list was written, and nobody updated it. The helper now asks the pipeline’s format registry instead.

Why

A mock replaces the code it patches with your assumptions about that code. When every test mocks a function, its tests check only those assumptions, and the function itself is never run. A catch-all except inside the function then removes the last signal you’d get in production: no traceback, no failed request, just a default value that looks legitimate.

A language model downstream makes this worse. A parser fed an empty string usually fails loudly, while a model returns its best guess in the usual format, confidence field included, whatever the input.

Takeaway

Mock the expensive or external dependency (here, the model) and let the function under test run on a real input. If it has a catch-all except, give it a test with an input that should succeed, so a bug can’t hide behind the except.

Grep your tests for patch( targets that point at your own module’s private helpers, and for each one, find a test that calls the helper unmocked.

Sandbox-blocked tests called "environmental", then committed

Mon, 13 Jul 2026 15:38:00 +0200

My coding agent finished a feature, ran the full test suite, and reported:

Full suite: 135 passed, 97 skipped; 22 DB suites failed due to no container runtime, plus sandboxed stub bind. Commit only handoff implementation files.

Then it committed. I ran pnpm test myself and got three real failures. The production build it had also skipped was broken too, and nobody found out for two months.

If your agent’s test run shows Error: Could not find a working container runtime strategy or Error: listen EPERM: operation not permitted 0.0.0.0:8080, those tests never ran, and a summary that counts them as “blocked” tells you nothing about the code.

The setup

Codex CLI on macOS, with its default sandbox
A pnpm monorepo: a Next.js 15.5.19 app plus a few packages
Vitest 2.1.9, with Testcontainers 10.28.0 starting Postgres for the database tests and a local stub of an external service listening on port 8080

The agent split the feature into five slices and gave each to a parallel subagent. Each subagent ran its own focused tests, and the orchestrator ran the full suite and the build at the end.

What went wrong

The sandbox blocks more than writes outside the project. Here it blocked three things the checks needed:

Testcontainers couldn’t reach Docker: Could not find a working container runtime strategy.
The service stub couldn’t bind its port: listen EPERM: operation not permitted 0.0.0.0:8080.
next build couldn’t write its compiler cache: EPERM writing ~/Library/Caches/next-swc.

Every database suite failed in beforeAll, and Vitest reported the tests inside them as skipped. The agent’s final report was accurate:

- Typecheck pass
- Focused tests pass
- Full: 135 passed; DB suites blocked by no Docker
- Build blocked by Next SWC cache permissions

Every line of that is true. But the suite has 232 tests, and 135 + 97 = 232: over 40 percent of it never ran, including every test that touched the database, where this feature did most of its work.

On the host, the same suite printed:

Test Files 2 failed | 63 passed (65)
 Tests 3 failed | 229 passed (232)

One new test forgot to start the service stub, so its client found no endpoint configured. Two older tests broke because the feature added an ownership check and their fixtures sent requests without a logged-in user (Error: authentication required). The agent fixed all three in two minutes once it saw them.

The build was worse. The commit re-exported a server action from another "use server" file:

"use server";

export { finalizeArtifact } from "../../reports/actions.ts";

tsc accepts this, and so do the tests. Only next build rejects it:

Failed to compile.

./app/reports/[id]/actions.ts
Error: x Only async functions are allowed to be exported in a "use server" file.

Two months later, a different session ran a production build, hit this error, and deleted the line.

The same sandbox, the same day

Half an hour before the bad commit, another session in the same repository hit the same two errors. At first it called them environmental too. Then it decided:

Full suite blocked by sandbox: Testcontainers cannot see Docker; stub bind gets `EPERM :8080`. Since Docker is reportedly running, retry full suite with host access.

Two minutes later it reported “Host rerun green: 60 files, 222 tests”, and only then did it commit. The agent that fixed my three failures did the same: it reran the focused tests with host approval before committing the fix. Both sessions knew how to leave the sandbox; nothing required them to before a commit.

Fix

Treat “blocked” as “not run”, and make “not run” stop the commit.

In Codex, a command runs outside the sandbox when the agent requests sandbox_permissions: "require_escalated" and you approve it. Put the rule in the instructions file the agent reads every session (AGENTS.md, or CLAUDE.md for Claude Code):

A check that the sandbox blocked has not passed. Before committing, rerun every blocked test suite and build with host access (`sandbox_permissions: "require_escalated"`).
If that isn't possible, don't commit: report what didn't run.

Then read the agent’s numbers yourself. If passed plus skipped equals the total, and you never skip tests on purpose, every skipped test is one that didn’t run. If you have CI, have it run the full suite and the build before a merge.

Why it happens

The agent’s memory for this project already had a rule: “separate passed static or focused checks from runtime suites blocked by the environment, and reserve the final green claim for integrated verification.” The agent followed that rule to the letter: it never claimed green and listed exactly what had been blocked. Then it committed, because nothing in its instructions tied the commit to the green claim.

“Blocked by the environment” reads like an infrastructure problem that someone else will handle, so it ends up as a footnote in the report while the commit goes ahead. The rule made the report honest. It didn’t make anyone run the tests.

Takeaway

When a sandbox blocks a test or a build, nothing was verified. An agent that reports this honestly will still commit, unless its instructions say that a blocked check stops the commit.

Look at your agent’s last commit: if its summary says a suite or a build was “blocked”, rerun them on the host today.

ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY: your coding agent's pnpm 11 is not your pnpm 10

Mon, 06 Jul 2026 15:04:00 +0200

A coding agent ran pnpm test in my repo and got this:

Scope: all 6 workspace projects
[ERR_PNPM_ABORTED_REMOVE_MODULES_DIR_NO_TTY] Aborted removal of modules directory due to no TTY

If you are running pnpm in CI, set the CI environment variable to "true", or set "confirmModulesPurge" to "false".

The stack trace ended in runDepsStatusCheck (.../node_modules/.pnpm/pnpm@11.7.0/...). My package.json says "packageManager": "pnpm@10.33.0", and pnpm -v in my terminal prints 10.33.0.

If you see this error only inside an agent, the agent is running its own pnpm, and following the hint makes things worse.

The setup

macOS, pnpm 10.33.0 through Corepack, a pnpm monorepo with a Next.js 15 app
The Codex desktop app, which ships its own Node 24 and pnpm under ~/.cache/codex-runtimes/codex-primary-runtime/dependencies/: pnpm 11.7.0 in July, 11.25.0 in October
The agent runs pnpm test without a TTY

What went wrong

The agent followed the hint and reran with CI=true pnpm test. pnpm stopped asking and did what it had wanted to do from the start:

Recreating /Users/me/project/node_modules
Progress: resolved 378, reused 0, downloaded 137, added 12
...
[WARN] GET https://registry.npmjs.org/@next/swc-darwin-arm64/-/swc-darwin-arm64-15.5.19.tgz error (ERR_PNPM_ENOSPC). Will retry in 10 seconds. 2 retries left.
[WARN] GET https://registry.npmjs.org/next/-/next-15.5.19.tgz error (ERR_PNPM_ENOSPC). Will retry in 10 seconds. 2 retries left.
...
Progress: resolved 378, reused 0, downloaded 377, added 377, done

[ERR_PNPM_IGNORED_BUILDS] Ignored build scripts: cpu-features@0.0.10, esbuild@0.21.5, esbuild@0.28.1, protobufjs@7.6.4, sharp@0.34.5, ssh2@1.17.0

It deleted node_modules and downloaded every package again (reused 0) into a second store, and the disk ran out on the way. It left an untracked .pnpm-store/ in the repo and wrote placeholders into the tracked pnpm-workspace.yaml:

allowBuilds:
 cpu-features: set this to true or false
 esbuild: set this to true or false
 protobufjs: set this to true or false
 sharp: set this to true or false
 ssh2: set this to true or false

No tests ran. After that, even pnpm exec tsc --noEmit failed with the same NO_TTY error, so the agent switched to ./node_modules/.bin/tsc and ./node_modules/.bin/vitest, reverted the YAML, deleted .pnpm-store/ and committed with only the focused tests green.

Ten weeks later, next build failed:

 ⚠ Attempted to load @next/swc-darwin-arm64, but it was not installed
 ⨯ Failed to load SWC binary for darwin/arm64, see more info here: https://nextjs.org/docs/messages/failed-loading-swc

The July install had ended one package short (378 resolved, 377 added), and the SWC binary was one of the two downloads that hit ENOSPC. A later agent session “fixed” the build by installing the binary into /tmp and symlinking it into node_modules, a fix that lasts until macOS cleans /tmp.

Fix

First, ask the agent’s shell which pnpm it gets:

command -v pnpm
pnpm -v

Then make the agent use your version. Corepack reads packageManager, whatever is first on PATH:

corepack pnpm test # runs pnpm 10.33.0

Write that into AGENTS.md (or your agent’s equivalent), along with one rule: if pnpm wants to remove node_modules, stop and report.

To make the bundled pnpm harmless as well, turn off the automatic install in pnpm-workspace.yaml:

verifyDepsBeforeRun: false

That is the pnpm 10 default, so nothing changes for you. I checked it with the bundled pnpm 11.25.0: with the setting, it runs the script; without it, it installs first.

If you already followed the hint, clean up with your own pnpm:

git checkout -- pnpm-workspace.yaml
rm -rf .pnpm-store
corepack pnpm install --frozen-lockfile

Run that in a terminal. If pnpm asks to remove node_modules, say yes; this time it rebuilds against your own store.

Why

Three pnpm 11 changes line up:

verify-deps-before-run defaults to install (pnpm 10: false). pnpm test and pnpm exec check node_modules first and install if anything is off.
pnpm 11 uses store v11; pnpm 10 uses v10. node_modules/.modules.yaml records the store it was linked from, so to pnpm 11 a working pnpm 10 install looks incompatible and has to be purged. confirmModulesPurge defaults to true; a purge without a TTY aborts, and CI=true drops the confirmation but not the purge.
Ignored build scripts are an error, and pnpm writes the allowBuilds placeholders for you to fill in.

On my Mac, the two stores now sit side by side in ~/Library/pnpm/store/: 851 MB in v10, 588 MB in v11.

Why didn’t packageManager catch it? pnpm 11 does honor it: run the bundled binary directly and pnpm -v prints 10.33.0, because it switches to the pinned version. The wrapper in the October runtime turns that off:

#!/usr/bin/env bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export pnpm_config_pm_on_fail=ignore
exec "${SCRIPT_DIR}/../../node/bin/node" "${SCRIPT_DIR}/../../node/node_modules/pnpm/bin/pnpm.mjs" "$@"

The environment variable beats anything in the repo. With pmOnFail: error in pnpm-workspace.yaml, or devEngines.packageManager with "onFail": "error", the wrapper still runs 11.25.0.

Takeaway

An agent runtime can bring its own toolchain and switch off the check that would have caught it. When an agent reports “tests blocked by the environment”, find out which binary it ran before you believe the report.

Run pnpm -v in your agent’s shell today and compare it with packageManager.

TS2724 "has no exported member" in a git worktree: a symlinked node_modules typechecks the main checkout's source

Mon, 22 Jun 2026 19:21:00 +0200

A coding agent cherry-picked a subagent’s commit into an integration worktree, ran pnpm typecheck and got:

apps/web/lib/uploads.ts(3,59): error TS2724: '"@app/persistence"' has no exported member named 'recordRawUpload'. Did you mean 'recordUpload'?

recordRawUpload was right there, exported from packages/persistence/src/index.ts in that worktree. If tsc can’t see an export that you can, and the worktree’s node_modules is a symlink to another checkout, tsc is reading the other checkout’s source.

The setup

A pnpm 10 monorepo on macOS. The workspace packages @app/* point exports straight at ./src/index.ts, so there is no build step between packages.
An orchestrating agent that creates a git worktree, a branch, and a subagent per issue, and cherry-picks finished work into an integration worktree.
A full disk.

What went wrong

Every worktree had its own install, and du counted up to 513 MB of node_modules in each. A dozen worktrees later, the disk was full:

Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk3s5 460Gi 420Gi 116Mi 100% 6.6M 1.2M 85% /System/Volumes/Data

To save space, the agent replaced the node_modules of each new worktree (the root, apps/web and every package) with a symlink to the main checkout’s copy:

ln -s ../project/node_modules ../project-integration/node_modules
ln -s ../../../project/apps/web/node_modules ../project-integration/apps/web/node_modules

The first typecheck after that passed. Later, a change that added an export to @app/persistence and used it in apps/web failed with TS2724. readlink shows why:

$ readlink apps/web/node_modules/@app/persistence
../../../../packages/persistence

pnpm links workspace packages with relative symlinks. A relative link resolves from the directory it really sits in, and that directory is now in the main checkout. So @app/persistence in the worktree was /Users/me/project/packages/persistence, the main checkout’s source, which didn’t have the new function yet.

The typecheck that passed also validated the main checkout.

Two subagents in their own worktrees hit the same error. One deleted the links, reinstalled, and its typecheck passed. The other wrote “Validation blocked by worktree node_modules links targeting older shared package. Source diff reviewed; commit now.” and committed without a passing typecheck.

Fix

Give each worktree its own install. pnpm keeps package files in a content-addressable store, so an offline install only links them into place:

rm node_modules apps/web/node_modules packages/*/node_modules # the symlinks
pnpm install --offline --frozen-lockfile

This took 3.6 seconds and reported resolved 378, reused 378, downloaded 0. If the store is missing a package, --offline fails; drop the flag.

Then check that every workspace package resolves inside the worktree. Run it with bash: zsh aborts on the first glob that matches nothing (no matches found), and the agent’s own fix hit exactly that.

#!/usr/bin/env bash
# Print workspace packages that resolve outside this checkout.
root=$(pwd -P)
for link in node_modules/@app/* */*/node_modules/@app/*; do
 [ -e "$link" ] || continue
 target=$(realpath "$link")
 case $target in "$root"/*) ;; *) echo "$link -> $target" ;; esac
done

No output means every workspace package comes from this worktree.

If you have to share node_modules, share only the third-party packages. The agent’s fix was an overlay: a real node_modules directory per package, with every entry symlinked to the main checkout except @app, whose entries point at the worktree’s own packages. Copying the main checkout’s @app directory with cp -RP gives the same result, because the copied relative links resolve from their new place.

Why

TypeScript (with the default preserveSymlinks: false) and Node both resolve a package to its real path. Third-party packages are the same files in every checkout that shares a lockfile, so following the link to the main checkout is harmless. Workspace packages are your source code, and that differs between checkouts by design. Once a branch changes its dependencies, the third-party half goes stale as well.

The disk savings were probably smaller than they looked. pnpm clones or hard-links files from its store, and du counts those blocks again in every tree, so its per-worktree numbers overstate what each install cost. The disk was also full for other reasons, and nobody measured how much the links saved.

Takeaway

A worktree that borrows another checkout’s node_modules gets that checkout’s workspace packages too, and typecheck and tests then run against the wrong code, passing or failing. Run pnpm install --offline in every new worktree.

If your agents create worktrees, run the check above in one of them today.

Next.js "Cannot read properties of undefined (reading 'length')" in installChunk: two processes share .next

Mon, 22 Jun 2026 13:01:00 +0200

I opened the dashboard of my Next.js app and got a 500 error:

 GET /app/dashboard 500 in 285ms
 ⨯ TypeError: Cannot read properties of undefined (reading 'length')
 at <unknown> (.next/server/app/app/accounts/page.js:685:47)
 at Object.<anonymous> (.next/server/app/app/accounts/page.js:688:3) {
 page: '/app/accounts'
}
 GET /app/accounts 500 in 278ms

If the stack points into generated .next/server/.../page.js and the full trace ends in installChunk in webpack-runtime.js, your code is fine. Another Next process has written into the .next directory your dev server is reading from.

The setup

Next.js 15.5 with webpack in dev, in a pnpm monorepo (the app lives in apps/web)
pnpm dev running in my terminal on port 3000
A coding agent working in the same checkout, which checks its work with pnpm build

What went wrong

I asked the agent to fix the 500. It did what anyone would do with reading 'length': it grepped the app for .length. The accounts page didn’t touch a collection at all, so the agent followed the authenticated layout into the auth code and suspected an import there. Then it opened the generated page.js at line 685 and found webpack glue, not application code.

To reproduce, the agent ran pnpm build, which succeeded, and then started its own next dev -p 3001 next to mine. The first request to /app/accounts returned 200. After a request to the dashboard, both routes returned 500, and this time the error page carried the whole stack:

TypeError: Cannot read properties of undefined (reading 'length')
 at installChunk (apps/web/.next/server/webpack-runtime.js:249:41)
 at __webpack_require__.f.require (apps/web/.next/server/webpack-runtime.js:259:15)
 ...
 at apps/web/.next/server/app/app/dashboard/page.js:718:47

Line 249 is in webpack’s chunk loader:

var installChunk = (chunk) => {
 var moreModules = chunk.modules, chunkIds = chunk.ids, runtime = chunk.runtime;
 // ...
 for(var i = 0; i < chunkIds.length; i++)
 installedChunks[chunkIds[i]] = 1;
};

chunk is whatever require("./" + chunkFile) returned. chunk.ids was undefined, so the file on disk was not the chunk this runtime expected. A ps showed why: two next-server processes, mine on 3000 and the agent’s on 3001, both compiling into apps/web/.next. On top of that, the agent’s next build had written its production output into the same directory a few minutes earlier.

Both of the usual ways to reproduce a bug, a fresh build and a second server, are ways to cause this one. Before you reproduce, look for other Next processes on the checkout:

ps -axo pid,ppid,command | grep '[n]ext-server'

Fix

Give next dev its own output directory in next.config.mjs:

export default {
 // `next dev` and `next build` generate incompatible webpack chunk graphs.
 // Keep their output separate so a build cannot break a running dev server.
 distDir: process.env.NODE_ENV === "development" ? ".next-dev" : ".next",
};

Ignore it in .gitignore:

.next/
.next-dev/

Add its generated types to tsconfig.json, next to the .next ones:

"include": ["next-env.d.ts", "**/*.ts", "**/*.tsx", ".next/types/**/*.ts", ".next-dev/types/**/*.ts"]

The running dev server picked up the config change and rebuilt into .next-dev. From then on, next build writes to .next and the dev server never reads it.

To recover without the config change, stop every Next process except one, delete .next and restart pnpm dev.

If you’re on Next.js 16, you already have this: next dev writes to .next/dev, so dev and build can run concurrently .

The follow-on trap: `next-env.d.ts`

Next rewrites next-env.d.ts on every dev start and every build, and the file names the types of whichever directory ran last:

/// <reference path="./.next-dev/types/routes.d.ts" />

After every next build, it says ./.next/types/routes.d.ts instead. The file was tracked, so it showed up as modified in later sessions, and the agents kept “fixing” it with git show HEAD:apps/web/next-env.d.ts > apps/web/next-env.d.ts. Do what the .gitignore from create-next-app does and ignore the file:

echo 'next-env.d.ts' >> apps/web/.gitignore
git rm --cached apps/web/next-env.d.ts

Why

next dev and next build both default to .next in Next.js 15, and they write different chunk graphs into it. A second next dev is no better: two webpack compilers write into one directory, and neither knows about the other. The runtime that is already loaded require()s a chunk by the name from its own graph, gets a file the other compilation wrote, and fails at chunk.ids.length. The error says length, so you go looking for an array in your code.

Coding agents check their work with next build, usually while your dev server is still running, so you’ll hit this more often than you’d think.

Takeaway

An error from inside installChunk points at the build directory, so look for a second Next process before you read your own code. Never let two Next processes share a distDir.

If an agent or a script runs next build in the checkout where your dev server runs, give dev its own distDir today.

Reference data published after the period: provisional values, then frozen observations

Sun, 21 Jun 2026 22:00:00 +0200

A tax engine converts foreign-currency trades into the reporting currency. The law allows two methods per tax year: the central bank’s daily rate for each transaction date, or a single annual rate per currency that the tax authority publishes. Conservative mode always uses the annual rate.

The annual rate for a year is published in the first months of the next year. Users also want in-year numbers, so for most of a calendar year, conservative mode needs a value that does not exist yet.

The same thing happens with any reference value fixed after the period it covers: annual average rates, inflation indices used for indexation, final settlement prices, revised statistics. This post covers the design that handled it, and the bug in the first version of its “immutable cache”.

Make the status part of the value

While the annual rate is unpublished, conservative mode computes on the daily rate as a proxy. Without a label, a reviewer who sees conservative mode using daily rates in September reports it as a bug. So every conversion carries its status:

type Conversion =
 | { ok: true; amount: Decimal; status: "provisional"; rateRefs: RateKey[] } // daily proxy
 | { ok: true; amount: Decimal; status: "final"; rateRefs: [RateKey] } // annual rate published
 | { ok: true; amount: Decimal; status: "frozen"; frozenFrom: "provisional" | "final";
 rateObservations: RateObservation[] } // pinned at handoff
 | { ok: false; reason: "missing-rate"; needed: RateKey[] };

The dashboard shows a provisional badge next to each figure computed that way.

Publication is not a stored flag. The annual rate counts as published when its rate record exists:

const annual = await readRate(tx, annualRateKey(currency, taxYear));
if (annual) return { ok: true, amount: multiply(amount, annual.rate), status: "final", rateRefs: [annualKey] };
await requestAnnualRate(tx, annualKey); // enqueue a fetch job, deduplicated per key
return daily; // the provisional proxy

An open assessment always reads through this function. Once the fetch job stores the annual rate, the next read returns final. No “rate published” boolean can disagree with the data.

While the case is open, everything recomputes live. When it is handed to an advisor, the conversions freeze. That gives two rules: open assessments track the newest data, and handed-off assessments stay reproducible forever.

“Immutable, keyed by the rate’s identity” cannot store a correction

The first schema followed the design doc literally: a rate is a historical fact, so store it once and never refetch.

create table rate (
 source text, base text, quote text, rate_date date,
 rate numeric not null,
 fetched_at timestamptz not null default now(),
 primary key (source, base, quote, rate_date)
);
grant select, insert on rate to app; -- no UPDATE, no DELETE
-- writes: insert ... on conflict do nothing

The same design doc also said: “a correction is a new record, not an overwrite.” The primary key makes that impossible. A provider that corrects a published value sends a second row with the same key, and on conflict do nothing drops it without an error. The cache stays immutable by losing the corrected value.

The second migration separated the two requirements. Observations are append-only, and a small pointer table says which observation is current:

create table rate_observation (
 id uuid primary key default gen_random_uuid(),
 source text, base text, quote text, rate_date date,
 rate numeric not null, fetched_at timestamptz not null
);

create table rate_canonical (
 source text, base text, quote text, rate_date date,
 observation_id uuid not null references rate_observation(id) on delete restrict,
 primary key (source, base, quote, rate_date)
);

create view rate as
 select o.* from rate_canonical c join rate_observation o on o.id = c.observation_id;

A security definer function performs every write. It takes a per-key advisory lock. A retry with an equal value returns the existing observation. A different value inserts a new observation and moves the pointer. Open assessments read the rate view and pick up the correction. A frozen assessment stores the observation IDs it used, one per conversion leg, and never reads the pointer again. A currency that the central bank doesn’t list converts through a pivot currency, so its conversion pins two observations.

Three Postgres gotchas from making it immutable

The first implementation of the freeze broke 10 tests. One fix commit covered three defects:

No NUL in text. The advisory lock key concatenated the key parts with chr(0) as a separator. Postgres text cannot contain a null byte, so every write raised an error. Use a separator that cannot appear in the parts and is legal in text, such as chr(31):

perform pg_advisory_xact_lock(hashtextextended(
 concat_ws(chr(31), p_source, p_base, p_quote, p_rate_date::text), 0));

FOR SHARE needs UPDATE privilege. At handoff, the freeze must not race a correction that moves the pointer. The first version locked the pointer row with select ... for share. In Postgres, every row-locking clause requires UPDATE privilege on the table, and the app role has none, on purpose. The fix takes the same advisory lock the writer takes, so the advisory lock becomes the single point of serialization. When one handoff freezes many conversions, it acquires the locks in a fixed order (sorted by conversion id), so two concurrent handoffs cannot deadlock.

node-postgres turns date into a shifted Date. By default, pg parses a date column (OID 1082) into a JavaScript Date at local midnight. Serialized back as UTC in a timezone east of Greenwich, 2026-03-02 becomes 2026-03-01T23:00:00.000Z, the previous day. The schema types already said string. Keep the literal:

import { types } from "pg";
types.setTypeParser(1082, (value) => value); // 'YYYY-MM-DD' stays a string; timestamptz unaffected

Two smaller rules

Exact decimals end to end. The central bank quotes some currencies per 100 units. The parser divides the listed rate by the quoted quantity in exact decimal arithmetic, and rejects a result that would not terminate rather than rounding it. No rate passes through an IEEE float.
Crypto is an asset, not a currency. The coding agent’s first plan gave crypto its own rate subsystem. I corrected that during the design session: crypto is priced through its trading pair (BTC/EUR, ETH/USD), like a stock, and the pair’s quote currency converts through an ordinary rate. Only a crypto-to-crypto swap needs a separate market valuation, and that is asset valuation, not currency conversion.

Takeaway

When a reference value is published after the period it describes, put the status in the value (provisional, final, frozen) so that a proxy cannot pass for the real number. Let the existence of the record mean “published”. Treat “immutable” and “correctable” as two tables, not one primary key: append observations, point to the current one, and let frozen results pin observation IDs.

If your cache is immutable through ON CONFLICT DO NOTHING, send it a corrected value for an existing key and check whether anything records that the value changed.

Before building a reference catalog, check whether the attribute reaches the computation

Sun, 21 Jun 2026 21:00:00 +0200

A tax engine for investment accounts has to put every instrument into a category, because the category decides the rules. A real share sold after the holding period can be exempt. A CFD on the same stock is a derivative and is taxed no matter how long you hold it. Brokers don’t make this easy. One reports CFDs under “stocks”. A position can be a fractional share or a whole one. An ETF’s ISIN doesn’t say whether the fund pays out dividends or reinvests them.

During the design session, the coding agent recommended a central instrument catalog: ISIN → {category, asset class, accumulating/distributing, derivative, fractional, …}, seeded from a market data provider and maintained over time. Per-broker adapters would propose a category and reconcile it against the catalog, and anything unresolved would go to a review queue. It argued that the accumulating/distributing flag in particular had to come from the catalog.

I rejected it in two sentences:

If we get the dividend, we have to deal with it. If the fund gets the dividend, it’s non-taxable for the individual.

The attribute decides whether an event exists

A distributing fund pays the investor a dividend, and the broker reports it as a dividend event, which is taxed. An accumulating fund keeps the dividend inside the fund. The investor receives nothing, and the broker reports nothing, so nothing is taxed until the units are sold, and the sale is an ordinary sell event.

The flag the catalog was meant to supply only decides whether an event exists, and the transaction ledger already records whether one did. Removing the flag from the catalog changes no input to the engine. It was also the catalog’s main justification.

The reusable check: for every attribute you are about to maintain as reference data, flip its value and ask which computation input changes.

Attribute	What changes if it flips	Already in the events?	Source
Accumulating / distributing	Whether a dividend event exists	Yes: the event is present or absent	None needed
CFD / real share	The rules applied to the same sell event	No	The broker’s own export (instrument type, leverage)
Fractional / whole share	Whether an exemption is arguable	No	The broker’s idiom
Crypto / token / derivative on crypto	Which rules and caps apply	No	The broker’s idiom

The remaining attributes do change the computation. All of them can be read from the broker that produced the record, so classification belongs in the per-broker adapter. The adapter already parses that broker’s export and knows its quirks. A global catalog would only reconcile against them afterward.

Classify where the source idiom lives, and refuse to guess

The adapter for the broker that labels CFDs as stocks classifies by the type and leverage columns of its own export:

function classify(type: unknown, leverage: unknown): Category | { explanation: string } {
 const kind = header(type);
 if (/cfd|contract for difference/.test(kind)) return "derivative";
 const multiplier = Number(text(leverage).replace(/x$/i, ""));
 if (!kind || !Number.isFinite(multiplier) || multiplier < 1) return { explanation: "Type or Leverage is missing or invalid." };
 if (kind.includes("crypto")) {
 return multiplier === 1 ? "crypto" : { explanation: `Crypto Type conflicts with leveraged position (${text(leverage)}).` };
 }
 if (multiplier > 1 && /(stock|etf|real|cfd|contract)/.test(kind)) return "derivative";
 if (multiplier === 1 && /(stock|etf|real|share|security)/.test(kind)) return "security";
 return { explanation: `Type ${text(type)} and Leverage ${text(leverage)} cannot be classified.` };
}

No default branch picks the most likely category. A row the rules cannot classify becomes a review item (AMBIGUOUS_CATEGORY with the raw row reference) and stays out of the tax computation. A misclassified instrument produces a wrong filing with no warning sign, which is worse than an incomplete one.

The ISIN and ticker stay on the record for cross-checking and audit, but nothing uses them to decide a category.

What dropping the catalog removed

A licensed data feed and the work of keeping up with new listings, delistings, and corrections.
A reconciliation layer between adapter output and catalog values, plus the conflict UI that goes with it.
A global table keyed by ISIN that every user’s trades would look up. In this design, real instrument identity is visible only inside the adapter at ingestion, and the stored record carries a pseudonymous ID. That design is described in When the server must compute on confidential data, split the data by who may read it .

The cost moved into the adapters: classification is now only as correct as each adapter. Every adapter has its own test suite for its broker’s export format, and adding a broker means writing its classification rules and tests.

Caveat: some tax systems tax events that never happened

The shortcut holds where tax follows money the investor actually receives. Some tax systems tax deemed income on fund holdings that paid nothing out. Germany’s Vorabpauschale, UK reporting funds’ excess reportable income and the US PFIC regime are examples. There, the accumulating flag produces an income line that no event carries, and you need the attribute from somewhere. That is exactly the case the flip test catches: the computation input changes even though no event changes.

Takeaway

Before you maintain reference data for an attribute, flip it and see which input of your computation changes. If the only effect is whether an event happens, and your event stream already records that, you don’t need the attribute. If an input does change, get the value from whoever produced the record, and send anything it can’t classify to review.

The agent’s recommendation was internally consistent and sounded responsible. It took someone who knew where the taxable event comes from to see that the attribute changed nothing. When an agent proposes reference data, run the flip test on every attribute before accepting it.

When the server must compute on confidential data, split the data by who may read it

Sun, 21 Jun 2026 20:00:00 +0200

I designed a web app that computes a person’s tax on their investment trades. Professional advisory firms run it for their own clients. Clients connect broker APIs or upload statements, and the server runs lot matching, currency conversion, and the holding-period tests. A client with a messy case can hand it to an advisor.

The confidentiality requirement may sound contradictory. The advisor owes the client professional secrecy from the first contact. The platform operator should not be able to read client data at rest, and the same goes for a stolen backup, a tenant boundary bug, or a subpoena served on the platform. The server still has to compute on all of it, including in background jobs while the user is offline.

Neither usual answer works:

Disk or database encryption protects against a stolen disk. The operator, its DBAs, and a subpoena still get plaintext.
Client-side end-to-end encryption fails because the server is the calculator. Background sync, recomputation after new data, and the advisor’s view all need plaintext on the server. In a web app it is weaker than it sounds anyway: the server ships the JavaScript on every load, so an operator that serves malicious code gets the password.

What worked was a question I had skipped: which fields does the computation actually read?

List what the engine needs

For every realized event, the tax engine needs a date, a quantity, amounts in the reporting currency, fees, an event type, and an asset category (security, crypto, derivative). To match lots, it has to know that two trades are the same instrument. It doesn’t need the instrument’s name, ticker, or ISIN, and it doesn’t need the broker, account number, or person.

One step needs real identity: classifying an instrument into a category. A broker may report a CFD under “stocks”, and a share can be fractional or whole. That happens once, at ingestion, inside a per-broker adapter. The adapter sees identity in memory, emits a de-identified, categorized record, and the identity goes elsewhere.

So the data splits into two zones, each with its own readers:

	Zone 1: tax skeleton	Zone 2: identity vault
Contents	surrogate instrument ID, category, dates, quantity, amounts, fees, event type	surrogate → name/ticker/ISIN, broker, account, personal data
Encrypted under	per-tenant key encryption key in a KMS (envelope encryption)	key chain rooted in the user’s password
Readers	engine (in memory, during compute), the firm after handoff	the user in session, the firm after handoff
Platform operator	ciphertext at rest; if it decrypts, it gets anonymous arithmetic	ciphertext only
Background jobs	read and write	write only, through a public-key layer
Lost password	survives; the case can still be computed and filed	lost unless the user has the recovery code

Zone 1 is intentionally not under the user’s key. Under the user’s key, the background sync, the in-year recompute, and the advisor’s computation would all stop working as soon as the user logged out. Background jobs write to Zone 2 by encrypting to the user’s public key. Only a logged-in user can decrypt.

Surrogate IDs: a keyed hash, scoped per user

Lot matching groups trades by instrument, so the skeleton needs a stable ID that is not the instrument. The design is an HMAC with a per-user key held in the KMS:

// Stable across one user's accounts and years, unlinkable across users.
export async function deriveSurrogateInstrumentId(userId: string, instrument: string, broker: string) {
 if (!instrument || !broker) throw new Error("surrogate requires instrument identity and broker");
 return kmsHmac(`surrogate:user:${userId}`, JSON.stringify([instrument, broker]));
}

Each alternative breaks one requirement:

A plain hash of the ISIN falls to a dictionary attack. The set of listed instruments is small, so hashing them all takes seconds.
One global HMAC key makes the same instrument correlate across every user in a dump.
A random ID per trade cannot be recomputed. Next year’s sync, or a second account at another broker, has to land on the same ID without reading the vault, or carried-in lots stop matching.

The keyed hash protects a database dump. An operator who can call the KMS live can still test “does this user hold this instrument”. That is past the threat model’s at-rest boundary, so write it down instead of claiming more.

Access and liability begin at the same instant

Before handoff, the firm’s advisors have no access to a prospect’s case. For identity, the user’s key enforces that cryptographically. The anonymous skeleton sits under the firm’s own key, so for it, access control is the barrier. It matches the legal position: no engagement contract, no professional liability, only precontractual confidentiality.

Handoff is one atomic event. The engagement contract forms, the firm’s anti-money-laundering (AML) and know-your-customer (KYC) duties start, and the vault’s relevant portion is rewrapped to the firm’s key. Data access and liability start together, so neither exists without the other.

The coding agent’s first draft at this step was wrong. It proposed default-minimal disclosure: share only the vault fields the flagged items needed. I rejected it because an advisor who takes on liability and an AML duty cannot sign off on a case half-blind. The boundary that held was purpose, not field sensitivity: everything the past outcome was built from transfers in full, and anything forward-looking (plans, rebalancing ideas, notes) never transfers. Data minimization still applies, but the line runs between past and future, not through the case.

One password, auto-unlock, never transmitted: pick an aPAKE

The first plan was standard session login plus a client-side KDF for the vault key, with OPAQUE as a later upgrade. That leaves two bad choices:

Two passwords: one for login, sent to the server, and one for the vault, used only in the browser. Users hate it.
One password, still POSTed for login, also used to derive the vault key. It gives auto-unlock, and it quietly hands the operator the vault password on every login. Zone 2 stops protecting anything.

With a standard login, one password cannot also stay off the server. An asymmetric PAKE fixes the login side. OPAQUE (RFC 9807) authenticates without the server learning the password, and gives the client an export_key that the server never sees. Root the vault in it:

// K_vault = HKDF(export_key); unwraps user key → RSA private key → vault entries
function deriveVaultKeyBytes(exportKey: Uint8Array): Uint8Array {
 if (exportKey.length === 0) throw new Error("export_key is required");
 return hkdf(sha256, exportKey, new Uint8Array(0), K_VAULT_INFO, 32);
}

A password reset is a new OPAQUE registration with a new export_key. The mandatory recovery code rewraps the vault chain to the new root, and Zone 1 is untouched. Use a vetted library: the registration and login flows and the export_key handling are now part of the vault’s security.

The tenant boundary belongs in the database

Each firm is a tenant, and one firm’s prospects must never show up in another firm’s queries. A forgotten WHERE firm_id = ? should return nothing, not another tenant’s rows. With Postgres row-level security, three details decide whether it fails closed:

alter table client owner to app_migrator; -- the app role is not the owner
alter table client force row level security; -- the owner obeys policies too
create policy firm_isolation on client
 using (firm_id = current_setting('app.firm_id')::uuid)
 with check (firm_id = current_setting('app.firm_id')::uuid);

await db.transaction().execute(async (tx) => {
 await sql`select set_config('app.firm_id', ${firmId}, true)`.execute(tx); // true = transaction-local
 return work(tx);
});

Table owners bypass RLS unless it is forced. Superusers and BYPASSRLS roles bypass it always, so the app must connect as neither.
current_setting without missing_ok, plus the ::uuid cast, raises when the tenant was never set or is empty. A missing tenant is an error, not an empty match.
set_config(..., true) resets at commit or rollback. With false, a pooled connection carries one firm’s ID into the next request, and a transaction-mode pooler can hand the setting to a different client.

Write down the floors

The design makes specific claims and lists what it does not cover. That list is part of the design:

Plaintext exists in memory during ingestion and compute. The claim is “zero to the operator at rest”, never “the platform never sees it”.
amount ÷ quantity approximates a unit price that can be matched against market data. It is exploitable only by someone who already decrypted Zone 1.
The web JavaScript trust gap stays open. OPAQUE stops offline cracking of a stolen verifier table, not a malicious bundle. A native client is the eventual fix.
An API-connected account reveals which broker it is, because the token names it. A manual upload keeps the broker in Zone 2. The user gets that choice in the UI.
A forgotten password without the recovery code loses the labels. The skeleton survives, and the user is told to keep their own copies of the source statements.

Takeaway

When the server has to be the calculator, don’t choose between “encrypt the disk” and “encrypt everything client-side”. List the readers (user, professional, engine, operator) and what each one needs. Then check whether the computation can run without identity. When it can, identity moves into a zone the computation never opens, and the remaining decisions follow from that: keyed surrogates, a handoff that rewraps keys, an aPAKE for the single password.

In practice, I ran the design as an interview with a coding agent. It asked one question at a time, proposed options with a recommendation, and recorded each decision as an ADR with a “Why this is non-obvious” section: the design a reader would expect, and why it was rejected here. Several recommendations were reasonable but wrong. Two of my corrections removed whole planned pieces: the default-minimal handoff, and an instrument catalog that event-driven classification made unnecessary.

An MCP server has more outputs than tool results

Tue, 19 May 2026 20:00:00 +0200

I designed a local stdio MCP server in Go that gives an LLM scoped access to a shared password vault. Two facts set the threat model. Any member of a shared vault can write an entry comment that another member’s agent later reads, so stored text is attacker-controlled. And the model can call every tool in the manifest, so whoever steers the model gets that authority too.

The plan started with a prompt-injection fence around user strings. Most of the work ended up elsewhere.

Keep the secret off the transcript

Once get_password returns plaintext, the value sits in the model’s context and the host’s chat history, and no scope rule can take it back. So the everyday path never crosses that boundary:

copy_password_to_clipboard takes an ID, writes the value to the OS clipboard from inside the server, and returns Password (id=42) copied to clipboard. Auto-clear compares a SHA-256 of the current clipboard first, so it can’t wipe something the user copied later.
create_password_generated generates the value server-side and returns only the new row’s ID.
get_password returns the literal <redacted; revealPlaintext=never> by default, and its description points the model at the clipboard tool.

The confirmation first included the fenced entry title. That was removed: the model already has the title, and “action just completed” is a bad moment to hand it attacker-written text. A tool result should echo as little as possible.

Plaintext in chat remains for cases where the model has to reason about the value, such as a reuse audit. It sits behind a setting: never, per_call or always.

That setting began as a boolean, and the trap was in the annotations. A read is correctly readOnlyHint=true, so the host’s confirmation never fires, and one injected instruction could walk list_passwords → get_password across every entry in scope. Marking the read as destructive doesn’t help: annotations are static at registration, and hosts may ignore the cue on a read-only tool.

So per_call is an OS-native modal raised inside the server before the value is serialized, with a 30-second timeout that fails closed. If no GUI is available, the server refuses to start in that mode.

Two more manifest details matter. destructiveHint and openWorldHint default to true in the spec, which is why the Go SDK types them as *bool, and every tool here sets them explicitly. And the manifest itself is a control: if the clipboard probe fails, the tool is never registered, so the model can’t try it.

Authority lives in code

Tool descriptions are instructions to something an attacker may be steering, so they count as guidance, not control. Scope is a deterministic filter on (vault, [tags], mode), and the type system gates writes:

func (s *scopedOps) DeletePassword(ctx context.Context, rule *scope.ResolvedRule, id ID) error

A rule only comes from the scope resolver, so a handler that skips the check has nothing to pass and doesn’t compile. A runtime check catches the wrong rule: another vault, or a read-only mode. The scope itself lives in the OS keychain, and the server only reads it so that no tool call can widen it.

The input schema enforces less than it looks like

With the Go SDK, mcp.AddTool derives the input schema from struct tags. An earlier tag format, jsonschema:"required,description=...", failed at registration because the jsonschema tag is now the property description and nothing else. The tags were rewritten as prose:

Mode string `json:"mode,omitempty" jsonschema:"Strategy. (Schema hints: enum=memorable; enum=random.)"`
Length int `json:"length,omitempty" jsonschema:"Length. (Schema hints: minimum=6; maximum=32.)"`

That reads like a constraint and enforces nothing. The generated schema has only type and description, and a call with mode=evil, length=999 reaches the handler. The handler’s own validation is the only check, and it has to reject: the generator below it silently clamps, so the model would ask for one thing and get another.

The schema does give you types, required fields, and additionalProperties: false. When it rejects a call, the SDK answers before your handler runs, so your audit middleware never sees it, and the message echoes the offending value.

Beyond that, arguments are IDs, enums, and bounded integers. Omitted and [] mean different things on an update, so that field is a *[]int64. The model can’t push a row out of its own visibility, because one allowed tag must remain.

Stored content is deliberately not filtered. A comment reading “ignore previous instructions” is legal data, and the regular clients show it verbatim. Escaping happens when text leaves for the model.

The fence, and the encoder after it

Read tools wrap each user string, and their descriptions tell the model that fenced text is data:

<user_content>{escaped}</user_content>

&, < and > are HTML-escaped so a stored payload can’t close the tag. C0 controls are escaped too, because a stored ESC paints ANSI sequences on the terminal of anyone inspecting raw traffic, and the nine bidi overrides are replaced so stored text can’t reorder what a human approves.

“Ignore previous instructions” passes through unchanged. Against natural-language injection, the fence makes the boundary explicit. The real limit is what a steered model can reach.

Two gaps remain, both visible only by running it:

The tags don’t arrive as written. Returning a typed Out lets the SDK serialize it with encoding/json, which HTML-escapes by default. A title reaches the model’s text block as <user_content>..., and & becomes &. The payload still can’t close the fence, but the literal the tool description promises isn’t the literal on the wire. To control it, set result.Content yourself using an encoder with SetEscapeHTML(false).
The escape table works on bytes. It covers bytes below 0x20, so DEL and U+009B, the 8-bit CSI encoded as C2 9B, pass through. A rune-based check with unicode.IsControl catches both.

Each sink needs its own treatment

Sink	Data	Treatment
Tool result	Free text	Escape, replace bidi overrides, fence. Keep ZWJ and RLM, which Arabic, Indic scripts, and emoji need
Tool result	Vault and tag names	Strip control and format runes, cap at 256, then fence, which still protects a handler that forgets the fence
Startup error	Backend-supplied version	Allowlist regex, otherwise `unknown`
Consent dialog	Entry title	Reject controls, line separators and bidi; truncate to 80 runes
Subprocess	Entry title	Environment variable, never script source
Audit log	Content fields	Dropped; a search query is logged as its length
Tool error	Anything	Fixed text

The consent dialog shows why one sanitizer isn’t enough. The AppleScript is a constant, and the title arrives as data:

set t to system attribute "ENTRY_TITLE"
display dialog "Reveal password \"" & t & "\" to your AI assistant?" -- ...

That stops code injection and leaves another problem. The dialog still renders the title, so newlines, bidi overrides, or a long payload could bury the warning and harvest a click on Allow. Data-not-code and safe-to-display are separate properties. The interpreter is called by absolute path as well, because a PATH-shadowed binary would receive the title in its environment.

The error path talks to the model

In the Go SDK, a non-nil error from a tool handler becomes isError content, so err.Error() goes to the model. Every handler here runs through a middleware that classifies the error, returns fixed text, and sends the reason to the audit log:

func wrapHandler[In, Out any](s *Server, name string,
	inner func(context.Context, In) (Out, error),
) mcp.ToolHandlerFor[In, Out] {
	return func(ctx context.Context, _ *mcp.CallToolRequest, args In) (
		result *mcp.CallToolResult, out Out, err error, // named: recover writes these
	) {
		defer func() {
			if r := recover(); r != nil {
				result, out, err = errGeneric(), *new(Out), nil
			}
		}()
		// deadline, session check, inner(), classify, audit
	}
}

The named returns matter. Without them, a recovered panic returns (nil, zero, nil), which the host reads as an empty success. The panic value is logged by type, and by content only if it’s a string or error, because %v on a struct could print a password.

What still had to leave the error text:

Scope details. A bypass error included the rule’s vault ID and mode, so a steered model could map the scope one call at a time. Those fields moved to a typed error for the audit log.
Subprocess output. fmt.Errorf("pbcopy: %w: %s", err, out) could carry clipboard content. Stdout now goes to /dev/null, and stderr never joins the error.
Identity. An “account changed” error named the username and backend. It can fire on every tool call, putting N copies of an email address into a transcript that may get exported or summarized. The final text names nobody.
Tamperable integers. A version-mismatch error quoted a schema number from a local file, whose author then controlled a value the model reads as authoritative.

The audit log is an output too. Its redactor is a key-name denylist over any, and it needed six fixes: nested maps, []map[string]any, pointers and typed maps, nested structs, Password/URL casing, and map keys in result metadata. A new field called token would still be logged. An allowlist of loggable fields changes that failure from a leaked secret to a missing log value.

The host spawns the server once and keeps it alive for the chat. An ADR said the session is read on every call, so a login as a different account gets refused. The implementation read it once at startup, and the error sentinels existed with no code path producing them. A second login with a chat open left the server acting as the old account, with no warning.

The check now runs in the middleware, before each handler. It came with a race: a check could pass, then a sibling refresh could swap the credentials before the handler used them. The check and the credential snapshot now happen in one critical section, and the handler receives that snapshot through its context.

What stays open

Natural-language injection is contained, not solved. The scope rules limit the blast radius.
The tag filter runs on the server after the backend returns the whole vault, so the binary is trusted. A server-issued scoped token is the real fix.
Auto-clear dies with the process, and clipboard managers keep their own history.

Takeaway

Treat the SDK as part of your attack surface: read what your schema really contains, and look at the bytes a tool result becomes. Then list every place a value can go, including errors, dialogs, and logs. Remove the sinks you can, as the clipboard tool did for secrets, and give each remaining sink the treatment its grammar calls for.

Codex 401 in an agent shell: check credentials before blaming API key precedence

Tue, 05 May 2026 00:00:00 +0000

A regular terminal reported a ChatGPT login, but Codex requests inside an agent’s shell failed with:

HTTP error: 401 Unauthorized

An inherited OPENAI_API_KEY looked suspicious. The useful next step was a controlled credential-context comparison. The recorded investigation never proved that the key overrode the ChatGPT login, and it never showed a successful retry after removing it.

What the May 5 investigation established

The installed CLI was codex-cli 0.128.0. The user reported that codex login status returned Logged in using ChatGPT before launching the agent. Inside the agent, the same status command returned Not logged in, and this request repeatedly failed:

codex exec --skip-git-repo-check "echo hello"

An early environment check found OPENAI_API_KEY present. A separate request to the OpenAI models endpoint, explicitly using that variable, returned HTTP 401. That established that the credential failed that direct probe. It did not establish which credential Codex selected.

Later checks found the variable absent, yet Codex still failed. One retry explicitly supplied an empty OPENAI_API_KEY and also received 401 responses. The chain contains no controlled env -u comparison that restored subscription access. The document-cleanup task continued with Python; Codex authentication was not shown to recover.

Calling this a confirmed case of an environment key shadowing OAuth would overstate the evidence. The login-status mismatch deserved investigation before declaring the subscription token expired or replacing it with an API key.

Compare the launch contexts without exposing secrets

Run these checks locally in both the working terminal and the failing agent shell:

command -v codex
codex --version
codex login status
python3 - <<'PY'
import os
for name in ("HOME", "CODEX_HOME", "OPENAI_API_KEY", "CODEX_API_KEY"):
 print(f"{name}: {'set' if name in os.environ else 'unset'}")
PY

Keep login-status output private if it includes credential details. The Python check prints names and presence only. Do not dump the environment, search credential files, or paste tokens into a troubleshooting conversation.

Compare the actual HOME and CODEX_HOME paths privately, too. Presence alone cannot tell you whether two processes use the same directory. Check the executable and launch configuration for differences before changing authentication.

OpenAI’s authentication documentation distinguishes ChatGPT subscription sign-in from API-key access. It also describes credential storage in a file under CODEX_HOME or an OS credential store. A profile HOME mismatch or unavailable credential store is a separate explanation for a missing login.

To isolate the suspected variable on a Unix-like system, keep the same executable, working directory, HOME, and CODEX_HOME, then compare:

codex login status
env -u OPENAI_API_KEY codex login status

Removing one variable affects only the second process. It does not delete the saved login or change the parent shell. If you intend to make a model request, repeat the same minimal prompt with and without that variable and compare completion and exit status. A status check alone does not prove a request will succeed. This paired request test was not completed in the May investigation.

Verification on October 31: avoid a universal precedence rule

A separate check on 2026-10-31 used installed Codex 0.166.0 with isolated, empty HOME and CODEX_HOME directories. codex login status returned Not logged in both without a key and with a dummy OPENAI_API_KEY. No model request or real credential was used. This checks status behavior without a saved login; it does not test precedence over OAuth.

The upstream authentication loader , inspected that day, has an environment-key precedence branch for CODEX_API_KEY, conditional on that mechanism being enabled and API-key authentication being allowed. That is not a universal rule about OPENAI_API_KEY, nor proof of what version 0.128.0 selected in May.

The earlier trusted-directory error was separate: --skip-git-repo-check bypassed that check and exposed the authentication failure. Initializing unrelated directories as Git repositories would not repair credentials.

Before logging out or deleting a working login, compare the credential context of your two launch paths and change one variable at a time.

Tilde pointed at a profile home, so the export landed elsewhere

Mon, 04 May 2026 00:00:00 +0000

The export command finished, but the expected directory was empty. Then Python reported [Errno 2] No such file or directory for a script that existed in the user’s home. The runner’s HOME pointed at a profile-specific directory. Fixing this required explicit paths for both the export destination and the Python environment.

Two homes, two plausible directory trees

An initial attempt to read a helper resolved to a profile-home path where the file did not exist. During the later export, cd ~/transfer succeeded because that directory existed too. A virtual environment had been created there, but the actual transfer script lived under the user’s expected home.

The terminal output settled the discrepancy: printing HOME returned the profile-specific home. Comparing the two transfer directories found the script only under the expected user-home root. Comparing output directories found exported data under that root and an empty directory under the profile home. The report also existed only under the expected root.

Changing directories alone would still leave Python’s Path("~/data").expanduser() following the process’s HOME. The repair replaced tilde-based output defaults with explicit absolute destinations, including separate report and log paths. The script supported overrides such as KB_OUT_ROOT. Running it from the intended transfer directory also selected that directory’s .venv/bin/python.

This was observed in a historical Hermes runner environment. It does not establish that every Hermes profile changes HOME; consult the current Hermes documentation for your setup. A configured profile directory and the home directory you expect are separate facts to check.

Check the process before searching for files

Run these inside the same terminal tool or wrapper that launches the job:

printf 'HOME=%s\n' "$HOME"
id
pwd
python3 -c 'import sys; from pathlib import Path; print(sys.executable); print(Path.home())'

Compare those paths with the expected project root. Limit directory checks to those known roots; a filesystem-wide search adds noise and can expose unrelated private files. A successful cd proves only that a directory exists, not that it contains the intended script or environment.

Shell expansion happens before the child starts

In Bash, unquoted ~/data expands in the calling shell. In env HOME=/somewhere python3 ... ~/data, the argument has already expanded before env changes the child’s environment. Single-quoted '~/data' stays literal; Python must expand it explicitly. On Unix, Path.expanduser() uses HOME when it is set.

This isolated example demonstrates both cases without touching real home directories:

SCRATCH="${TMPDIR:?Set TMPDIR to your scratch directory}/home-path-demo"
mkdir -p "$SCRATCH/caller-home" "$SCRATCH/profile-home"
PYTHON="$(command -v python3)"
export SCRATCH PYTHON
HOME="$SCRATCH/caller-home" bash --noprofile --norc <<'SH'
env HOME="$SCRATCH/profile-home" "$PYTHON" -c \
 'import sys; print(sys.argv[1])' ~/data
env HOME="$SCRATCH/profile-home" "$PYTHON" -c \
 'import sys; from pathlib import Path; print(sys.argv[1]); print(Path(sys.argv[1]).expanduser())' '~/data'
SH

The first command prints the caller’s home path. The second prints the literal ~/data, then the profile-home path.

Pin the interpreter and verify the artifact

Here is a standalone local inventory example. The program reads its input directory as read-only; the output root is explicit. It requires Python’s venv module and uses only scratch fixtures.

SCRATCH="${TMPDIR:?Set TMPDIR to your scratch directory}/home-path-demo"
mkdir -p "$SCRATCH/source" "$SCRATCH/exports"
printf 'fixture\n' > "$SCRATCH/source/sample.txt"
python3 -m venv --without-pip "$SCRATCH/venv"
export DATA_ROOT="$SCRATCH/source" KB_OUT_ROOT="$SCRATCH/exports"
"$SCRATCH/venv/bin/python" <<'PY'
import os
import sys
from pathlib import Path

source = Path(os.environ["DATA_ROOT"])
out = Path(os.environ["KB_OUT_ROOT"])
if not source.is_absolute() or not out.is_absolute():
 raise ValueError("Use absolute input and output roots")
artifact = out / "inventory.txt"
artifact.write_text("\n".join(sorted(p.name for p in source.glob("*.txt"))) + "\n")
assert artifact.read_text() == "sample.txt\n"
print("interpreter:", sys.executable)
print("verified artifact:", artifact.resolve())
PY

For a real export, validate expected contents at the selected destination and check the report there. In the failed run, a pipeline ending in tail returned zero even though Python could not open the script. Command status alone missed both the launch failure and the path mismatch. Check your job’s resolved interpreter and output root before its next run.

Filename HTML injection: entities bypass a raw URL blocklist

Mon, 20 Apr 2026 00:00:00 +0000

A filename filter that rejects https:// can still admit an HTML attribute containing https://. If an application inserts that filename into HTML without escaping it, the parser decodes the character references and produces a usable URL. Put the repair at the rendering boundary: render filenames as text, even when another client already validates them.

This example synthetically demonstrates the encoding mismatch. It makes no claim about a particular product, deployment, or mail-client version.

The filter and the renderer see different strings

The vulnerable flow has two independent mistakes:

A producer stores attacker-controlled text that another component assumes is safe because a UI restricts filename characters.
An HTML generator interpolates that text as markup. A server-side blocklist searches its stored spelling for literal URL schemes.

An API, import job, or older client can bypass the first UI’s restrictions. The value can remain harmless in a web page that escapes it, then become markup in a generated email, export, or share preview that does not. In the API flow, validate the stored value only as data; do not treat it as safe HTML.

In an authorized test, a probe filename could contain:

<img src="https&#58;&#47;&#47;example.invalid&#47;pixel">.txt

Here : represents a colon and / a slash. Attribute parsing resolves those references before the image loader consumes the URL. The reserved .invalid domain keeps this example from contacting a real collection endpoint.

An image request can demonstrate remote-content loading under the tested settings. It does not establish JavaScript execution, arbitrary file access, or credential theft. A mail client may strip event handlers while retaining an image element, and it may block external images entirely.

Reproduce the mismatch without sending mail

Save this as check_filename.py and run python3 check_filename.py. It uses Python’s HTML parser to inspect attribute values; it does not open a browser or make network requests.

from html import escape
from html.parser import HTMLParser

filename = '<img src="https&#58;&#47;&#47;example.invalid&#47;pixel">.txt'
assert "http://" not in filename and "https://" not in filename

class Inspect(HTMLParser):
 def __init__(self):
 super().__init__()
 self.images = []
 self.text = []

 def handle_starttag(self, tag, attrs):
 if tag == "img":
 self.images.append(dict(attrs).get("src"))

 def handle_data(self, data):
 self.text.append(data)

unsafe = Inspect()
unsafe.feed(f"<p>Shared file: {filename}</p>")
unsafe.close()
assert unsafe.images == ["https://example.invalid/pixel"]

safe = Inspect()
safe.feed(f"<p>Shared file: {escape(filename, quote=True)}</p>")
safe.close()
assert safe.images == []
assert "".join(safe.text) == f"Shared file: {filename}"
print("unsafe: URL decoded; safe: filename preserved as text")

The safe version escapes the ampersands as well as the angle brackets. The parser displays the original entity spelling as filename text; it does not reparse that displayed text into another HTML element.

Prevent markup at every filename sink

Use your template engine’s normal autoescaping for text content. In a DOM API, assign textContent; avoid passing the filename to innerHTML. Apply context-appropriate encoding if a value belongs in an attribute. A filename placed inside a URL requires URL construction and validation as well as HTML attribute escaping.

If a feature intentionally accepts rich HTML, use a maintained allowlist sanitizer with an explicit policy for external resources and URL schemes. Decoding and then extending the blocklist still leaves the application maintaining a second, incomplete interpretation of HTML. OWASP’s XSS prevention guidance covers the context-specific encoding rules.

Keep server-side filename rules for filesystem and application constraints. Those rules serve a different purpose from making text safe for an HTML sink.

Verify the actual consumer

Test a filename created outside the restrictive UI, then follow it into each renderer. Inspect the generated HTML or MIME source as well as the visible result. For generated mail, test composition separately from recipient viewing.

On an authorized receiving client, use a controlled endpoint and record whether external content was enabled. Distinguish an outbound request from visible rendering and from script execution; one result does not prove the others. After the fix, repeat the probe and confirm that the filename appears literally, no injected element exists, and no request attributable to it occurs.

Add this filename to the HTML generator regression tests. Testing only the upload validator would miss the boundary where the text became active HTML content.

If GPG is missing, signature verification must fail closed

Mon, 20 Apr 2026 00:00:00 +0000

An updater effectively contained the equivalent of if gpg: verify_signature(), then extracted the downloaded archive unconditionally. On a minimal installation without the optional binding, unsigned content took the trusted path. If authenticity is required, treat an unavailable verifier as a hard failure, not permission to continue.

This pattern is CWE-347, improper verification of a cryptographic signature . It becomes more dangerous when the next stage expands an archive, loads a plugin, flashes firmware, or executes an installer, because the bypass then feeds directly into execution.

Optional crypto turns verification into best effort

This shape is unsafe:

if verifier is not None:
 if not verifier.verify(payload, signature):
 raise SignatureError("bad signature")

extract(payload) # Runs when no verifier exists.

The absence branch is the bypass. During review, search for trust decisions guarded by import success, feature flags, platform checks, or broad exception handlers, then follow each branch to the first sensitive consumer. A green path should require a positive verification result.

An attacker still needs influence over the downloaded payload or its source. That may come from a compromised update endpoint, repository, mirror, DNS or TLS failure, or another write primitive. Missing verification does not by itself prove remote code execution; it removes the authenticity barrier that should contain those failures.

Archive extraction can turn bypass into a file write

A second bug often makes the impact concrete: archive members such as ../../outside or symlinks that escape the destination. Joining a destination with an attacker-controlled member name does not remove traversal so the archive bug can turn the bypass into a file write.

Python added extraction filters in 3.12, and Python 3.14 changed the default to the data filter. The current tarfile documentation still recommends explicit filtering and additional limits because no built-in filter stops every denial-of-service or filesystem hazard.

This local fixture asserts both failure paths. Save it as verify_extract.py and run it only in a disposable directory:

import io
import tarfile
import tempfile

class VerificationError(Exception):
 pass

def verify_and_extract(payload, signature, verifier, destination):
 if verifier is None:
 raise VerificationError("signature verifier unavailable")
 if not verifier(payload, signature):
 raise VerificationError("signature rejected")
 if not hasattr(tarfile, "data_filter"):
 raise RuntimeError("safe tar filter unavailable")
 with tarfile.open(fileobj=io.BytesIO(payload), mode="r:*") as archive:
 archive.extractall(destination, filter="data")

def traversal_archive():
 output = io.BytesIO()
 with tarfile.open(fileobj=output, mode="w") as archive:
 body = b"test"
 member = tarfile.TarInfo("../outside")
 member.size = len(body)
 archive.addfile(member, io.BytesIO(body))
 return output.getvalue()

payload = traversal_archive()
with tempfile.TemporaryDirectory() as destination:
 try:
 verify_and_extract(payload, b"sig", None, destination)
 raise AssertionError("missing verifier was accepted")
 except VerificationError:
 pass

 try:
 verify_and_extract(payload, b"sig", lambda data, sig: True, destination)
 raise AssertionError("traversal member was accepted")
 except tarfile.FilterError:
 pass

print("missing verifier and traversal archive rejected")

The fake verifier isolates extraction behavior; it is not production signature logic. The fixture passes on Python 3.12 or newer when the filter is available.

Make verification a prerequisite state

Package the verifier as a required dependency for update support. If it cannot initialize, turn off updating and report a clear error. Pin the expected public key or trust root, validate the signature over the exact bytes later consumed, and reject missing, malformed, expired, or wrong-signer results. Do not catch verification errors and continue.

After verification, extract into a new temporary directory with filter="data". Reject links if they are unnecessary, limit member count and total expanded size, allow only expected file types, and publish the completed tree atomically. Run the updater with the least filesystem privilege it needs.

Regression tests should cover a valid signature, an invalid signature, a missing signature, an unavailable verifier, a corrupt archive, a traversal member, an escaping link, and an oversized archive. Assert that no file outside the staging directory changes on every failure.

Audit every if crypto_library: branch as if the dependency were absent, because eventually it will be.

Use public SDK integration tests to find API coverage questions

Mon, 20 Apr 2026 00:00:00 +0000

A well-tested API can still leave you unsure where to investigate next. Compare its exposed operations with the vendor’s public SDK integration tests, then inspect what those tests actually assert. The result is a list of coverage questions you can investigate within an authorized scope.

The starting assumption in one review was that the system was well tested, so ordinary scenarios were unlikely to reveal much. Reading the SDK tests changed the next question: which behaviors did this particular suite demonstrate? That comparison produced candidate leads. It did not provide a complete coverage audit or confirm a vulnerability.

Build an operation inventory before searching tests

Start with the documented API operations available to the account or environment under review. Record the API version and the SDK release or commit beside that inventory. An operation introduced after the SDK release should not be counted as an unexplained omission.

Map each operation to SDK methods and then to integration tests. Follow wrappers and shared fixtures; a test name does not have to match an operation name. Keep unresolved mappings marked as unknown.

Text searches help locate candidates, but they do not measure runtime coverage. A matching method name might appear only in setup, a comment, or an assertion about a mock. A missing name might be hidden behind a helper or parameterized test. Check whether each test calls the service, what inputs it supplies, and which results it verifies.

Record assertions, not just test presence

This clean-room fixture is invented. Its operation names and test behavior do not describe any real service:

Operation	Evidence in the fictional public suite	Next coverage question
`create_item`	Creates an item and asserts that an ID exists	Does an invalid field produce the documented rejection?
`update_item`	Updates an item owned by the test account and reads it back	Does a different test account lack permission to update it?
`get_item`	No direct test located; helper usage unresolved	Does a shared fixture exercise retrieval and assert its result?

The last row stays unresolved until someone reads the helper. It would be premature to label it “untested.”

For each mapped test, note its credentials’ role without recording secrets, fixture ownership, expected failure behavior, and assertions. A successful request under one account says little about authorization under another. A negative test that merely catches any exception may pass on a network failure without demonstrating the intended denial.

Use precise notes such as “no assertion for this behavior found in the inspected public suite.” Vendors may have private service tests, separate security suites, or tests in another SDK. Public test omissions cannot prove those checks are absent.

Follow the data to its consumer

A coverage gap can justify a closer look at an input field if an API accepts and returns a string, which proves only acceptance and retrieval. Storage may legitimately preserve arbitrary text.

For a rendering-related hypothesis, maintain a small dataflow record: input field, stored representation, consuming component, output context, and observed handling. The consumer determines whether it treats the string safely. An API response alone cannot demonstrate unsafe rendering or a security impact.

Before any live follow-up, confirm the allowed scope, test accounts, and rate limits; use owned test data and the least disruptive check that answers the question.

If the consumer cannot be inspected or exercised within that scope, leave the hypothesis unresolved. Do not turn a stored value into a vulnerability claim to make the investigation sound complete.

Verify the map before choosing a lead

Have another pass through the fixtures and shared helpers, reconcile SDK and API versions, and separate source inspection from tests actually executed. For every proposed follow-up, write down the missing assertion and the observation that would settle it.

Pick one unresolved row in your own operation map and trace it through the test’s assertions before sending another request.

Google Drive drive.file + Picker: picking a folder grants the folder, not its contents

Sun, 19 Apr 2026 00:17:00 +0200

A web app lets the user pick a Google Drive folder in the Google Picker, then lists that folder from the server with the Drive API v3. The folder was full of files. The listing came back like this, every time:

{"files":[]}

The status was 200, with no 403 and nothing in the response that mentioned permissions.

If you request the drive.file scope and let users pick a folder, this is expected: the grant covers the folder itself and none of the files inside it.

Setup

The browser gets an OAuth access token with the drive.file scope and opens the Picker on a folder view:

const view = new google.picker.DocsView(google.picker.ViewId.FOLDERS)
 .setMimeTypes('application/vnd.google-apps.folder')
 .setSelectFolderEnabled(true)
 .setIncludeFolders(true);

const picker = new google.picker.PickerBuilder()
 .addView(view)
 .setOAuthToken(token)
 .setDeveloperKey(API_KEY)
 .setCallback(onPicked)
 .build();

The callback sends the picked folder ID to the server, which walks the folder with files.list:

const url = new URL('https://www.googleapis.com/drive/v3/files');
url.searchParams.set('q', `'${folderId}' in parents and trashed = false`);
url.searchParams.set('fields', 'nextPageToken, files(id, name, mimeType, size)');
url.searchParams.set('supportsAllDrives', 'true');
url.searchParams.set('includeItemsFromAllDrives', 'true');

drive.file looks like the obvious scope here. The Picker guide tells you to use it, and Google’s scope list marks it non-sensitive, so a public app needs only basic verification. The app only ever reads folders the user picked, which sounds like exactly what the scope is for.

The dead end: setAppId

The first suspect was PickerBuilder.setAppId(). The setup above doesn’t call it, and the Picker guide says to “set the Drive App ID using the Cloud project number to allow the app to access the user’s files”. The theory was that without an app ID, the grant belongs to the access token that was live at pick time, so the server’s later call with a refreshed token sees nothing.

 .setAppId(CLOUD_PROJECT_NUMBER) // the numeric project number, not the project ID

If you stay on drive.file, set it, because the guide asks for it. It didn’t change the result here. The listing was still {"files":[]}.

What’s actually going on

To see where the files went, the server logged the raw body of every files.list response. A truly empty folder and a folder the app can’t see into give the same answer: status 200 and "files": []. The app only had access to the selected folder. Its children were never shared, so Drive left them out of the result and didn’t report an error.

The scope’s description says this once you read it with folders in mind. drive.file covers files “that you open with an app or that the user shares with an app” — picking a folder shares one item: the folder. The docs never say the grant extends to its contents, and it doesn’t.

Fix

Pick the scope that matches what the server does. A server that reads everything under the picked folder can’t do that with drive.file, so the app switched to drive.readonly:

const DRIVE_SCOPE = 'https://www.googleapis.com/auth/drive.readonly';

It kept the Picker for choosing the folder and removed the setAppId call, which drive.readonly doesn’t need.

drive.readonly is a restricted scope , so a public app needs Google’s restricted-scope verification to use it. The app in this story was internal to one Google Workspace domain, so verification didn’t apply.

Update the consent text in your own UI as well. “The app only sees folders you pick” was true under drive.file and false after the switch. The new text says the app has read-only access to the whole Drive and downloads only from the folders the user picks.

Two other ways out may fit your constraints better. If users can pick files instead of a folder, enable multi-select in the Picker: every picked file is shared with the app, and drive.file covers it. Or use a service account and have users share the folder with its address, which moves the access decision into Drive’s own sharing dialog. We ruled that out here for organizational reasons.

Takeaway

Under drive.file, the Picker shares exactly the items the user selects. A folder is one item, and its children aren’t part of the grant, so listing it returns an empty files array with status 200.

If your app requests drive.file and calls setSelectFolderEnabled(true), list a folder you know isn’t empty from the server and log the raw response before you ship.

rdflib field notes: ns.format matches nothing, and parse() goes online

Sat, 11 Mar 2023 12:37:00 +0100

UUpdate 2026-11-08: retested on rdflib 7.6.0 and added the ConjunctiveGraph deprecation, pyoxigraph, and the RDF 1.2 and SPARQL 1.2 status.

This lookup returns an empty list, with no exception and no warning:

from rdflib import Graph, Literal, Namespace, URIRef

DCT = Namespace("http://purl.org/dc/terms/")
doc = URIRef("http://example.org/doc")

g = Graph()
g.add((doc, DCT["format"], Literal("text/csv")))

list(g.objects(doc, DCT.format)) # []
list(g.objects(doc, DCT["format"])) # [rdflib.term.Literal('text/csv')]

Namespace subclasses str, so DCT.format is the bound method str.format, and a lookup on it matches nothing. RDF code tends to fail this way: you get fewer triples, and that looks exactly like sparse data.

I collected these while re-reading a crawler I wrote in early 2023, and rebuilt each as a minimal repro on rdflib 6.3.2 and 7.6.0 under Python 3.11. Both versions behave the same unless noted. They show what the code does. I have no production logs that say what any of them cost.

Use `ns["format"]` for names that `str` already owns

format, count, index, join, and every other str method name resolve to the method. title is the exception, special-cased to return a URIRef. Writes are safer than reads here: g.add() raises AssertionError on the method object.

For published vocabularies, import from rdflib.namespace. DCTERMS.format is a real URIRef and DCTERMS.nonsense raises AttributeError. For your own vocabulary, ClosedNamespace rejects unknown terms, but .format on it is still the str method, so keep the brackets.

A typo in the base IRI is still a valid IRI

NS = Namespace("https://example.org/vocab/mediaTyp")
NS.mediaType # https://example.org/vocab/mediaTypmediaType

My crawler shipped this. The branch using it asked for a predicate no document contains. ClosedNamespace produces the same glued IRI, because it validates the term and has no opinion on the base. A one-line test covers it, with an allowlist for the rare namespace that ends differently:

assert str(NS)[-1] in "#/"

A failed parse keeps what it already read

Feed Graph.parse() five good N-Triples lines and one bad one. It raises ParserError, and len(g) is 5. Turtle does the same with BadSyntax. If you parse into a shared graph, a truncated download leaves half a document behind.

Give each document its own store and merge only after the parse succeeds:

import tempfile
from pathlib import Path
from rdflib import Graph, URIRef, plugin
from rdflib.store import Store

def load_document(source, fmt, target):
 with tempfile.TemporaryDirectory() as tmp:
 path = str(Path(tmp) / "graph")
 store = plugin.get("LevelDB", Store)(identifier=URIRef(path))
 graph = Graph(store, identifier=URIRef("urn:doc"))
 graph.open(path, create=True)
 try:
 graph.parse(source, format=fmt)
 target += graph
 finally:
 graph.close()

With a bad document, this raises an exception and target stays empty. The disk store also caps memory. Loading 200,000 synthetic triples (14 MB of N-Triples, one run on Apple Silicon, rdflib 6.3.2, 30 MB baseline after imports):

Store	Time	Peak RSS	Disk
rdflib `Memory`	2.6 s	351 MB	0
`rdflib-leveldb`	15.3 s	85 MB	39 MB
pyoxigraph 0.5.13 `bulk_load`	0.5 s	200 MB	10 MB

Read that as a rough shape and benchmark your own data. rdflib-leveldb isn’t on PyPI, so install it from git. Its last commit is from December 2021, though the snippet above still passes on 7.6.0. On macOS, I used plyvel-ci because plyvel has no Mac wheels. Pyoxigraph has its own API and isn’t a drop-in rdflib store.

A disk store bounds the triples and leaves the input alone. The Turtle parser calls stream.read() on the whole source, with the comment # Not ideal.

`parse()` can go online

Parsing JSON-LD fetches remote @context values. The RDFLib security page says so. Direct cycles are caught (recursive context inclusion after 2 requests). A chain of contexts that each point to a new one isn’t: with Python’s default recursion limit, one small document cost 967 requests to my local server before RecursionError.

Block the network during the parse step:

import urllib.error, urllib.request

class NoNetwork(urllib.request.BaseHandler):
 handler_order = 100 # before the stock HTTPHandler (500)

 def http_open(self, req):
 raise urllib.error.URLError(f"blocked: {req.full_url}")

 https_open = http_open

urllib.request.install_opener(urllib.request.build_opener(NoNetwork()))

Leave out handler_order and the stock handler answers first, so the request goes through. My first attempt did exactly that. The opener is process-global, and the RDFLib docs rank OS-level isolation such as containers above Python-level controls, so use it in addition to those on untrusted input.

Then make the following links a separate, bounded step that you control:

from collections import deque

def crawl(seed, fetch, max_depth=3):
 seen, queue = {seed}, deque([(seed, 0)])
 while queue:
 iri, depth = queue.popleft()
 references = fetch(iri) # fetch, parse in isolation, return IRIs to follow
 if depth < max_depth:
 for ref in references:
 if ref not in seen:
 seen.add(ref)
 queue.append((ref, depth + 1))

On a graph with a cycle and a chain, this visits each IRI once and stops at depth 3.

Keep a workaround’s reason in a test

rdflib #1195 reported that a CONSTRUCT response with application/rdf+xml; charset=UTF-8 broke the plugin lookup. My code registered that literal string as a parser name and pointed a comment at the issue.

Against a fake endpoint sending that header, rdflib 5.0.0 raises PluginException and 6.0.0 works without the workaround (both on Python 3.9). My pin was already 6.0.0 in the commit that added it, so on this path it never changed, and it stayed that way through 6.3.2. Watching the issue wouldn’t have helped: it closed in August 2022 on a PR that only added tests.

A test that exercises the upstream behavior with the workaround off tells you the day it can go. plugin.get() itself still rejects that string on 7.6.0, so strip parameters if you call it directly.

`Dataset` isn’t a drop-in for `ConjunctiveGraph`

ConjunctiveGraph raises a DeprecationWarning on 7.6.0 and the replacement is Dataset, which has a different default. With one quad in a named graph:

Class	`triples()`	`SELECT ?s ?p ?o`
`ConjunctiveGraph()`	1	1
`Dataset()`	0	0
`Dataset(default_union=True)`	1	1

Migrate with default_union=True, unless you want queries limited to the default graph.

On standards, as of November 2026: RDF 1.2 Concepts is a Candidate Recommendation and SPARQL 1.2 Query is a Working Draft, so SPARQL 1.1 remains the Recommendation.

Takeaway

Most of these bugs arrive as an empty result, and nothing downstream complains. For each predicate your pipeline depends on, keep a fixture with one known triple and assert that your code finds it.

Start with this search for unbracketed str names in your own code:

grep -rnE '\b[A-Za-z_]+\.(format|count|index)\b([^(]|$)' --include='*.py' .

Celery ignored task_task_routes: silent failures from growing a crawler on Redis

Sat, 11 Mar 2023 12:19:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

This sat in my crawler’s Celery config:

task_task_routes = {
 "tasks.urgent": {"queue": "celery"},
 "tasks.post": {"queue": "celery:2"},
}

The setting is called task_routes. Celery doesn’t check setting names, so the app started normally, app.conf.task_routes stayed None, and three different tasks all landed in the default queue:

{'celery:1': 3}

That typo arrived with the commit that moved the crawler to Redis priority queues, and it wasn’t the only failure from that stretch that raised nothing. Two small changes catch most of them: compare your settings against Celery’s list of known names, and set task_create_missing_queues = False.

I found these by rereading the code and rebuilt each one against a private Redis, on Python 3.11 with the locked versions (Celery 5.2.7, kombu 5.2.4, redis-py 4.5.4) and on Celery 5.7.0. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

How the crawler got here

Date	Change
2018	Celery on Redis, with the crawl state in Redis too
2020	Mongo for reports; separate priority and regular tasks
2021-09	A monitor task that decides when the crawl is done
2023-02-03	Large structures move from Redis to SQLite, then to Postgres the same night
2023-02-07	Mongo dropped
2023-02-10	Bulk inserts from generators
2023-02-12	Named priority queues; each input parsed into an on-disk graph in a temp directory

Redis ended up mostly as the broker and a few caches. The traps below surfaced in the 2023 part of that table.

Celery ignores settings it doesn’t know

A test finds the typo. DEFAULTS holds every setting name Celery recognizes:

from types import ModuleType

from celery.app.defaults import DEFAULTS

import celeryconfig


def test_celery_settings_are_known():
 unknown = {
 name for name, value in vars(celeryconfig).items()
 if name.islower() and not name.startswith("_")
 and not isinstance(value, ModuleType)
 and name not in DEFAULTS
 }
 assert not unknown

On my config, it reports {'task_task_routes'}, on both Celery versions.

A queue with no consumer looks like a finished crawl

The same commit changed what the worker reads, and the post-processing chain kept an old queue name:

celery -A app worker -Q celery,celery:1,celery:2

chain(step_one.si(), step_two.si()).apply_async(queue="query")

task_create_missing_queues is on by default, so sending to query creates it. In the rebuild, the message sat in that key, and no step ran. Declare your queues and turn creation off:

from kombu import Queue

task_create_missing_queues = False
task_queues = [Queue("celery"), Queue("bulk")]

celery.exceptions.QueueNotFound: "Queue 'query' missing from task_queues"

That protects the sending side only. The -Q list on the worker is still a separate copy, so build it from the same list.

broker_transport_options = {
 "priority_steps": [0, 1, 2],
 "sep": ":",
 "queue_order_strategy": "priority",
}

With these options, kombu stores priority 1 of queue celery queue under the Redis key celery:1. I also named a queue celery:1. A message sent with queue="celery", priority=1 showed up in that key. Priority 0 gets no suffix, so celery:0 never exists.

Pick one scheme: a single queue with priority=, or named queues whose names don’t contain the separator.

`LLEN` 0 doesn’t mean the crawl is done

Every ten minutes, the monitor computed this and declared the crawl done at zero:

enqueued = red.llen("celery") + red.llen("celery:0") + red.llen("celery:1")

It counts the key that never exists and skips celery:2. Even with the right keys, a list length misses work:

State	Monitor	Lists + `HLEN unacked`
Task waiting in `celery:2`	0	1
Task running, `task_acks_late=True`	0	1
Retry waiting on a 6 s countdown	0	1
Chain stranded in `query`	0	0

def outstanding(client, queues=("celery", "celery:1", "celery:2")):
 with client.pipeline() as pipe:
 for queue in queues:
 pipe.llen(queue)
 pipe.hlen("unacked")
 return sum(pipe.execute())

unacked is kombu’s internal bookkeeping hash, and it shows running tasks only because my config sets task_acks_late=True. With early acknowledgment, a running task has already left it. The last row shows the limit of any count: it can’t see a queue you forgot to list.

When you know the fan-out up front, a chord gives a real completion signal. The docs say members must not ignore their results, and I copied that warning into a comment. On the Redis backend, a chord of ignore_result=True members still ran their bodies with all three values, on both versions. I’d follow the docs anyway, since I tested one backend. My tasks also spawned more tasks, which a chord can’t express.

`decode_responses=True` on a shared pool breaks a binary cache

In 2021, my HTTP cache shared the application’s Redis pool, which decoded responses to strings. requests-cache stores pickles, and a pickle starts with byte 0x80. Two identical GETs:

requests-cache	Decoding pool	Separate pool
0.9.8	2 origin hits, one log line	1 origin hit
1.3.3	Second GET raises `UnicodeDecodeError`	1 origin hit

The log line on the old version is all you get:

Unable to deserialize response with key 53630720a0b4bd02: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

The commit that fixed it says what to do: “a separate Redis pool without decoding”:

app_pool = redis.ConnectionPool.from_url(url, decode_responses=True)
cache_pool = redis.ConnectionPool.from_url(url)

`execute()` drains a generator, `group()` doesn’t

I passed generators to session.execute(insert(Row), rows()) to save memory. SQLAlchemy reads them to the end first: on 2.0.7 and 2.1.8, with a SQLite engine, the generator had yielded 5,000 of 5,000 rows before the first INSERT. Batch it yourself:

from itertools import islice


def batches(items, size=1000):
 items = iter(items)
 while batch := list(islice(items, size)):
 yield batch


for batch in batches(rows()):
 session.execute(insert(Row), batch)
session.commit()

With 2,500 rows, the three INSERTs fired after 1,000, 2,000, and 2,500 yields.

group(generator).apply_async() works the other way and publishes as it iterates. With a generator that yielded two signatures and then raised, one task was already in Redis, and the caller got the exception with no GroupResult. Make those tasks safe to run twice. I still prefer group to chunks, which loses retries .

Takeaway

Most of this was introduced or fixed during ten days of refactoring in February 2023, and almost none of it raised an error then. A queue migration needs a written list of each queue, who sends to it, who reads it, and what counts it.

Try the settings test above on your own Celery config.

Celery time_limit on the gevent pool: a 1 s limit let a CPU task run 3 s

Wed, 08 Feb 2023 23:36:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 with gevent 22.10.2, and on Celery 5.7.0 with gevent 26.9.0.

I gave a Celery task time_limit=1 and ran it with -P gevent. The CPU-bound task kept going for 3 s, finished all its work, and was then reported as failed:

state=FAILURE result=TimeLimitExceeded(1) got_after=3.06s

The gevent pool enforces the limit with gevent.Timeout, which can only fire when the task yields to the event loop. CPU-bound code never does. Keep gevent for I/O and run CPU work on a prefork worker.

The workers guide says this in two sentences: the gevent pool “does not implement soft time limits” and “will not enforce the hard time limit if the task is blocking”. It leaves out what that looks like in a task, and how an exception handler makes it worse.

I found this by rereading old code and rebuilt each case on a real worker with a private Redis, on Python 3.11. Both version pairs behaved the same. I have no production logs from that time.

What a 1 s limit does on each pool

Each task does 3 s of work:

Task, 1 s limit	gevent	prefork
CPU loop	Ran the full 3 s, then `TimeLimitExceeded`	Stopped after about 1 s
`time.sleep`	Stopped after about 1 s	Stopped after about 1 s
Sleep inside `try` / `except Exception`	Handler skipped, stopped after about 1 s	Stopped after about 1 s
Sleep inside `try` / bare `except:`, then 3 s more	`SUCCESS` at 4.01 s	`FAILURE` after about 1 s
`soft_time_limit=1` with sleep	Ignored, `SUCCESS` at 3.01 s	`SoftTimeLimitExceeded` after about 1 s

The first row is the nasty one. The work completed, its side effects happened, and the task still reports failure. The timer was pending the whole time and fired at the first switch after the loop.

A CPU-bound task also stalls everything else in that worker process. With -c 8, a task sent during a 3 s CPU loop waited 2.8 s to start, against a few milliseconds on an idle worker. On prefork with -c 2, it still started within a few milliseconds.

Split fetch and parse into separate workers

app.conf.task_routes = {
 "tasks.fetch": {"queue": "fetch"},
 "tasks.parse": {"queue": "parse"},
}

chain(fetch.s(url), parse.s()).apply_async()

celery -A app worker -n fetch@%h -P gevent -c 60 -Q fetch
celery -A app worker -n parse@%h -P prefork -c 2 -Q parse

With time_limit=1 on parse, a chain with a 3 s CPU parse ended in TimeLimitExceeded after 1.1 s, and a 0.1 s parse succeeded. Pass a file path or an object-store key between the two rather than the body.

Why a timeout can’t interrupt CPU work

celery/concurrency/gevent.py in 5.7.0 wraps each task like this:

try:
 with Timeout(timeout):
 return apply_target(target, args, kwargs, callback,
 accept_callback, getpid(),
 propagate=(Timeout,), **rest)
except Timeout:
 return timeout_callback(False, timeout)

Timeout is a timer on the event loop, and everything runs in one thread. The loop gets control only when the current greenlet yields, typically at a monkey-patched socket call or sleep. Prefork stops a task from outside, and the hard limit replaces the child process.

The soft limit is simpler: on_apply takes soft_timeout through **_ and never uses it. The worker log says nothing about that.

A bare `except:` cancels the limit

gevent.Timeout derives from BaseException , so except Exception lets it pass and a bare except: catches it. My code had a fallback like this several calls below the task:

try:
 return lookup(url)
except:
 log.exception("lookup failed, trying the fallback")
 return fallback(url)

The timer fires once. After the handler ate it, the task had no limit at all, and in the rebuild it finished three seconds past the deadline as SUCCESS.

My fix at the time, a commit called “Fail the task on timeout”, added except Timeout at the top of the task. A Timeout stops at the nearest handler that matches, so the nested bare except: on the fallback path kept swallowing it. Change the nested handler:

except Exception:
 log.exception("lookup failed, trying the fallback")
 return fallback(url)

If you need cleanup, catch Timeout and re-raise it. In the rebuild, the cleanup ran at 1.01 s, and Celery recorded TimeLimitExceeded, so there was no need to set the state by hand.

The soft limit goes the other way on prefork: SoftTimeLimitExceeded is an Exception, so a task that moves between pools needs its handlers checked for both.

For a Python loop you own, gevent.sleep(0) on each iteration gives the loop a turn, and the task then stopped at 1.01 s. That can’t help inside a library call that never yields.

The sibling trap on the same crawler: Retry is an Exception, and except Exception eats it .

Takeaway

On the gevent pool, time_limit is only as good as the task’s willingness to yield, so treat it as a limit on I/O waits.

Run grep -rnE --include='*.py' 'except:[[:space:]]*$' . on anything that runs under -P gevent.

Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Tue, 07 Feb 2023 00:01:00 +0100

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

My crawler called Celery’s retry three different ways, and only one did what I meant:

Call	Result
`self.retry(5)`	Task fails with the error below, no retry
`self.retry(timeout=5)`	Retries in 180 s
`self.retry(countdown=5)`	Retries in 5 s

TypeError: 'int' object is not iterable

Even the correct call can go wrong. retry() publishes the new message and then raises celery.exceptions.Retry, an ordinary Exception. An except Exception around it marks the work as failed while the retry is already on its way. Pass countdown= by keyword and let TaskPredicate pass through your broad handlers.

I found these by rereading old code and rebuilt each case on a real worker with a private Redis as broker and result backend, on Python 3.11. Both Celery versions behaved the same. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

Re-raise `TaskPredicate` before `except Exception`

from celery.exceptions import TaskPredicate


@app.task(bind=True, max_retries=2)
def crawl(self, url):
 try:
 fetch(self, url) # calls self.retry(exc=err, countdown=1) on a network error
 except TaskPredicate:
 raise
 except Exception:
 mark_failed(url)
 return "gave up"
 mark_ok(url)
 return "done"

TaskPredicate is the base class of Retry, Ignore and Reject. With this handler, result.get() returned 'done' about 1 s later, after the retry had run.

My handler already knew about one of the three:

except Exception as err:
 if isinstance(err, Ignore):
 raise

Retry needed the same treatment and didn’t get it.

Why the three calls differ

The signature on both versions is:

retry(self, args=None, kwargs=None, exc=None, throw=True, eta=None, countdown=None, max_retries=None, **options)

The first positional parameter is the replacement args for the retried task, so 5 reaches tuple(args) and raises before anything is published.

timeout isn’t a parameter, so it lands in **options and travels with the message. Spying on kombu’s Producer.publish showed timeout=5 arriving there, where it limits how long publishing may take. The delay falls back to default_retry_delay, three minutes by default . Nothing raises. The only hint is the worker log:

Task tasks.crawl[62d0a257-...] retry: Retry in 180s

What `except Exception` does to a retry

Remove the except TaskPredicate branch from the task above and let the first attempt hit a network error:

Time	Event
0.01 s	Retry published, `Retry` caught, URL marked failed
0.01 s	`result.get()` returns `'gave up'`, state `SUCCESS`
1.01 s	Retry runs and marks the URL ok
Later	The backend still says `'gave up'`

The last row surprised me. On the Redis backend, _store_result ignores any write once the stored state is SUCCESS, so the retry’s own return value is dropped. Anyone reading the result sees the attempt that gave up.

Code review won’t catch this reliably, because the placement decides it. A retry inside the try body, even several calls deep, is caught. A retry in a sibling except clause is safe:

try:
 fetch(task, url) # a retry raised in here hits except Exception
except RateLimited as err:
 task.retry(countdown=err.delay) # sibling handlers don't see this one
except Exception:
 mark_failed(url)

My code had both forms in the same function, so some retries worked. Writing raise self.retry(...) changes nothing, and a bare except: catches Retry too. In an older module, a bare except: also caught the TypeError from self.retry(5), which recorded a rate-limited link as a broken one.

A dedupe mark written first makes the retry skip itself

@app.task(bind=True, max_retries=2)
def crawl(self, url):
 if redis_client.sadd("seen", url) == 0:
 return "skipped"
 ...

attempt 0 sadd=1
attempt 1 sadd=0

The retry sees its own mark, returns 'skipped', and the URL is never fetched. Combine that with the swallowed Retry and the URL stays failed for good: in the rebuild, the final status was failed and the result was 'gave up'.

Moving the SADD after the work fixed it: the retry returned 'fetched'. Two concurrent tasks can then fetch the same URL, so the work has to be safe to repeat.

When the retry condition is just an exception type, autoretry_for=(Flaky,), retry_kwargs={"countdown": 1, "max_retries": 2} saves calling retry yourself. The exception has to leave the task body, so a broad handler inside it defeats this as well.

Time limits have a similar problem on the gevent pool, in the opposite direction: Timeout isn’t an Exception .

Takeaway

self.retry() is a raise whose side effect has already happened, so any handler between it and the worker can make your bookkeeping disagree with the queue.

Run grep -rn --include='*.py' '\.retry(' . on your tasks and check each hit for keyword arguments and an enclosing except Exception.

Encrypt sensitive fields inside the ORM

Sat, 30 Oct 2021 18:03:45 +0200

Your database may run where you don’t control the environment: embedded in a desktop app, or in the cloud for a web app. If it stores sensitive data, encrypt those fields so a leaked copy doesn’t expose them. In this article, I show how I did that in two projects, and the drop-in text field for Python ORMs that came out of it.

Encrypted fields in an embedded SQLite

The first case was a desktop app with an SQLite database that only authorized software could read.

I considered encrypting the SQLite file and encrypting fields within it, and I chose the latter. That approach is reusable, and you never have to load the whole database into memory to decrypt it or leave it in plaintext for a moment.

The app already had its database logic, and the goal was to change as little as possible. So I put the encryption inside the ORM, and the rest of the app stayed business as usual.

The details:

Cipher and key: AES-256-CTR with a single key for the whole database. There was no reason for anything more complex. The key is generated on the first run and on the upgrade from the old version.
Storage: binary blobs made querying the database in a terminal for debugging a pain. Base64-encoded ciphertext in a text field works better. A new column holds the row’s initialization vector (IV), a random array of bytes, also base64-encoded.
Migration: it’s not only a schema change. Every row needs the fields encrypted for the updated logic to work.

You can’t query against encrypted fields. The ciphertext is a function of the plaintext, the key, and the IV, and the IV is specific to the row. This is a property of the privacy, not a defect. When you need to query encrypted values, the solution is domain-specific: sometimes you can modify the data or compute hashes. Discuss it with the customer in terms of the security objectives they want to achieve. For the same reasons, the data migration is domain-specific too.

An encrypted payload in MongoDB documents

On another project, the database was MongoDB and had many sensitive fields. I split each document into the sensitive fields and the metadata used for querying. I encrypted the sensitive fields together into a single field. The document also gets the IV and an HMAC over the rest of the document, ciphertext included:

def create_secure_json(form, keys):
 data = get_sensitive(form)
 data_secure_b64, iv_b64 = cryptjson.encrypt_json(data, keys['db'])

 payload = get_metadata(form)
 payload['created'] = str(datetime.datetime.now())
 payload['iv'] = str(iv_b64, 'ascii')
 payload['data'] = str(data_secure_b64, 'ascii')

 hmac_b64 = get_hmac(payload, keys['hmac'])
 payload['hmac'] = str(hmac_b64, 'ascii')
 return payload

This payload is what goes into the database. Use a separate key for the HMAC, and verify it before you decrypt: AES-CBC alone doesn’t detect tampering.

For the encryption, I use the cryptography package from the Python Cryptographic Authority. Encrypting a JSON is no magic: dump it into bytes and encrypt those, here with AES-CBC:

import base64
import secrets

from cryptography.hazmat.primitives import padding
from cryptography.hazmat.primitives.ciphers import Cipher, algorithms, modes


def encrypt_bytes(pt, key):
 iv = secrets.token_bytes(16)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 encryptor = cipher.encryptor()
 padder = padding.PKCS7(128).padder()
 padded_pt = padder.update(pt) + padder.finalize()
 ct = encryptor.update(padded_pt) + encryptor.finalize()
 return base64.b64encode(ct), base64.b64encode(iv)


def decrypt_bytes(b64ct, b64iv, key):
 ct = base64.b64decode(b64ct)
 iv = base64.b64decode(b64iv)
 cipher = Cipher(algorithms.AES(key), modes.CBC(iv))
 decryptor = cipher.decryptor()
 padded_pt = decryptor.update(ct) + decryptor.finalize()
 unpadder = padding.PKCS7(128).unpadder()
 return unpadder.update(padded_pt) + unpadder.finalize()

A drop-in encrypted text field

Both solutions were domain-specific. Still, they showed that the use case is common and that encryption in Python is easy to work with. So I wrote cryptbase, a small library that adds an encrypted text field to SQLAlchemy and Django ORM.

Inside, there is a CryptoContainer. You create it from a ciphertext and an IV, or from a plaintext, and it gives you access to the ciphertext, the plaintext, and the IV in a unified way:

container = CryptoContainer(ciphertext=base64.b64decode(ciphertext),
 iv=base64.b64decode(iv),
 encryptor=encryptor)

container = CryptoContainer(plaintext=value, encryptor=self.encryptor)

A separate IV column complicates the migration, and the goal is a drop-in replacement for text fields. The base64 alphabet has no @, so the field stores both values in one column, with @ as the separator:

<base64 ciphertext>@<base64 IV>

The encryptor is an object with a simple interface, so there can be one per encryption method, such as AES256CTREncryptor:

def encrypt(self, plaintext: bytes) -> Tuple[bytes, bytes]:
 pass

def decrypt(self, ciphertext: bytes, iv: bytes) -> bytes:
 pass

In the end, the ORM gets a simple interface:

# SQLAlchemy
sensitive = Column(EncryptedText(key=DB_KEY))

# Django
sensitive = EncryptedTextField(key=DB_KEY)

You work with these fields the same way as with a text field. When the application reads the value, it gets the plaintext from the container. When the value is written to the database, it’s encrypted, and the __str__ of the CryptoContainer joins the base64-encoded ciphertext and IV:

str(CryptoContainer(plaintext=value, encryptor=self.encryptor))

Dynamic DNS for home routers the hacker way

Thu, 28 Oct 2021 17:48:15 +0200

Skip public DDNS services: Python can handle the same job. This post shows how to modify a DNS record in a zone file, upload the zone over SSH, and expose the update as a small web service.

The setup

A MikroTik router with a dynamic public IPv4 address
A name for the router in my own zone, on my primary DNS server
Python with dnspython, paramiko, Flask, and flask_jwt_extended

I first looked at dynamic DNS over a standard protocol, but I had to abandon the idea. MikroTik also offers its cloud service for a similar result, but I find it less convenient than a name in my own zone. That led me to a way to modify the zone file with Python.

The examples cover IPv4 only. Extending them to IPv6 is a small change.

Modify the zone file

The update has three steps: open the zone file, modify the record, and save the file.

def update(zone, name, newip4):
 original = f'zones/{zone}.zone'
 updated = update_dynamic_ip4(original, name, newip4, 3600)
 with open(original, 'w') as file:
 file.write(f'$ORIGIN {zone}.\n')
 file.write(updated)

The record update itself:

Load the zone file.
Find the record.
Replace it, or stop if the address hasn’t changed.
Update the SOA serial.
Return the zone as text.

Don’t forget the SOA serial. Without the bump, the secondary servers of your zone don’t see the change.

dnspython does the job, but its API is not my favorite cup of tea:

from datetime import date

import dns.rdataclass
import dns.rdataset
import dns.rdatatype
import dns.zone


class NoUpdate(ValueError):
 pass


def update_dynamic_ip4(zonefile, name, newip4, ttl):
 z = dns.zone.from_file(zonefile)
 new_data = dns.rdataset.from_text(dns.rdataclass.IN, dns.rdatatype.A, ttl, newip4)
 node = z.find_node(name)
 oldip4 = node.get_rdataset(dns.rdataclass.IN, dns.rdatatype.A).to_text().split()[3]
 if str(oldip4) == newip4:
 raise NoUpdate
 node.replace_rdataset(new_data)

 soa = z.find_node("@").find_rdataset(dns.rdataclass.IN, dns.rdatatype.SOA)
 soa_updated = update_soa(soa)
 z.find_node("@").replace_rdataset(soa_updated)
 return z.to_text(relativize=False)

I keep the serial as a date (%Y%m%d) and a two-digit revision. If the serial is from today, the revision increases by one; otherwise, it starts at zero for today. That allows 100 changes a day, plenty for one dynamic address.

def update_soa(soa):
 soa_parts = list(soa.to_text().split())
 version = soa_parts[-5]
 today = date.today().strftime("%Y%m%d")
 increment = 0
 if version.startswith(today):
 increment = int(version[len(today):]) + 1
 version = f'{today}{increment:02d}'
 soa_parts[-5] = version
 soa_new = " ".join(soa_parts[3:])
 ttl = soa_parts[0]
 return dns.rdataset.from_text(dns.rdataclass.IN, dns.rdatatype.SOA, ttl, soa_new)

Upload the zone file and reload the DNS server

By hand, you’d SSH to the DNS server, replace the file, and reload the server. paramiko does the same:

import paramiko


def upload_zone(localfile, remotefile, remotehost, remoteport, keyfile):
 with paramiko.SSHClient() as client:
 client.load_system_host_keys()
 client.connect(remotehost, remoteport, key_filename=keyfile)
 with client.open_sftp() as ftp:
 ftp.put(localfile, remotefile)
 client.exec_command('nsd-reload') # the reload command of your DNS server

The DNS server’s host key must already be in your known_hosts.

Expose it as a REST API

The router sends a request, and the app updates the zone. The new address comes from the request itself, and the router identifies itself with a JWT. remotefile, remotehost, remoteport, and keyfile come from the configuration.

This example creates a new secret and token each time the app starts, so the router needs the new token after a restart:

import uuid

from flask import Flask, request
from flask_jwt_extended import JWTManager, create_access_token, get_jwt_identity, jwt_required

app = Flask(__name__)
app.config["JWT_SECRET_KEY"] = str(uuid.uuid4())
jwt = JWTManager(app)
with app.app_context():
 token = create_access_token(identity={'name': name, 'zone': zone},
 expires_delta=False)

The endpoint:

@app.route('/api/update/ipv4', methods=['POST'])
@jwt_required()
def update_endpoint():
 identity = get_jwt_identity()
 name = identity['name']
 zone = identity['zone']
 if 'X-Forwarded-For' in request.headers:
 newip = request.headers['X-Forwarded-For']
 else:
 newip = request.remote_addr
 try:
 update(zone, name, newip)
 except NoUpdate:
 return {'status': 'unchanged'}
 upload_zone(f'zones/{zone}.zone', remotefile, remotehost, remoteport, keyfile)
 return {'status': 'updated'}

Most periodic calls find the address unchanged, so NoUpdate is the common path, not an error. Read X-Forwarded-For only when the app runs behind your own reverse proxy that sets it; otherwise, any client can put an arbitrary address there. Before deploying, add input validation and handling for the remaining exceptions.

Once deployed, set up the router to call the endpoint periodically.

Keep the configuration out of the code

Wed, 27 Oct 2021 23:12:58 +0200

Sometimes it’s not a design pattern but a small idea borrowed from another language that gets the job done.

The setup

A factory method picked a Style subclass based on the configuration:

class Style {
public:
 void apply(Application *app);
 static void install(Application *app, const Config &config);
protected:
 // implemented by each subclass
 virtual std::map<std::string, std::string> getMap() const = 0;
 virtual std::vector<std::string> getList() const = 0;
};

void Style::install(Application *app, const Config &config) {
 if (config.setting1 == 'a') {
 Style1().apply(app);
 } else if (config.setting1 == 'b') {
 Style2().apply(app);
 }
}

void Style::apply(Application *app) {
 // reads getMap() and getList() from the subclass
 // and applies the style to app
}

Over time, this block grew substantially. The number of possible values for setting1 skyrocketed, and other settings came into play.

The idea from Python

In Python, I’d map each value to a class and instantiate the one I need:

styles = {
 'a': Style1,
 'b': Style2,
}
styles[setting1]().apply(app)

This turned out tricky to pull off in C++. It did make me look at what the Style classes actually held: each one only returned a fixed set of properties. In effect, each class was a JSON document.

The fix: one JSON file per configuration

Instead of selecting a class, the code can select a JSON file and read the same values from it. I replaced all the Style subclasses with JSON files stored in the resources, one per configuration. install extracts the maps and lists from the JSON and processes them with the code from the original apply. Neither the inheritance nor the chain of conditions is needed anymore:

class Style {
public:
 static void install(Application *app, const Config &config);
};

void Style::install(Application *app, const Config &config) {
 // load the maps and lists from the JSON for this configuration
 // process them with the code from the original apply()
 // apply the style to app
}

When subclasses differ only in the values they return, they are configuration, not code.

SPARQL harvesting traps: a flat OPTIONAL returns the whole store, and rdflib hides the 429

Tue, 14 Sep 2021 11:35:00 +0200

Update 2026-11-08: rebuilt every repro on rdflib 6.3.2 and 7.6.0, and on Celery 5.2.7 and 5.7.0.

I wanted a resource and three hops of its neighbors from a SPARQL endpoint. This query returns every triple in the store:

CONSTRUCT { <http://example.org/root> ?p1 ?o1 . ?o1 ?p2 ?o2 . ?o2 ?p3 ?o3 . }
WHERE {
 <http://example.org/root> ?p1 ?o1 .
 OPTIONAL { ?o1 ?p2 ?o2 . }
 OPTIONAL { ?o2 ?p3 ?o3 . }
}

Move the second OPTIONAL inside the first, and you get the four triples you meant. A comment in my 2021 crawler says nested optionals are a must, otherwise the query will not finish. This post explains why and covers other ways the crawler could mistreat an endpoint without noticing.

Everything below comes from rereading that code and rebuilding each case against a local fake endpoint, under Python 3.11. It shows what the code does. I have no production logs that say what it cost anyone.

Flat `OPTIONAL`s join an unbound variable against everything

The test store holds 20,005 triples: a root with a literal label, a four-hop chain, and 20,000 unrelated triples. Times are from one run.

Query shape	Solutions	Triples returned	Oxigraph 0.5.13	rdflib 7.6.0
Flat `OPTIONAL`s	20,006	20,005	28 ms	897 ms
Nested `OPTIONAL`s	2	4	1.0 ms	3.2 ms
No `OPTIONAL`	1	3	0.1 ms	1.7 ms

The literal label triggers it. When ?o1 is that literal, the first OPTIONAL can’t match, because a literal is never a subject, so ?o2 stays unbound. The second OPTIONAL then has nothing constraining it. ?o2 ?p3 ?o3 matches every triple, and the CONSTRUCT template writes them all out. Most resources have a label, so most lookups hit this.

Removing the OPTIONALs is the wrong repair: the label disappears, as the last row shows.

I can’t rerun the 2021 endpoint, and I only tested these two engines. I believe this is what “will not finish” meant, since on a store with millions of triples each lookup becomes a full dump.

A variable you never bind does the same thing. I once committed OPTIONALs on ?dataset in a query that bound ?ds, and renamed it 18 minutes later. It’s a valid query, so nothing raises an error.

Nesting fixes the unbound variable and does nothing for hubs. A node with a million outgoing triples still returns them all, so add a LIMIT if a partial answer is acceptable.

Send one `CONSTRUCT` per named graph, then read your copy

Graph(store=SPARQLStore(url)) looks like a local graph, and every value() call on it is an HTTP request. With 50 datasets and three property reads each:

Approach	Requests to the endpoint
Read through the store-backed graph	151
One `CONSTRUCT`, then read `.graph`	1

The graph that .query(...).graph returns lives in a local Memory store, so later reads cost nothing. Scope the query with FROM <graph> and a failure or a retry costs you one named graph.

Prepared queries belong on that local copy too. A prepareQuery() object sent through SPARQLStore dies on a bare AssertionError, because the store only accepts strings. My git history shows I tried the remote version first.

`SPARQLStore` hides what the endpoint tells you

Both rdflib versions behave the same:

Input	Result
`session=requests.Session()`	Accepted, never used
No `headers=`	`User-Agent: Python-urllib/3.11`
`timeout=1`, server hangs 4 s	Waits the full 4 s
429 with `Retry-After`, default GET	`ValueError: You did something wrong formulating either the URI or your SPARQL query`
429, `method="POST"`	Returns the tuple `(429, 'HTTP Error 429: Too Many Requests', None)`, then `AttributeError` on `.graph`

The 7.6.0 connector docstring says extra keyword arguments “can be used to set up timeouts etc.”, but the code reads only params and headers from them. socket.setdefaulttimeout() does work for the whole process.

My extraction query passed session= and no headers=, and logged ValueError at debug level. An endpoint asking for a pause would have produced a debug line and a skipped graph, sent by an anonymous-looking client.

For CONSTRUCT, skip the store and use the SPARQL protocol directly:

import requests
from rdflib import Graph

class Backoff(Exception):
 def __init__(self, seconds):
 self.seconds = seconds

def construct(session, endpoint, query):
 response = session.post(
 endpoint,
 data={"query": query},
 headers={"Accept": "application/n-triples, text/turtle;q=0.9, application/rdf+xml;q=0.8"},
 timeout=(5, 120),
 )
 if response.status_code in (429, 503):
 value = response.headers.get("Retry-After", "") # may also be an HTTP date
 raise Backoff(int(value) if value.isdigit() else 60)
 response.raise_for_status()
 media_type = response.headers["Content-Type"].split(";")[0].strip()
 return Graph().parse(data=response.content, format=media_type)

session = requests.Session()
session.headers["User-Agent"] = "my-harvester/1.0 (+https://example.org/about)"

Against the fake endpoint, this returned all 200 triples and raised Backoff(120) on the 429. The same post() with a 1-second read timeout raised ReadTimeout on the hanging server. Stripping the media type parameters matters for the reason in my rdflib field notes .

Celery `chunks` runs your task as a plain function

task.chunks(args, n) takes a chunk size , and my docstring claimed it was several tasks. Ten graphs with n=4 make three tasks. Items inside a chunk run in sequence, while the chunks run in parallel, so chunks puts no cap on concurrency against the endpoint.

The worse part is retries. I raised self.retry() on the second of ten graphs:

Canvas	Outcome
`task.chunks(zip(graphs), 4)`	Chunk ends in `FAILURE`, no retry, the next two graphs never run
`group(task.s(g) for g in graphs)`	Second graph retried once, all ten finish

Each chunk is a celery.starmap task that executes task(*item). A direct call leaves request.called_directly true, and in that case retry() re-raises the exception and schedules nothing. Celery 5.2.7 and 5.7.0 agree.

Use a group, and give each endpoint its own queue with one worker slot:

@app.task(bind=True, max_retries=5)
def inspect_graph(self, endpoint, graph_iri):
 try:
 graph = construct(session, endpoint, query_for(graph_iri))
 except Backoff as exc:
 raise self.retry(exc=exc, countdown=exc.seconds)
 extract(graph)

group(inspect_graph.s(endpoint, g) for g in graphs).apply_async(queue="endpoint-a")

celery -A harvester worker -Q endpoint-a --concurrency=1

A dereference fallback is traffic too

When fetching an IRI failed, my crawler asked the dataset’s own endpoint for that resource. The idea holds up, since plenty of linked data has IRIs that don’t resolve while the triples sit in an endpoint.

Mine fell back on every failure, each one costing a three-hop CONSTRUCT. That included a robots.txt disallow on the resource, with only the endpoint’s own robots.txt still checked. Send fallbacks through the same queue, and stop when the site said no.

Check IRIs against the SPARQL grammar

Crawled IRIs ended up in <{iri}> inside query strings. rfc3987.match() rejects > and quotes, and accepts http://example.org/a\n, because its pattern ends in $, which matches before a trailing newline. URIRef(x).n3() passes that newline too, along with \r and a newline mid-string, since rdflib’s list of invalid characters has no control characters.

That newline can’t close the IRI, so this isn’t an injection. You get a malformed query and a wasted request. I added .strip() at the time, on one of the code paths. The IRIREF production is short enough to check directly:

FORBIDDEN = set('<>"{}|^`\\') | {chr(c) for c in range(0x21)}

def iri_ref(value):
 if FORBIDDEN & set(value):
 raise ValueError(f"not safe in a SPARQL IRI: {value!r}")
 return f"<{value}>"

Takeaway

Politeness here came down to two things my code got wrong: how much it asked for, and whether it could hear the answer.

Try this on your own harvester: load one unrelated triple into a test store, run each CONSTRUCT, and fail the test if that triple comes back.

Flask @cached decorator traps: POST gets the cached GET, no-store is stored, and X-Last-Modified: None

Fri, 19 Oct 2018 21:26:00 +0200

Update 2026-11-08: I rebuilt every repro on Flask 2.2.3 with Werkzeug 2.2.3 and Flask-Caching 2.0.2, and on Flask 3.1.3 with Werkzeug 3.1.9 and Flask-Caching 2.5.1, under Python 3.11.

A Flask view decorator I kept for years failed in a way that only showed up when a POST got the body of an earlier GET. The view never ran, and the client got a 200:

GET /item 200 "GET ran" X-Cache: MISS
POST /item 200 "GET ran" X-Cache: HIT from Server

The cache key was request.full_path, so any two requests with the same URL shared an entry, regardless of method, user, or Accept header. To fix this, let only anonymous GET and HEAD requests touch the shared entry, build the key from the response inputs, and leave the HTTP headers to Werkzeug.

The decorator entered a 2018 project with a docstring pointing at a Code Review question and Jake Archibald’s caching article . I rebuilt each case from rereading it. My routes were all GET-only and public, so the POST and private cases never fired, and I have no logs showing what the others cost anyone.

A decorator with one shared entry per representation

from functools import wraps
from urllib.parse import urlencode

from flask import make_response, request
from flask_caching import Cache

cache = Cache()


def cached(params=(), offers=("application/json",), server_timeout=300):
 """Server-cache a public view. Declare the query parameters it reads."""

 def decorator(view):
 @wraps(view)
 def wrapper(*args, **kwargs):
 if (
 request.method not in ("GET", "HEAD")
 or request.authorization
 or request.cookies
 ):
 return view(*args, **kwargs)

 query = urlencode([(p, request.args.getlist(p)) for p in params], doseq=True)
 media_type = request.accept_mimetypes.best_match(offers, offers[0])
 key = f"view/{request.path}?{query}#{media_type}"

 response = cache.get(key)
 if response is None:
 response = make_response(view(*args, **kwargs))
 if response.status_code != 200:
 return response
 response.add_etag()
 response.cache_control.public = True
 response.cache_control.no_cache = True
 response.vary.add("Accept")
 cache.set(key, response, timeout=server_timeout)
 return response.make_conditional(request)

 return wrapper

 return decorator

Both decorators show the same probes, identical on both version sets:

Probe	Old decorator	Fixed decorator
POST after GET	Cached GET body, view skipped	View runs
Same URL, cookies for two users	Second user sees the first user’s page	Each sees their own
Anonymous request after a logged-in one	200 with the logged-in body	401
JSON, then `Accept: text/turtle`	JSON for both	Turtle for the second
100 requests with `Cache-Control: no-cache`	100 view runs	0 after the first
100 requests with a junk `?x=N`	100 view runs, 100 entries	0 after the first, 1 entry
`If-None-Match` with the `W/` form of the tag	200	304
304 headers	No `ETag`, `X-Last-Modified: None`	`ETag`, `Cache-Control`, `Vary`

Two caveats. Bypassing any cookie is blunt, since one analytics cookie turns the cache off for that browser, so narrow it to your session cookie. And params must list every argument the view reads, or two different URLs collide again. For per-user caching, put the user ID in the key and send private.

`no-store` and `private` don’t bind your own cache

The bare @cached() sent Cache-Control: no-store and still called cache.set:

GET /nostore 200 Cache-Control: no-store X-Cache: MISS
GET /nostore 200 Cache-Control: no-store X-Cache: HIT from Server

no-store and private are instructions to the browser and to proxies. An entry your application writes to Redis is outside their reach. The arguments cacheable=False and client_only=True read like server switches and only picked a header. With private, the second user received hello alice from the shared entry.

Decorator order makes it worse. With @cached above a login check, a hit never reaches the check: after one logged-in request, an anonymous client got a 200 with account of alice.

The method guard covered writes only

request.method in ["GET", "HEAD"] protected cache.set. The cache.get above it had no guard. A POST on a cold cache runs the view, so a test that only sends POST passes.

Flask-Caching’s @cache.cached(timeout=60) did the same on a GET and POST route, in both versions. Add unless=lambda: request.method != "GET" there.

Any client could pick the miss

if response is not None and request.headers.get("Cache-Control", "") != "no-cache":

The comment says “respect the hard-refresh”. It also hands every client a way to run the expensive view on demand: 100 requests, 100 runs. The comparison is exact, so 100 requests with no-cache, no-store or max-age=0 ran the view once. A junk query parameter does the same and adds an entry per request.

Your application’s cache is part of the origin, and no RFC asks it to obey a client’s request header. Invalidate from the server with cache.delete. The price: a hard refresh shows the cached body until server_timeout passes.

One typo, one bytes object

The 304 branch read a header that was never written:

response.headers.add("X-Last-Modified", str(now))
headers["X-Last-Modified"] = response.headers.get("X-LastModified") # None

Werkzeug turns None into a string, so every 304 carried X-Last-Modified: None and no error. It also lacked the ETag that RFC 9110 requires on a 304.

The tag was binascii.hexlify(os.urandom(4)), which returns bytes:

Werkzeug	`ETag` value
2.2.3 to 2.3.8	`a79d7c42`
3.0.0 and 3.1.9	`b'77155122'`

Neither has the double quotes that the grammar requires. Both round-trip through If-None-Match unchanged, so the change in Werkzeug 3.0 raises no error. The comparison was a literal ==, where If-None-Match calls for weak comparison , and a random tag changes every time the entry refills even if the body doesn’t. add_etag() hashes the body and make_conditional() does the comparison.

`no-cache` means revalidate

The old comments had this right: “no-cache doesn’t mean ‘don’t cache’” and “must-revalidate doesn’t mean ‘must revalidate’”.

Directive	Stored?	Reused without asking the server?
`no-store`	No	No
`no-cache`	Yes	No
`must-revalidate, max-age=900`	Yes	For 900 s, then it must revalidate

Then every call site passed must_revalidate=True, client_timeout=900, which sent no-cache, public, max-age=900. An unqualified no-cache forces validation on every reuse, so the 900 seconds bought nothing. I had the explanation in the file and still passed both.

Takeaway

A server-side entry goes to everyone whose request maps to its key, so list what the response depends on and put each item in the key or refuse to cache.

Try it on your own cached views: GET a URL, POST to it, then GET it with another user’s cookie, and fail the test if any two bodies match.

Lessons learned

Offline login in an iOS or Android app: design it as offline unlock

What an offline unlock can and cannot prove

A state machine where a stale device never prompts

Wrap keys; never encrypt data with the password

Why the password path needs the device too

A six-digit PIN is a million candidates

The biometric prompt must unlock a key

Android: pass a CryptoObject

iOS: a Secure Enclave key and ECIES

Data Protection classes that fail after lock

Offline expiry needs a clock that counts sleep

Documents: chunked AEAD and plaintext that outlives the app

Lock, logout, and Keychain items that survive uninstall

Autofill is offline unlock under another name

Verify who is asking

Users without a password

Caveats

Test the failure paths

Takeaway

Backing up ~/.claude and ~/.codex without archiving your AI logins

Where AI client state lives on macOS and Linux

What to back up and what to leave out

Why copying the whole folder is the wrong default

Dead ends worth skipping

Restore into a staging profile

Verify the backup set

If malware may already have run

Caveats

Swift Testing withKnownIssue can hide snapshot failures

Restrict which issues count as known

A mutation must change the rendered output

Keep the exception smaller than the assertion

A shared Swift timeout left an earlier task without a deadline

One timer could not cover overlapping fetches

Put the deadline beside the operation

Prove that the waiter needs this deadline

A cancellation deadline is not a hard timeout

SwiftUI large title disappears after tapping the selected tab again

The title still existed

A local modifier changed later navigation bars

Why the one-line hotfix was reversed

Verify the interaction, not just the label

Xcode Archive can ship the wrong display name despite correct source settings

Source checks missed the artifact

Check the processed plist during the build

Prove that wrong builds fail

A latest-row query cannot tell you whether every device is current

Reproduce the lost device with synthetic data

State the measurement contract

Keep unknown devices out of the healthy bucket

Swift Testing: .serialized did not stop another suite deleting test files

A unique child directory still has a shared parent

Make ordinary test construction inert

Prove the constructor contract, then repeat the broader target

A CVE cross-reference is not an advisory entry

How the wrong relationship gets created

Filter by document role before matching identifiers

A prefix guard has a document contract

An offline cache TTL needs a clock-rollback policy

A negative age passed the upper-bound check

Retain the latest observed time

Refusing access did not mean deleting the cache

Test the rollback twice

A high-water mark cannot establish time that was never observed

iOS jailbreak protection in 2026: detection is a risk signal, not a security boundary

The short version

How a working check silently stopped working

When OWASP says to implement jailbreak detection

The current techniques are rootless and designed to hide

The response matters more than the Boolean

App Attest authenticates app instances, not clean devices

Test implementation, behavior, and bypass resistance

A reviewable 2026 “lege artis” baseline

Takeaway

References

Adapt security skills to your project's risks: from Go to OWASP LLM and iOS

Start with a review method

Adapt OWASP categories to the application

Repeat the adaptation for iOS

Android: pass a `CryptoObject`