A second launch of a resumable worker: 'OSError: [Errno 39] Directory not empty'

A resumable remote dataset driver failed while cleaning an unfinished task directory:

OSError: [Errno 39] Directory not empty

Another worker was still writing there. The second launch treated an unfinished task as safe to restart, even though the first process already owned it. Checking the surviving worker and validating its affected output recovered that work unit; no locking fix was installed during the session.

This was an early detached-run failure in the Claude Code, SSH, and OpenPBS workflow . Scheduler-managed execution later improved job tracking, but an application still needs to prevent conflicting owners.

The waiting wrapper had not disappeared #

The first launch created a detached wrapper that waited for a pilot, then became the production driver. Its structure was:

until grep -q "finished" pilot.log; do sleep 30; done
exec python -u run_dataset.py dataset >> production.log 2>&1

Before the transition, a search for the production process would not find that driver. After exec, a search for the wrapper’s old command name would no longer identify it. Neither snapshot alone described the complete lifecycle.

A later check did not establish that the waiting launch was gone. Another detached production command started. The resulting log contained two identical pending-work counts followed by a traceback from shutil.rmtree(gdir). Both drivers had selected the same unfinished task, and one attempted cleanup while the other’s solver wrote files.

The resume logic used completion markers to skip finished groups. It offered no exclusive claim on an unfinished directory.

Recovery required checking the survivor #

The follow-up inspected running processes, restored the affected task’s configuration, and checked the surviving computation. Its original log file had been lost during cleanup. A later driver result reported that the task was accepted.

A broad grep for success also matched text in the traceback. Tightening the check to a result-line pattern distinguished the actual accepted task from an error message that happened to contain matching characters.

That evidence supports recovery of the affected work unit. It does not prove that no intermediate data was lost or that the session did not add a lock. Eventual production completion checked in a later session also cannot establish that overlapping launches became safe.

Claim ownership before destructive resume #

For a single-host driver using a filesystem with working advisory locks, a whole-run lock is one possible improvement. On a system with util-linux flock, run from the project directory:

flock -n .production.lock python -u run_dataset.py dataset

Every launcher must use the same stable lock path. Do not delete and recreate the lock file while work is running. The payload named here is the application’s driver, not a standard tool.

This guard is a new recommendation. A small local check can verify contention without running a dataset:

import fcntl
import tempfile

with tempfile.TemporaryDirectory() as directory:
    with open(directory + "/worker.lock", "w") as first:
        fcntl.flock(first, fcntl.LOCK_EX | fcntl.LOCK_NB)
        with open(directory + "/worker.lock", "w") as second:
            try:
                fcntl.flock(second, fcntl.LOCK_EX | fcntl.LOCK_NB)
            except BlockingIOError:
                print("second owner rejected")
            else:
                raise AssertionError("overlapping owner accepted")

Cluster filesystems and multi-node workers need an ownership mechanism tested under their actual locking semantics. Per-task claims can allow parallelism that a whole-run lock forbids. Stable PBS shard assignment solves a different problem: preserving each worker’s task set across starts and restarts. It cannot exclude two copies of the same shard.

Before retrying unfinished work, establish whether a live process already owns it. A missing completion marker only says that completion has not been recorded.