A failed CUDA build over SSH still launched the old executable

A remote command returned a CUDA compilation error followed by a normal simulation startup banner:

error: calling a __device__ function("__sinf") from a __host__ __device__ function("inflowFluctuation") is not allowed

The simulation started, but it ran an older executable. Semicolons let the launch proceed after nvcc failed. Correcting the source and rebuilding recovered the test; guarding the build-to-launch transition would prevent that misleading result.

The Claude Code and OpenPBS setup explains the local-agent/remote-command boundary. This happened during an early SSH-driven test, before we moved the long-running work into scheduled jobs. The shell mistake also applies inside a PBS script.

Diagnostics did not control execution #

The command had this structure, with project paths removed:

nvcc -o solver solver.cu > build.log 2>&1;
grep -A3 error build.log | head;
nohup ./solver case.cfg > run.log 2>&1 < /dev/null &

The build wrote to an executable path that already existed. Compilation failed, leaving a runnable earlier build. Printing matching diagnostics didn’t stop the next command, and backgrounding the program didn’t establish whether the requested source had compiled.

The failed-build result included an ordinary banner. After replacing __sinf with sinf in the host/device function, the rebuilt program’s banner contained the newly added inflowTurb=0.08 parameter and subsequent progress lines. The earlier banner lacked that field. That difference provided evidence of which code path actually ran.

The transcript shows this operational recovery. Its retry still used semicolon-separated build and launch commands, so it did not repair the fail-open shell structure.

Make compiler status a launch prerequisite #

For a future build, preserve the compiler’s status explicitly. This generalized Bash pattern builds to a temporary path and only publishes it after success:

#!/bin/bash
set -euo pipefail
cd "${PROJECT_DIR:?Set PROJECT_DIR to the remote project directory}"

candidate=$(mktemp ./solver.build.XXXXXX)
trap 'rm -f "$candidate"' EXIT
if nvcc -o "$candidate" solver.cu > build.log 2>&1; then
    mv -f "$candidate" ./solver
else
    cat build.log >&2
    exit 1
fi
exec ./solver case.cfg

This is a recommended guard, not code installed during the incident. The named source and configuration must exist in the remote project. Use a single build owner: concurrent publishers to ./solver require a separate ownership or versioned-artifact design.

When running remotely, send the whole script through SSH so its directory selection and status checks happen in the same remote shell. Avoid embedding local path expansions accidentally. For batch work, run the payload in the allocated job rather than adding another unscheduled nohup launch.

Reproduce the shell failure without CUDA #

On a local machine with a POSIX shell, this harmless fixture leaves an existing command available after a simulated build failure:

sh -c 'solver() { printf "old executable ran\n"; }; false; solver'

It prints old executable ran and exits successfully. The guarded form prevents execution:

sh -c 'solver() { printf "old executable ran\n"; }; false && solver'

That form prints nothing and returns a failure status. For a real compiler test, deliberately introduce an error and verify that no new run starts. Also log a build identifier or other deliberate runtime identity; a healthy-looking startup line only proves that some executable ran.

Check your next remote build command for the exact status dependency between compilation and launch, especially where diagnostic pipelines sit between them.