Claude Code over SSH: submit long-running work to OpenPBS

·4 mins

Claude Code can prepare a computation on your laptop and operate a remote cluster through ordinary SSH commands. In this setup, Claude Code ran locally on macOS; source uploads, compilation, queue submissions, and checks ran on a Linux server. OpenPBS handled the long-running computation after submission, so the laptop didn’t need to keep an SSH session open.

The operating rule was explicit: work estimated at one hour or more belonged in PBS. That threshold was a local policy, not an OpenPBS default. Use your site’s rules, including its restrictions on short tests and compilation on login nodes.

Establish the SSH connection yourself

Configure authentication and verify the server’s host key before asking an agent to connect. For example, in your local ~/.ssh/config:

Host compute-login
    HostName 192.168.0.2
    User researcher
    BatchMode yes
    ConnectTimeout 10

The address and account are placeholders. With your site’s actual values and authentication configured, run this from the laptop:

ssh compute-login 'hostname; pwd; command -v qsub; command -v python3'

BatchMode makes missing authentication fail instead of waiting for a password. It does not approve the connection or replace host-key verification. Give Claude permission for the intended remote actions through your normal tool-approval workflow; never paste a private key into the conversation.

Make the execution boundary explicit

A useful project instruction is:

Claude runs on the laptop. Use SSH for remote operations.
Keep this project's remote files in ~/batch-demo.
Submit tasks estimated at one hour or more through OpenPBS.
Follow site policy for shorter tests; do not bypass the scheduler.
Record the job ID after submission. Check logs and expected outputs before reporting completion. Ask before cancellation or deletion.

Save this with the project’s agent instructions. A prose rule still needs checking: our early sessions launched substantial work directly before the PBS rule was captured. Also, having a GPU on a login host does not grant permission to use it.

Submit a small smoke test first

These are generalized setup examples, not a copy of the original scientific workload. Create run.py locally:

import socket
import sys

print(f"worker host: {socket.gethostname()}")
print(f"python: {sys.executable}")
print("smoke test complete")

Create run.pbs beside it:

#!/bin/bash
#PBS -N agent-smoke
#PBS -l select=1:ncpus=1
#PBS -l walltime=00:05:00
#PBS -j oe

set -euo pipefail
cd "${PBS_O_WORKDIR:?PBS submission directory missing}"
exec "$PWD/.venv/bin/python" -u run.py

From the laptop, prepare and submit it:

ssh compute-login 'mkdir -p ~/batch-demo && cd ~/batch-demo && python3 -m venv .venv'
scp run.py run.pbs compute-login:batch-demo/
ssh compute-login 'cd ~/batch-demo && qsub run.pbs'

Record the exact job ID that qsub returns. The batch script selects its directory and interpreter without depending on an activated interactive shell. Install real dependencies according to site policy and check that the environment and project directory are accessible on the execution node. Shared paths and compatible Python installations are prerequisites, not promises made by SSH.

For GPU work, request the site’s documented GPU resources. The original array used select=1:ncpus=2:ngpus=1 per element. GPU visibility variables and binding depend on cluster configuration; inspect them inside an allocation rather than choosing an arbitrary device from the login host.

Follow the job through its outputs

On the login host, use qstat -f with the recorded ID for detailed status. Read the output file named in that status. Sites retaining finished-job history may also support qstat -xf for accounting. The upstream qsub manual documents submission options and PBS_O_WORKDIR; the qstat manual covers status queries.

An empty queue is insufficient evidence of success. For this smoke test, check the reported worker and Python path and the completion line. For production, also verify expected task outcomes and output contents.

The subsequent failures were about execution ownership and verification:

Before handing over a long run, submit the smoke test and check its output on the execution node’s actual environment.