HTTP 500 that still commits: never retry an ambiguous write

·5 mins

We ran the client’s real retry loop against a fake server and counted the writes on the wire, before and after the fix:

before:  push PUT 500 -> 200   writes_on_wire=2   -> reported ok
after:   push PUT 500 -> 200   writes_on_wire=1   -> FlexiBeeError(maybe_committed=True)

The fake server played what the real API does now and then: it answers a PUT with 500 Internal Server Error after it has already saved the invoice. The client retried; the second PUT came back 200 OK, and the batch reported success. On a real API, that invoice is on the books twice. If your API can return a 5xx after committing, a retry that succeeds is the worst outcome you can get: you end up with a duplicate and nothing in the logs.

Setup

The API was the REST interface of ABRA Flexi, an ERP, called from Python through urllib and http.client. It returns 500 now and then after the write has gone through. That much was known; one client already looked the record up by document number after a failure.

Five hand-written clients for that API were merged into one transport layer with a shared RetryPolicy. An AI coding agent did the merge, two commits after writing an ADR that warned about duplicate writes.

What went wrong

The policy didn’t know the method

RetryPolicy decided from the response alone, so it retried a 500 on a PUT the same way it retried one on a GET. The test for that case fed in three 500s and checked only the flag on the final error. It pinned the number of retries and never asked whether a write should be retried in the first place.

The safety nets sat outside the loop

The first fix made the method a required argument and turned off retries of ambiguous writes by default. Two callers opted back in, because each had a safety net. In outline:

# MCP server: an idempotency key per tool call
def write_tool(key, payload):
    if idem_seen(key):            # checked once, before the first attempt
        return cached_result(key)
    result = request("PUT", url, payload)   # retries happen in here
    idem_record(key, result)      # recorded only after success
    return result

# webhook worker: look the record up when the write fails
def upsert(doc):
    try:
        return request("PUT", url, doc)     # retries happen in here
    except ApiError:
        if find_by_number(doc.number):
            return "committed-via-verify"
        raise

A review five days later, by a different model, caught it: “_idem_seen runs before this function, and _idem_record runs only after success, so neither prevents this retry from duplicating a write.” Likewise, the GET check runs only when request() raises, and a 500 followed by a successful retry never raises. Both checks were correct; the retry loop inside request() simply never consulted them.

Fix

Writes are never retried on an ambiguous outcome, and no caller can opt back in. The error goes up to the caller and says the write may have committed. The policy, trimmed to the part that matters:

WRITE_METHODS = {"POST", "PUT", "PATCH", "DELETE"}

class RetryPolicy:
    def __init__(self, method: str):   # required: a default would let the next client forget it
        self._is_write = method.upper() in WRITE_METHODS

    def next_retry(self, status: int, body: str):
        if is_rate_limited(status, body):        # the server refused: nothing committed
            return self._backoff("rate-limit")
        if 500 <= status < 600 and not self._is_write:
            return self._backoff("transient-5xx")
        return None                              # a 5xx on a write: raise, maybe_committed=True

    def next_retry_for_network_error(self):
        # no response at all: the server may have processed it before the connection died
        return None if self._is_write else self._backoff("network")

The new test counts requests, because a retry that succeeds looks exactly like a first attempt that did:

def test_500_on_a_write_is_never_retried(self):
    fake = FakeConn([FakeResponse(500, b"boom"), FakeResponse(200, b"{}")])
    with patch.object(client, "_get_conn", return_value=fake):
        with self.assertRaises(ApiError) as cm:
            client.put("invoice.json", {"invoice": [{}]})
    self.assertEqual(len(fake.requests_made), 1)
    self.assertTrue(cm.exception.maybe_committed)

Reads still retry, and they make up most of the traffic in a batch. A write that comes back maybe_committed is checked by whoever started it, before anything resends it.

Why

The policy asked, “may I retry a 5xx?” For a write, the question is “may I resend something whose outcome is unknown?” Two outcomes leave a write ambiguous: a 5xx, and no response at all. A 429, or a 400 rejecting a stale user role, is the server refusing the write; nothing committed, so a retry can’t duplicate anything.

The HTTP method doesn’t settle it either. RFC 9110 calls PUT idempotent, and urllib3’s default allowed_methods includes PUT, so a status_forcelist=[500, 502, 503] resends it. This API’s PUT to a collection creates a record each time.

If you do want writes retried automatically, the check has to run between the failure and the resend: before every attempt, look the record up or send a key the server itself deduplicates on. A check before the first call or after the last one can’t see the retries in between.

Takeaway

A 500 on a write means you don’t know what happened, so pass that up the stack instead of resending. Test it by counting requests on the wire, since the final result looks the same either way.

Find the retry loop in your own HTTP client and check whether it knows the method it is retrying.