Celery self.retry(5) raises TypeError, and except Exception swallows the retry

Update 2026-11-08: rebuilt every case on Celery 5.2.7 and 5.7.0.

My crawler called Celery’s retry three different ways, and only one did what I meant:

TypeError: 'int' object is not iterable

Even the correct call can go wrong. retry() publishes the new message and then raises celery.exceptions.Retry, an ordinary Exception. An except Exception around it marks the work as failed while the retry is already on its way. Pass countdown= by keyword and let TaskPredicate pass through your broad handlers.

I found these by rereading old code and rebuilt each case on a real worker with a private Redis as broker and result backend, on Python 3.11. Both Celery versions behaved the same. I have no production logs from that time, so this shows what the code does and leaves open what it cost.

Re-raise TaskPredicate before except Exception #

from celery.exceptions import TaskPredicate


@app.task(bind=True, max_retries=2)
def crawl(self, url):
    try:
        fetch(self, url)  # calls self.retry(exc=err, countdown=1) on a network error
    except TaskPredicate:
        raise
    except Exception:
        mark_failed(url)
        return "gave up"
    mark_ok(url)
    return "done"

TaskPredicate is the base class of Retry, Ignore and Reject. With this handler, result.get() returned 'done' about 1 s later, after the retry had run.

My handler already knew about one of the three:

except Exception as err:
    if isinstance(err, Ignore):
        raise

Retry needed the same treatment and didn’t get it.

Why the three calls differ #

The signature on both versions is:

retry(self, args=None, kwargs=None, exc=None, throw=True, eta=None, countdown=None, max_retries=None, **options)

The first positional parameter is the replacement args for the retried task, so 5 reaches tuple(args) and raises before anything is published.

timeout isn’t a parameter, so it lands in **options and travels with the message. Spying on kombu’s Producer.publish showed timeout=5 arriving there, where it limits how long publishing may take. The delay falls back to default_retry_delay, three minutes by default . Nothing raises. The only hint is the worker log:

Task tasks.crawl[62d0a257-...] retry: Retry in 180s

What except Exception does to a retry #

Remove the except TaskPredicate branch from the task above and let the first attempt hit a network error:

The last row surprised me. On the Redis backend, _store_result ignores any write once the stored state is SUCCESS, so the retry’s own return value is dropped. Anyone reading the result sees the attempt that gave up.

Code review won’t catch this reliably, because the placement decides it. A retry inside the try body, even several calls deep, is caught. A retry in a sibling except clause is safe:

try:
    fetch(task, url)  # a retry raised in here hits except Exception
except RateLimited as err:
    task.retry(countdown=err.delay)  # sibling handlers don't see this one
except Exception:
    mark_failed(url)

My code had both forms in the same function, so some retries worked. Writing raise self.retry(...) changes nothing, and a bare except: catches Retry too. In an older module, a bare except: also caught the TypeError from self.retry(5), which recorded a rate-limited link as a broken one.

A dedupe mark written first makes the retry skip itself #

@app.task(bind=True, max_retries=2)
def crawl(self, url):
    if redis_client.sadd("seen", url) == 0:
        return "skipped"
    ...

attempt 0 sadd=1
attempt 1 sadd=0

The retry sees its own mark, returns 'skipped', and the URL is never fetched. Combine that with the swallowed Retry and the URL stays failed for good: in the rebuild, the final status was failed and the result was 'gave up'.

Moving the SADD after the work fixed it: the retry returned 'fetched'. Two concurrent tasks can then fetch the same URL, so the work has to be safe to repeat.

When the retry condition is just an exception type, autoretry_for=(Flaky,), retry_kwargs={"countdown": 1, "max_retries": 2} saves calling retry yourself. The exception has to leave the task body, so a broad handler inside it defeats this as well.

Time limits have a similar problem on the gevent pool, in the opposite direction: Timeout isn’t an Exception .

Takeaway #

self.retry() is a raise whose side effect has already happened, so any handler between it and the worker can make your bookkeeping disagree with the queue.

Run grep -rn --include='*.py' '\.retry(' . on your tasks and check each hit for keyword arguments and an enclosing except Exception.