The task.request.retries
attribute contains the number of tries so far,
so you can use this to implement exponential back-off:
from celery.task import task
@task(bind=True, max_retries=3)
def update_status(self, auth, status):
try:
Twitter(auth).update_status(status)
except Twitter.WhaleFail as exc:
raise self.retry(exc=exc, countdown=2 ** self.request.retries)
To prevent a Thundering Herd Problem, you may consider adding a random jitter to your exponential backoff:
import random
self.retry(exc=exc, countdown=int(random.uniform(2, 4) ** self.request.retries))