Turns out it cant be done at queue level for multiple workers. IT can be done at queue level for 1 worker. Or at queue level for each worker. So if u say 10 jobs/ minute on 5 workers. Your workers will process upto 50 jobs per minute collectively. So to have only 10 jobs … Read more
As far as I know, celeryd is just an old name for the celery worker command. celerybeat is a scheduler that sends predefined tasks to a celery worker at a given time. You only need to bother with this if you want to run a task on a schedule. For example, if you had a … Read more
To mark a task as failed without raising an exception, update the task state to FAILURE and then raise an Ignore exception, because returning any value will record the task as successful, an example: from celery import Celery, states from celery.exceptions import Ignore app = Celery(‘tasks’, broker=”amqp://guest@localhost//”) @app.task(bind=True) def run_simulation(self): if some_condition: # manually update … Read more
It works using AsyncResult. (see this answer) So first create the task: from cel.tasks import add res = add.delay(3,4) print(res.status) # ‘SUCCESS’ print(res.id) # ‘432890aa-4f02-437d-aaca-1999b70efe8d’ Then start another python shell: from celery.result import AsyncResult from cel.tasks import app res = AsyncResult(‘432890aa-4f02-437d-aaca-1999b70efe8d’,app=app) print(res.state) # ‘SUCCESS’ print(res.get()) # 7
You can set task time limits (hard and/or soft) either while defining a task or while calling. from celery.exceptions import SoftTimeLimitExceeded @celery.task(time_limit=20) def mytask(): try: return do_work() except SoftTimeLimitExceeded: cleanup_in_a_hurry() or mytask.apply_async(args=[], kwargs={}, time_limit=30, soft_time_limit=10)