Celery unable to use redis
Try to install Redis as in your virtual environment as well: pip install Redis
django-celery
Celery auto reload on ANY changes
Celery –autoreload doesn’t work and it is deprecated. Since you are using django, you can write a management command for that. Django has autoreload utility which is used by runserver to restart WSGI server when code changes. The same functionality can be used to reload celery workers. Create a seperate management command called celery. Write … Read more
How to send periodic tasks to specific queue in Celery
Periodic tasks are sent to queues by celery beat where you can do everything you do with the Celery API. Here is the list of configurations that comes with celery beat: https://celery.readthedocs.org/en/latest/userguide/periodic-tasks.html#available-fields In your case: CELERYBEAT_SCHEDULE = { ‘installer_recalc_hour’: { ‘task’: ‘stats.installer.tasks.recalc_last_hour’, ‘schedule’: 15, # every 15 sec for test ‘options’: {‘queue’ : ‘celery_periodic’}, # … Read more
Why would running scheduled tasks with Celery be preferable over crontab?
I’ve been using cron for a production website, and have switched to celery on a current project. I’m far more into celery than cron, here is why: Celery + Celerybeat has finer granularity than cron. Cron cannot run more than once a minute, while celery can (I have a task run every 90 seconds which … Read more
using class methods as celery tasks
Celery has experimental support for using methods as tasks since version 3.0. The documentation for this is in celery.contrib.methods, and also mentions some caveats you should be aware of: https://docs.celeryproject.org/en/3.1/reference/celery.contrib.methods.html Be aware: support for contrib.methods removed from Celery since 4.0
rabbitmq-server fails to start after hostname has changed for first time
Remove the old installation of RabbitMQ to fix this problem. Here are steps to reinstall RabbitMQ. These commands are run as the root user: Stop RabbitMQ: rabbitmqctl stop Change /etc/hosts Change /etc/hostname Uninstall old RabbitMQ: dpkg -P rabbitmq-server Remove RabbitMQ’s database: rm -rf /var/lib/rabbitmq Find erlang’s process that is running rabbit: ps ax | grep … Read more
Detect whether Celery is Available/Running
Here’s the code I’ve been using. celery.task.control.Inspect.stats() returns a dict containing lots of details about the currently available workers, None if there are no workers running, or raises an IOError if it can’t connect to the message broker. I’m using RabbitMQ – it’s possible that other messaging systems might behave slightly differently. This worked in … Read more
Difference between different ways to create celery task
Don’t use #2 unless you are using celery v3. If you are using celery v4, use #1. Use #3 in instances where you are writing a reusable library or django app. For example, if you are writing an open source set of tasks that allow you to manage aws ec2 instances using celery, you would … Read more
Django Celery Logging Best Practice
When your logger initialized in the beginning of “another module” it links to another logger. Which handle your messages. It can be root logger, or usually I see in Django projects – logger with name ”. Best way here, is overriding your logging config: LOGGING = { ‘version’: 1, ‘disable_existing_loggers’: True, ‘formatters’: { ‘simple’: { … Read more
Retry Celery tasks with exponential back off
The task.request.retries attribute contains the number of tries so far, so you can use this to implement exponential back-off: from celery.task import task @task(bind=True, max_retries=3) def update_status(self, auth, status): try: Twitter(auth).update_status(status) except Twitter.WhaleFail as exc: raise self.retry(exc=exc, countdown=2 ** self.request.retries) To prevent a Thundering Herd Problem, you may consider adding a random jitter to your … Read more