Scaling celeryworker service each with different queue #3427
-
|
I am using cookiecutter-django with docker and I want to scale my celeryworkers each with different queue. But could not find a way. If I use "--scale celeryworker=4" command it creates 4 different workers with same settings. But I need to set each one different queue. Is there anybody have the similar issue or have any idea about how to solve this problem? Any help would be very appreciated. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
Hello @koztay , You might try looking at these articles Basically you want to create Celery routes which designates a task to a certain queue
Next, you can scale up queues as needed Scaling queues
|
Beta Was this translation helpful? Give feedback.
Hello @koztay ,
You might try looking at these articles
article 1 & article 2
Basically you want to create Celery routes which designates a task to a certain queue
CELERY_ROUTES = { 'core.tasks.too_long_task': {'queue': 'too_long_queue'}, 'core.tasks.quick_task': {'queue': 'quick_queue'},}Next, you can scale up queues as needed
Scaling queues
celery --app=proj_name worker -Q too_long_queue -c 2celery --app=proj_name worker -Q quick_queue -c 2