Task Queue
A task queue manages background work by placing tasks in a queue and processing them asynchronously with one or more workers. Producers (your application) add tasks, and workers consume and execute them independently.
How Task Queues Work
- Producer creates a task with a payload and adds it to the queue
- Queue stores the task until a worker is available
- Worker picks up the task, executes it, and reports the result
- Result is stored and made available to the producer
Why Use a Task Queue?
- Decouple processing: Your web server responds immediately while heavy work happens in the background
- Reliability: Tasks survive server restarts and crashes — the queue persists them
- Scalability: Add more workers to handle increased load without changing your application code
- Rate limiting: Control how fast tasks are processed to avoid overwhelming external services
Common Use Cases
- Sending emails and notifications
- Processing uploaded files (images, videos, PDFs)
- Syncing data between systems
- Running scheduled reports
- Executing webhook callbacks
- AI/ML inference jobs
Task Queues vs Message Queues
Though often used interchangeably, task queues and message queues serve different purposes:
- Message queues (RabbitMQ, SQS) focus on message delivery between services
- Task queues (AsyncQueue, Celery, BullMQ) focus on executing work and tracking results
Task queues include built-in features like retries, scheduling, priority, and result storage that message queues lack by default.