mardi 7 juillet 2020

Parallel processing with RabbitMQ

Anyone have any suggestions or resources that I can read up on how to go about scaling parallel tasks with fair processing and consume using rabbitMQ? If there are over 200 users with 2000+ tasks per user and want to make sure the tasks get processed and consumed equally while also having ability for users to get their tasks done in parallel. Each tasks take about 1s.

Below is how the app is currently handling that. But I feel like there is a much better way.

So quick overview of what the app is currently doing. User can create a lists of todo tasks. Each tasks takes about 1.5s max. We have parallel processing of these tasks in place using rabbitMQ. There are total of 6 workers for each server with prefetch value of 1. There are about 200 queues, call it task_queue_1, etc. So each task_queue is dedicated to some range of todo tasks.

For example task_queue_1 is dedicated to first 10 items from task, and so on. Each of the items, starting from task 1-10, get priority assigned from 10 to 1 to ensure first task gets process first, and so on.

The queues itself also have priority. Workers will pull from task_queue_1 first but if there are messages in task_queue_2 waiting for more than one minute the it will have bigger priority, and so on. This is handled through stack and ordering.

This has been doing fine but I realized it's not exactly scalable. If I have bunch of users submitting lists with over 2k tasks, some users will be forced to wait to get their tasks processed because workers are busy processing tasks from queue_2, queue_3, etc due to wait time longer than 1 minute.

Aucun commentaire:

Enregistrer un commentaire