lundi 15 juin 2015

Worker management for durable queue in Redis

It is a well-known pattern to use LPUSH and BRPOPLPUSH (http://ift.tt/IQBE7K) to implement a durable queue in Redis. However, in order to scale up, the design needs to cater for multiple workers/consumers that BRPOPLPUSH from the main task queue.

So the norm seems to be that for each worker there is a separate processing_queue that records the task a specific worker is working on, such that the worker keeps track of what is left to do in case that it exits during processing.

I have two questions regarding this processing_queue:

  1. Is it correct to reason that there will be at most one item/task at any time in a worker's processing_queue? I assume that a worker starts by checking any left-over task in its own processing_queue before BRPOPLPUSH the main task queue. If so, we can use any of RPOP, LPOP, LREM to remove a task once the worker finishes processing (or it can simply delete the list). We can even use a set instead of a list. Is there any reason why so many people choose to use LREM but nothing else?

  2. I have seen many people recommend to identify the individual processing_queue using the process ID of the corresponding worker. But what happens when the old worker exits and a new one gets spawned with a (most likely) new process ID. How does the new worker look up its predecessor's processing_queue to finish a possible left-over task? I plan to use Supervisor to manage my worker processes, if that makes a difference.

Aucun commentaire:

Enregistrer un commentaire