It is a well-known pattern to use LPUSH and BRPOPLPUSH (http://ift.tt/IQBE7K) to implement a durable queue in Redis. However, in order to scale up, the design needs to cater for multiple workers/consumers that BRPOPLPUSH from the main task queue.
So the norm seems to be that for each worker there is a separate processing_queue that records the task a specific worker is working on, such that the worker keeps track of what is left to do in case that it exits during processing.
I have two questions regarding this processing_queue:
-
Is it correct to reason that there will be at most one item/task at any time in a worker's
processing_queue? I assume that a worker starts by checking any left-over task in its ownprocessing_queuebeforeBRPOPLPUSHthe main task queue. If so, we can use any ofRPOP,LPOP,LREMto remove a task once the worker finishes processing (or it can simply delete the list). We can even use a set instead of a list. Is there any reason why so many people choose to useLREMbut nothing else? -
I have seen many people recommend to identify the individual
processing_queueusing the process ID of the corresponding worker. But what happens when the old worker exits and a new one gets spawned with a (most likely) new process ID. How does the new worker look up its predecessor'sprocessing_queueto finish a possible left-over task? I plan to useSupervisorto manage my worker processes, if that makes a difference.
Aucun commentaire:
Enregistrer un commentaire