lundi 7 février 2022

Using RabbitMQ to parallel process parts

I have a service that receives a series of files from a 3rd party application. I need to process these files, and store the result in a single S3 object. Further, 3rd party application can send large number of files. So I do not want to wait until all the files are sent to start processing them. Additionally, for scaling purposes I want to process these files parallelly and then combine the results to form a single S3 objects. Hence I created a RabbitMQ exchange and bound a queue to it. I connected my consumers to this queue. These consumers will work on a set of files and store their temporary result (partial objects) to S3. Now, I want to send a final message to indicate that parts can be combined and uploaded to S3. As each consumer is processing these "sub" messages independently how do I know when the processing is complete? Are there any message design pattern I can use to solve this? Note that I can put a correlation ID for the messages if needed.

I tried to depict my setup here: RabbitMQ Setup

Aucun commentaire:

Enregistrer un commentaire