jeudi 18 mars 2021

Design help : Splitting a job to be processed by lambda : Issue in combining back

I am designing a system, where I need to split a job into pieces, put an job identifier in each piece to make sure I can combine them later, put them on SQS which triggers concurrent lambdas to process them.

Now since I'll be using asynchronous lambdas here, the output will be put in another SQS queue.

I can return the job's result only when I have processed (or failed) all such pieces. I am planning to keep a check for all the unique job IDs and the total pieces processed, after the second SQS queue. If the number of pieces reaches the total number of pieces in the input, then I can return the output.

This seems overly complicated and error prone. I think this should be a common problem, so there should be some simpler, more elegant way to do this.

Any ideas?

Aucun commentaire:

Enregistrer un commentaire