jeudi 6 avril 2023

AWS Step Function - Wait until a group of other Step Functions have finished then fire a different Step Function

I have a scenario where I need to post process results that have been produced by a group of discrete Step Functions. How can I orchestrate this arrangement such that, if I have Step Function A, B and C. Once A, B and C have completed successfully then trigger Step Function D.

Step Function D will take as a payload outputs from Step Functions A, B and C. A, B and C are triggered from an external Java Microservice. I have a Dynamo DB table containing details of A, B and C, so I know which execution IDs belong together.

This seems to be quite a common pattern so I was hoping that there was already some sort of robust design to address it.

I have thought about using SNS to trigger an event when Step Functions A, B and C complete but I need to capture these events together in a group. So if I had a Lambda which captured the event, I would need to somehow know which event this is and whether or not all prior events have been received. I could use a Dynamo DB table to track each Step Functions completion status, at the end of the Step Function update the row. Then the lambda when it receives the completion event can check if each of the rows pertaining to the group of executions is marked as completed? Would this introduce a race condition? is this a trustworthy method?

Aucun commentaire:

Enregistrer un commentaire