dimanche 28 août 2022

Maintain the status of events in multithreading

I want to read records (1000k) from 1 table and push them to some service.

So I have clubbed 200 records(based on the service limitations) in 1 event and used the executor framework and have created 10 executors. 10 events will be processed (i.e. 10*200 records) parallelly.

Now I want to maintain the status of these events, like statistics on how many were processed successfully and how many failed.

So I was thinking of

Approach 1: Before starting the execution, writing each event id + record id with status

event1 + record1 -> start

and on completion

event1 + record1-> end

And later will check how many have both start and end in the file and how many do not have end.

Approach 2 : Write all record ids in one file with status pending and writing all successful records in another file

And then check for the missing in the successful file by using pivot

Is there a better way to maintain the status of the records?

Aucun commentaire:

Enregistrer un commentaire