samedi 23 janvier 2021

Design a data structure/table that stores user operations as a sorted list?

Have a use case where I could accumulate different different types of financial operations for user account during the course of the day and these can only be processed at the end of day as part of a batch accounting process to balance the book of records. So during the course of the day user accounts can accumulate different “money in” operations(about 4-5 types of money-in categories) and a few money-out operations also(another few categories)

If a user account has simply only money-in , it can be processed without any issues in parallel. The money-out operations need to be based on the overall account balance, so we would need to process the money-in first and then the money-out .Even within money-out, there are different ranks for different money-out operations and these need to be sequenced

Not all accounts will accumulate all types of operations on the same day(depending on account activity). So if account A has just money-in and if account B had only a certain money-out, these 2 can be processed in parallel . An account C that has a money_in and money_out on the same day, has to have an orchestrated set of operations

We are building these as java micro services using queueing and event brokers .

I was looking to come up with table where I could have columns like accountId, current_operation,successor_operations,rank_of_operations .

As operations accumulate during the day, I can update this table with account activity as above.

When I want to process these in the nightly batch, I could have maybe 200 mil+ transactions and these need to be reconciled as soon as possible. So the idea is I want to start parallel processing the tens The plan I have is for very operation I would have separate queues and I would grab all the accounts with the “current_operation” (which means its the first for that account and hence can be done without any need for sequencing and drop it into the respective queues.

Once that operation is completed, I would update the table with the next operation

The thing is I need to have a central orchestration service that polls the table every now and then and routinely puts the right account into the right queues I would like to avoid this traffic cop service and am wondering if there is a better way to design the solution.

Aucun commentaire:

Enregistrer un commentaire