jeudi 28 mai 2020

Find a best practice for Business Intelligent data processing

I'm working in a system that manages human resources and it has a BI (Business Intelligent) part to collect and process data from main system, then visualize processed data into charts, tables, ..

For example, we want to see the relation between person age [in range 18 - 38] (in axis 1) and their monthly salary (in axis 2) [in full salary range]. The aggregation value is counting in person. There is also an additional step called Filter, to filter the result only in the organization A.

The expected result is like this:

                 Age_18<28   Age_28<38 Age_38<48
Salary_<1000         12          25       45
Salary_1000<5000     12          10       2
Salary_>5000         1           1        2

The current processing steps are as below:

  1. Search for axis1: Search all people with age range [18-38] in organization A
  2. Search for axis2: Seach all people in in organization A
  3. Merge results for axis1 and axis 2
  4. Counting people for each condition, for example number of people that has Age_18<28 AND Salary_<1000 is 12, and so on.
  5. Convert to json response

Because there are a lot of cases to handle, the logic becomes complicated to maintain. All steps are handled manually like above.

So I just wonder if this is a common problem and should have a common way to handle, For example a design pattern, or algorithm, or library (Java) or a specific concept to handle such things that I never know before.

Target: - make code more simple, readable and maintainable - easy to extend, i.e add new cases

What I'm about to try: - Apply chain of responsibility + strategy patterns

Aucun commentaire:

Enregistrer un commentaire