My use case is as follows:
1) Read 20 Million records from Db2 databse and read the filter criteria from Db2 where it involves with multiple columns and some of the columns has patterns like Column A with value %EMP%. 2) Now for each combination of the rules filter the data on 20M dataset and at the same time update the database column which has a flag indicating this record is filtered out. 3) At the end of the process, we will invoke a Informatica workflow which will take the unfiltered records for 20 M and process it.
We do not want to have the filtering logic on Informatica as it would be expensive so looking for an option to do it using Spring Batch where we can span multiple threads and run the filtering logic.
I am not sure if the Spring Batch is the right candidate for this. But I need some suggestions if I need to implement this on Java.
Please suggest
Aucun commentaire:
Enregistrer un commentaire