lundi 6 novembre 2017

Best practice for an Import/Export design?

In my work I come across this Import/Export problem very often. It is about getting data from one System to another just once like a migration or on a scheduled time (eg. once a day) or onChange of Data like a syncronisation.

In my opinion there are three/four essential steps to keep the code clean und overstandable:

  1. Raw import For instance getting data from an CSV/XLS File or a Database and put them in objects defined very near to the design that is given. (Every field/column is a member of a class which will be saved in a list for every row)
  2. Data organisation I design a classes that fits best for the given data and use mostly derived classes, independet from how it will be exported. Any first validation of the data will happen here too (Like special characters which can't be handled by all systems). I want to make it easily overstandable and robust. I try to connect data that have the parent/child hierarchy as following: A list of objects that fit my designed class (parents) and a list within the parents list which has the same design and acts as the children.
  3. Data computation/Export Here is the magic which will change and map all the data that will be needed for the target System. I often part the computation and the export so that i use the export like an API.

This may sound trivial but there are a lot of problems I faced which made me come up with this design (pattern).

Maybe there are more generic designs out there which i didn't consider before? It's always a question if I compute all the data at once and than export it or export it one by one to get early feedback from the target system. Maybe make a validation while computing it, so exporting will be easily with one hop? Sometimes its not even clear which is the unique key for both systems so one has to come up with something which is also generic.

I guess this is a more common question but i'm sure i'm not the only one with this design problems. Are there any further practices for this situation?

Kind regards
Christian

Aucun commentaire:

Enregistrer un commentaire