lundi 28 novembre 2022

Search from combined data of multiple providers

I am faced with the following problem:

The goal is to be able search company and employees data and to be able to search them on certain criterias that the client can choose.

Example of criterias/fields are:

  • On a company: The number of employees, the industry and the orientation (B2B, B2C)
  • On an employee: Seniority, department

I get this type of data from multiple providers API.

Some of the fields I only get from only 1 provider, for example ProviderA gives me the orientation.

Other fields I get in different format. ProviderB gives me the number of employees as a number, ProviderC give me a range.

For each provider I store the raw data in a relation database such as: Provider_A_company, Provider_A_employee

I've tried to combine Company rows of multiple providers by domain (example.com) and combine Employee rows by employee email in order to create a Company view and an Employee View. However I'm concerned that this will not scale as the number of providers increases.

So I'm a bit stuck on how to approach this problem, on how to normalize and combine this data to be able to search it.

Is there a particular design or pattern that you would suggest to handle this?

Thank you

Aucun commentaire:

Enregistrer un commentaire