dimanche 29 janvier 2023

Message broker/queue architecture design pattern

I'm designing the architecture of a platform to introduce a message broker in an existing data collection web application.

This web application is currently used to upload data from excel files that are then validated, processed and inserted into a database. Every time a new set of data is inserted we should run one or more scripts, depending on the nature of the data, to perform a deep analysis of the data.

These scripts are written by data scientists or other programmers, therefore are not in the same technology as the main application and some of them are also hosted on another machine. Some of this script could run for also 2~3 hrs to produce the results.

For the above reasons, I think that introducing a message broker, like RabbitMQ or SQS or Kafka, could be an added value to making these applications talks together and synchronize their running processes.

The messages should be two-way: The main application should notify the "scripts" that there is new data available to be processed, and the "scripts" should notify the main application when they have finished (so I can also run sequential operations...)

I foresee having around 200 messages exchanged per day with a max peek of 400msgs/day, splitter in ~10 different queues/topics.

The software architect is contesting me to have only one message broker he said that I need to make one broker for each way, so I need to set up one message broker for the messages produced by the main application and another one for the messages produced by the "scripts" and micro-services.

I think it's really a waste of resources considering the amount of data that I have to manage.

Am I wrong and there is something I'm missing in the design pattern?

my design with only one message broker to manage both consumers/producers

Aucun commentaire:

Enregistrer un commentaire