I have multiple producers (as of the producer/consumer design pattern). Every producer should create data for a given topic (for consumers to be consumed - however this consumers don't play a role in this problem here). The topics are stored in some stateful manner.
I have 2 issues to overcome:
- how to distribute the topics to the producers?
- how to redistribute topics of crashed/stuck producers?
In this microservice architecture there is at least a database (like MongoDB or PostgreSql) to store the states and other things. There is also a Message Broker (like RabbitMq or Kafka).
For the first issue my solution would be to distribute the work via the Message Broker and every producer takes as many topics as it can handle (and starts producing). However for the second issue I don't have a clue how to handle the detection of a failed producer and how to redistribute the work properly again. ( I fear race conditions, or 2 producers on the same topic. I also would like to minimize downtimes )
I would really appreciate if you could guide me to the correct search terms or could give me names of design patterns solving this issue.
(I don't think this is really relevant, but I'm using kubernetes and .net5 for this project.)
Aucun commentaire:
Enregistrer un commentaire