vendredi 30 octobre 2020

Consumer Producer design

I am looking for advice on how the following problem is handled in solutions that use Pulsar.

The scenario is: A producer is sending messages (in JSON) and the consumer is taking them and inserting data into database tables (specified in the message).

Suddenly the producer changes the structure of the data sent in the messages (let's say because the table structure in the database has a new column).

So for a moment, Pulsar has messages with the old data structure and now starts receiving messages with new data structure.

My doubt is regarding how the consumer should handle this scenario. What to do with the messages with old structure that are now invalid since they cannot be inserted in the database table since the table structure changed. Retry and then permanently fail (dead letter Q?).

Also, do you usually opt to sent the metadata along with your messages or do you normally handle this in a separate topic or other form.

Thanks for any advice

Aucun commentaire:

Enregistrer un commentaire