mercredi 25 juillet 2018

Manage messages with multiple consumers of one Kafka topic

I am implementing a workflow leveraging Apache Kafka where I have multiple producers and multiple consumers.

In a nutshell, something like an order processing workflow where:

  • Producer receives and publishes an 'Order' to a Topic. Consumer_1 takes the 'Order', validates it and publishes the same 'Order' to the same Topic.
  • Consumer_2 receives the 'Validated Order' and checks stock in Warehouse. Once confirmed it publishes same 'Order' to same Topic.
  • Consumer_3 receives 'Validated & Warehouse confirmed Order' and sends for dispatch to the same Topic etc...

I've created an object of type 'Order' for my consumers to process with State as a local variable that each consumer updates upon completion.

With the above approach of multiple Producers/Consumers and One topic I see an issue of maintaining state of my messages (validated, warehouse confirmed, dispatched etc), where each consumer has to process all messages and check that state corresponds to the current consumer's responsibility. Also with this approach all consumers will have to belong to different consumer groups in order to receive all messages (which is not ideal) and hence ack all messages (even those not destined for them).

Other approaches to this I thought about are creating multiple Topics (one per 'state', but in that case my 'Order' message will be used for publishing to multiple Topics, which seems to go against Kafka principles.

I could also create different 'types' of Orders, i.e. ValidatedOrder, WarehouseConfirmedOrder, DispatchedOrder whereby removing the state check in my consumers, but introducing a 'Type' filter instead, which seems to be very similar to my original state solution.

What are the best practices and alternative techniques one could apply here?

Any documentation on the above you could point me to?

Aucun commentaire:

Enregistrer un commentaire