I need to process large XMLs (2GB).
Processing the file can take longer than receiving the file. Let's say that there is a component A that receives the message and a component B that processes the message.
We need to use a queue, otherwise B might not have enough resources to process all messages as they arrive from A.
So A receives the message, sends it to a queue and B picks the message from the queue and processes it.
For efficiency, A and B could be part of the same process. That way, when A receives the message, it will already be in memory, ready to be processed by B. This would be helpful to decrease latency in situations where there are not many messages arriving on A.
If we would run A and B on multiple machines, how can we ensure that a message received by A on one machine will be processed by B on the same machine?
Aucun commentaire:
Enregistrer un commentaire