lundi 16 décembre 2019

Design to handle tcp data burst

We are collecting data from a source and writing to a file. We have multiple threads say around 15 which listen to different socket and port connection and write to a single file. We see that when there is a burst of data and volume suddenly goes like 150-200 times our process fails to keep up. Note that there is a lock applied while dumping the data in the file.We are using PROTOBUF.

We are using Poco's library where we fetch all read even list using Poco::Net::Socket::select and then in loop run and go through each of them parse them and then publish them. So effectively there is a bottleneck that even parsing of the message is being done in single thread and then also dumping of data is a single thread process.

I am expecting that writing of data in file would be in sequence but then how can i parse messages in parallel and only have a lock while writing in file to avoid queue from building up in tcp socket buffer?

Can someone point me to a good design that i can use or modify my present code to ?

Aucun commentaire:

Enregistrer un commentaire