We are collecting data from a source and writing to a file. We have multiple threads say around 15 which listen to different socket and port connection and write to a single file. We see that when there is a burst of data and volume suddenly goes like 150-200 times our process fails to keep up. Note that there is a lock applied while dumping the data in the file.We are using PROTOBUF.
We are using Poco's library where we fetch all read even list using Poco::Net::Socket::select and then in loop run and go through each of them parse them and then publish them. So effectively there is a bottleneck that even parsing of the message is being done in single thread and then also dumping of data is a single thread process.
I am expecting that writing of data in file would be in sequence but then how can i parse messages in parallel and only have a lock while writing in file to avoid queue from building up in tcp socket buffer?
Can someone point me to a good design that i can use or modify my present code to ?
Aucun commentaire:
Enregistrer un commentaire