vendredi 1 janvier 2021

performance issue for real-time data management

I have an application that allows me to visualize data in real-time. I receive messages via WebSocket connections, I process the messages and display/calculate them. The GUI part is updated every 100 ms. (I would like 50 ms but I don't have the performance)

The principle is that I click on a button to ask for information about something, and my program subscribes to the corresponding WebSocket to get the data. I can have dozens of different subscriptions that are displayed in separate windows.

I first tried to run everything on the same multithreaded process.

The problem is that sometimes I have big spikes of data that I have to store in tables, perform specific calculations/processing, and then display them on the screen. My computer has 4 cores, but there are too many things to do and I regularly have lags and big processing delays.

So I thought of separating my application into 2 parts: One part that manages the graphic application, some background calculations, etc, and another part that will be connected to the different WebSocket to process all the messages.

The problem is that I don't know how to make the 2 processes communicate.

I thought about the google service of push and sub, using sockets, database, etc.

The data to send can be "huge", I have tables of hundreds of lines that can be updated hundreds of times per second, sending back the updated tables will be very expensive. The multithread allowed me to use the same arrays and all this part was very simple to manage with a simple lock hierarchy.

I'm stuck because I don't know to design this kind of thing. I've done some research but I don't know what to look for, so I don't think my research is very effective.

I don't have any sample code to give out. It's a design and performance issue for real-time data management.

I don't expect to receive a ready-made solution but just leads to being able to work. If there is a lack of information, let me know.

Aucun commentaire:

Enregistrer un commentaire