Real Time video Processing Design issue

I am a newbie in the field and don't know if i am doing this right. So I need help with architectural and design issue, just to make sure if I am moving in right direction.

I am trying to process a real time video capturing from the client's camera. The captured video stream is converted into frames and send to the server for processing. After processing, server is sending it back to client's screen.

My backend code is written in Python and I am using SocketIo to send the frames from frontend to backend. You can have a look at this design to get a better idea about what's happening - https://docs.google.com/drawings/d/1_65mikOZaaL1oLHFEzTzaoKptgj98OX1_EALqCIwYqI/edit?usp=sharing .

1. My server(app.py) will be running in backend and client will be accessing index.html
2. SocketIo connection will get establish and video stream captured using webcam will be send to server frames by frames.
3. These frames will be then processed at the backend and emit back to the client.
4. Processed frames coming form the server can be shown in img tag.

So, my problem with this architecture is the lag. There is a noticeable delay even after limiting fps. From client to server then processing and back to client, each frame is take on an average 150 ms and is not giving a real time response. I have tried my image processing code separately with no socket programming on local and it is more near to real time (it is taking 80 ms for one frame).

But I fail to understand here that how can I minimize lagging with this design or improvise it? Is there any better way to do this task to get more real time results?

Thanks in advance!