I'm consuming and processing websocket data from markets 24/7, and I'm currently finding my way around AWS and trying to determine best practices. This is one of those "spent so much time thinking how to do it, but never considered if I should do it". I'd like to use what AWS has to offer, even though it's probably overkill for what I'm doing currently.
At the moment, I'm simply running a single, full-time Python script which does everything from a single EC2 instance, and it interacts with a separate RDS database to store raw data. I'd like to break this apart, as I'm now going to actually work with incoming data to maintain a market orderbook "state", probably in DynamoDB (since it's free and seems to tick the necessary boxes). I don't need bleeding edge speed and response times, but would appreciate answers that additionally cover that future, potential need.
-
Is there a better/leaner way than having an EC2 instance running a 24/7 Python script to receive and push any received websocket data to RDS? Do Docker images/instances come into play at all in my needs?
-
If I push everything I receive into an SQS queue, would I use one or more lambda functions to clean up the received data, and then have it store the result on RDS?
-
The market websockets I record from often drop out, so I'd like to make a solution whereby websocket receiver instances/images can be duplicated, and all feed their respective data into this system, and de-duped.
-
If I want to work with the received data, and store the market state on DynamoDB (as well as store it in its raw format in RDS), could this all be achieved with Lambda functions?
-
Would I just be better off doing all the work on the inital EC2 instance/Python script, or try as best as possible to utilise SQS/Lambda/others? Are there additional components I should know about?
Thanks!
Aucun commentaire:
Enregistrer un commentaire