mercredi 31 août 2022

How do I track users in my database without affecting response times?

I'm designing a big (for me) backend API with users and I want to try when users were last seen, and what IPs they are seen with (all of them).

The usual way I do this is with a simple middleware - the middleware checks if they are authenticated. If they are, it simply updates a database entry with the current time. Then the middleware just continues to the requested function.

However, I'm dealing with more data and I want this to be done as smoothly as possible - I want the middleware to finish ASAP with no blocking database operations. I have some ideas:

  1. I use FastAPI's background tasks to do this. I don't know how much overhead they induce, and given that they would be ran on EVERY route, this seems inefficient to be creating a background task just to update a timestamp in a database. If I want this to scale, I'd have to use celery or something, which might be even slower.

  2. Redis. This seems most likely; I could simply store the timestamp and IP keyed by their address. On an interval (15-60s), I could have the API look at all the entries and update the database accordingly. This should make the requests as fast they could be, no overhead, very scalable, and the true disk database would be updated in a period that is roughly 'real time'. I like this solution, but I worry it may be classified as premature optimization.

To be clear, the requirements I'm looking for are

  • Scalable. This has to work when 10,000 users are hitting my API simultaneously, otherwise I have to drop the feature.

  • Responsive - The design is probably going to use a middleware of sorts, and this has to exit instantly. I cannot have a working waiting on something like a DB transaction to go through. The response time must not be hindered by this feature.

  • Relatively Real-time - The database has to have these IPs and User IDs usable in less than 60 seconds. I could get fleixble with this requirement, but I'd rather not be waiting too long before things can be queried.

Aucun commentaire:

Enregistrer un commentaire