mercredi 27 mars 2019

Recommended strategy for a collection with large amount of inserts and multiple readers using different indexes

I need some help with a strategy for a collection that is used in our app. The collection receives a large amount of inserts which we'd like to process as rapidly as possible, yet we also have a UI that allow users to read from this collection using a number of different properties. We're looking for a strategy that permits the fastest insert rates into the collection, yet uses indexes appropriately for the reads.

At the moment it's mostly working, however we are getting to the point where we need to increase both read and write rates so need to decide what to do next. Some initial testing shows we can get better insert rates without the indexes yet clearly our UI falls to its knees. Ideally we'd replicate the collection that 'receives' the documents to another server which has the indexes on it and that is the collection the UI would read from. A short amount of latency is permissible here while the date is copied over. However when we've looked into this in the past using the default replication it seemed very hard to have different indexes on the secondary compared to a primary.

Are there any tools (builtin or otherwise) we can use to perform this? It's not really a traditional "BI" service as the data arrives in exactly the format we want it so there's no need to any aggregations etc and we don't want a "nightly" process as we'd like it to be as near-live as possible (a few seconds, maybe a minute's delay would be OK).

Thanks.

Aucun commentaire:

Enregistrer un commentaire