I am working on designing a scalable service(springboot) using which data will be indexed to elastic search.
Use case:
My application uses 6 databases(mySql) having same schema. Each database caters to specific region.
I have a micro service that connects to all these dbs and indexes data from specific tables to elasicsearch server(v6.8.8) in similar fashion having 6 elasticsearch indexes one for each db.
Quartz jobs
are employed for this purpose and RestHighLevelClient
. Also there are delta jobs running each second to look for changes using audit and indexes.
Current problem:
- Current design is not scalable - one service doing all the work(data loading, mapping, upsert in bulk). Because indexing is done through quarts jobs, scaling services(running multiple instances) will run the same job multiple times.
- No failover - Looking for a distributed elasticsearch nodes and indexing data to both nodes. How to do this efficiently.
I am considering spring data elasticsearch
to index data sametime when it is going to be persisted to db.
Does it offer all features ? I use :
- Elasticsearch right from installing template to creating/deleting indexes, aliases.
- Blue/green deployment - index to non-active nodes and change the aliases.
- bulk upsert, querying, aggregations..etc
Any other solutions are welcome. Thanks for your time.
Aucun commentaire:
Enregistrer un commentaire