mardi 25 août 2020

Scalable elasticsearch module with spring data elasticsearch possible?

I am working on designing a scalable service(springboot) using which data will be indexed to elastic search.

Use case:

My application uses 6 databases(mySql) having same schema. Each database caters to specific region.
I have a micro service that connects to all these dbs and indexes data from specific tables to elasicsearch server(v6.8.8) in similar fashion having 6 elasticsearch indexes one for each db.
Quartz jobs are employed for this purpose and RestHighLevelClient. Also there are delta jobs running each second to look for changes using audit and indexes.

Current problem:

  1. Current design is not scalable - one service doing all the work(data loading, mapping, upsert in bulk). Because indexing is done through quarts jobs, scaling services(running multiple instances) will run the same job multiple times.
  2. No failover - Looking for a distributed elasticsearch nodes and indexing data to both nodes. How to do this efficiently.

I am considering spring data elasticsearch to index data sametime when it is going to be persisted to db.

Does it offer all features ? I use :

  • Elasticsearch right from installing template to creating/deleting indexes, aliases.
  • Blue/green deployment - index to non-active nodes and change the aliases.
  • bulk upsert, querying, aggregations..etc

Any other solutions are welcome. Thanks for your time.

Aucun commentaire:

Enregistrer un commentaire