We have a monolith JAVA application which we are starting to re-write and possibly break down into smaller services. The application has all the REST APIs to power the UI. The application also has a number of background threads that start on application startup.
For example there are three single thread executers that are responsible for doing totally independent set of tasks.
These background jobs mostly are data pipelines. The database access layer is very poorly designed (currently) and are intertwined within the monolith, hence limiting different options.
The major issue we have seen with this setup is that it is non-scalable - Since these background jobs all operate within the main application, we have limited the number of parallel executions, to make sure the main application is not impacted.
For the new application, we will be using Java and Springboot.
[Separate note] We are using airflow for a separate pipeline, however we are trying to get away from it due to operational constraints. So trying to stay away from cloud solutions for now (but suggestions are welcomes)
What are the ways/best practices to design these use cases?
Aucun commentaire:
Enregistrer un commentaire