Design pattern or a framework for long running processes submitted through website / rest enpoints

I am bringing up this question again. Question like this has been answered before but it was asked and answered a while back I want to know what is the best way to do this nowadays.

So we run scans based on what user requests. scans are OS processes that run for a long time. The user can submit these jobs either using our REST endpoint or using our Web UI. As of now we are running the jobs using Java threads. That is we spawn a new thread for a new job and update the status periodically in the database.

Is there a better way to do this? That will:

• Be easier to manage
• Provide a mechanism to communicate progress/status to user
• Be a well tested framework that scales well

Thank you!