Flask - job not running as a background process

Still not sure what you meant by

I mean the app waits for all requests to be made at login and only then goes to homepage. It should go right away to homepage with requests being made at background

There are a few issues here:

  • Your queue is global to the process i.e. there is only one queue per gunicorn worker; you probably want the queue to be bound to your request so that multiple requests are not sharing the same queue in memory. Consider using context locals
  • If UploadTracks is writing to the database, there might be a lock on the table. Check your indices and inspect lock waits in your database.
  • SQLAlchemy might be configured with a small connection pool, and the second UploadTracks is waiting for the first to return its connection.

In your first example, the endpoint is waiting on all futures to finish before returning, whereas in your second example, the endpoint returns immediately after submitting tasks to the executor. If you want flask to respond quickly while the tasks are still running in background threads, remove the with concurrent.futures.ThreadPoolExecutor() as executor: and construct a global thread pool at the top of the module.

Using with, the context manager waits for all submitted tasks before exiting, but I am not sure if that's your main issue.