Viable design for multiple batch jobs

There are some tried and tested patterns for doing this, but if you'd like to save yourself some work and get extra functionality at the same time, you might want to check out Skoodat Relax. Skoodat have built a package up on the platform specifically for scheduling batches with as much power as possible and it sounds like a good match for your needs.

Otherwise, the limit of five jobs is being increased, and you can take the approach you've suggested. Depending on the exact details of the job you're running, you may be able to leverage triggers to do the processing on the other objects and just run the batch on your main record set.


You could also create you own custom apex batch queue. I did this myself and made it public as the GitHub project SObject Work Queue. I had the following design goals:

  • Must prevent Max 5 batch in parallel limit - We should never run into this limit with work that is processed over the queue.
  • The queue needs to be so generic that "work" on any type of database object needs to be enqueued.
  • Any type of modification of database objects need to be possible. This should be transparently handled by the queue.
  • Provides better error diagnostics like Batch. Knows last successful Id, full stacktrace and sends email to developers.
  • Secures data integrity like Batch or better. Failures should not leave data in inconsistent state or user of the infrastructure should be able to handle them.
  • Optimistic locking : Instead of locking all many records we do not to process work on Ids that have other work already scheduled.
  • Work that can be run synchronously, should not be queued and processed asynch.

Maybe you want to check if that could be used in you case or collaborate on GitHub to extend it for your purpose.

Here is an overview sequence diagram that shows how work is defined, enqueued and processed in such a custom Apex Queue:

enter image description here