How to properly test ActiveJob's retry_on method with rspec?

This is the format of specs needed for retry_on that finally worked for me:

it 'receives retry_on 10 times' do
  allow_any_instance_of(MyJob).to receive(:perform).and_raise(MyError.new(nil))
  allow_any_instance_of(MyJob).to receive(:executions).and_return(10)
  expect(Bugsnag).to receive(:notify)
  MyJob.perform_now(an_object)
end

it 'handles error' do
  allow_any_instance_of(MyJob).to receive(:perform).and_raise(MyError.new(nil))
  expect_any_instance_of(MyJob).to receive(:retry_job)
  perform_enqueued_jobs do
    MyJob.perform_later(an_object)
  end
end

For the first case, executions is an ActiveJob method that gets run, set and checked every time retry_on is executed. We mock it to return 10 and then expect it to call Bugsnag. retry_on only calls what you gave it in the block once all the attempts have been met. So this works.

For the second case, Then mock the error to raise for the job instance. Next we check that it's correctly receiving retry_job (which retry_on calls under the hood) to confirm it's doing the right thing. Then we wrap the perform_later call in the minitest perform_enqueued_jobs block and call it a day.

The following works fine for me, also for multiple testcases and for testing side effects of the retry_on block.

RSpec.describe MyJob, type: :job do
  include ActiveJob::TestHelper

  context 'when `MyError` is raised' do
    before do
      allow_any_instance_of(described_class).to receive(:perform).and_raise(MyError.new)
    end

    it 'makes 4 attempts' do
      assert_performed_jobs 4 do
        described_class.perform_later rescue nil
      end
    end

    it 'does something in the `retry_on` block' do
      expect(Something).to receive(:something)

      perform_enqueued_jobs do
        described_class.perform_later rescue nil
      end
    end
  end
end

Note that rescue nil (or some form of rescue) is required if you let exceptions bubble up at the end.

Note that perform_now doesn't count as "enqueued job". So doing described_class.perform_now results in one less attempts counted by assert_performed_jobs.

IMHO you should leave the testing of ActiveJob with the rails team.

You only need to make sure you're configuring the job properly:

it 'retries the job 10 times with 2 minutes intervals' do
  allow(MyJob).to receive(:retry_on)
  load 'app/path/to/job/my_job.rb'
  expect(MyJob).to have_received(:retry_on)
    .with(
      Exception,
      wait: 2.minutes,
      attempts: 10
    )
end