Can a black hole fall into another black hole's innermost stable orbit?

The innermost stable orbit is a theoretical concept that applies to a "test particle" orbiting a black hole. A test particle in this case would mean that it has negligible mass and so does not alter the metric (Schwarzschild, or more realistically Kerr) it is in.

If an orbiting black hole were much less massive than the black hole it orbited, then perhaps you could make this approximation - e.g. a stellar mass black hole orbiting a supermassive black hole.

If one cannot make the test mass approximation there can be no stable orbit. As has now been experimentally demonstrated, massive orbiting bodies are a source of gravitational waves. Two objects will inevitably spiral inwards and merge on a timescale that decreases with the sum and product of their masses and the inverse of their separation.


On top of Rob Jeffries' answer, I'd like to add that gravitational wave radiation makes the black hole lose orbital energy $E$. But for really huge black holes this won't make much of a difference ( see the calculation).

Very rough estimates for the energy:

Let's take an order of magnitude estimate: The gravitational wave luminosity in the 1st Post-Newtonian approximation is given by $L_{GW}=-\frac{dE}{dt}=\frac{1}{5}\frac{G}{c^5}<\partial_t^3I_{ij}\partial_t^3I^{ij}>$.

Let's say we have a black hole of $M_{big}=10^{14} M_{\odot}$ (hypothetical mega black hole that probably doesnt exist) and a stellar mass black hole of mass $M_{small}=10 M_{\odot}$

For timescale and radius estimates we take:

$r_{isco}=6GM_{big}/c^2\approx 29pc$.

$\Delta t\approx 2\pi \sqrt{\frac{r_{isco}^3}{GM_{big}}}\approx 1500 yr$

For a point particle of mass m at a distance a from origin in a frame of reference that is rotating with angular velocity $\omega$, $_I1=ma^2$, $I_2=I_3=0$, with $\epsilon I_3=I_1=ma^2$ the luminosity of the gravitational waves is (in first PN approximation):

$L_{GW}=\frac{32}{5}\frac{G}{c^5}m^2 a^4 \omega^6$

Now $a=r_{isco}\approx 29 pc$, $\omega=\frac{1}{\Delta t}\approx \frac{1}{1500 yr}$ and $m=M_{small}\approx 10M_\odot$

and we get: $L_{GW}\approx 10^{18} W$ (Watts)

Now the black hole kinetic energy is (newtonian approximation..): $E_{kin}=\frac{1}{2}M_{small} (\frac{GM_{big}}{r_{isco}})\approx 10^{47} J$ (joules)

Results

In order for the black hole to lose any non-negligible amount of energy ($L_{GW}\cdot t \approx 10^{47}$) we would need around $10^{20}$ years (note: age of the universe is $\approx 10^{10}$ years).

So at least gravitational radiation wont shake the orbit much, but it depends on the masses of the black holes.