How to compute a Jacobian using polar coordinates?

The Jacobians of the two functions aren't equal by the chain rule.

In actual fact, $D(\phi(\frac{1}{r}, \cos\theta)) × D\tilde{F}(r, \theta)= DF \times D(\phi(r, \theta))$

I don't think there is any contradiction here.

Consider the volume form $$ \omega_{\rm Cart} = dx \wedge dy.$$ Your first calculation shows that the pullback $F^\star(\omega_{\rm Cart})$ is given by $$ F^\star(\omega_{\rm Cart}) = - \frac{1}{(x^2+y^2)^2}\omega_{\rm Cart}.$$

Now consider the volume form $$ \omega_{\rm Polar} = dr \wedge d\theta.$$ Your second calculation shows that

$$ F^\star(\omega_{\rm Polar})=-\frac 1 {r^2} \omega_{\rm Polar}. $$

We can use this to recompute $F^\star(\omega_{\rm Cart})$. In view of the fact that $$ \omega_{\rm Cart} = r \omega_{\rm Polar},$$ we have: \begin{align} F^\star(\omega_{\rm Cart}) &= F^\star(r\omega_{\rm Polar}) \\ &= F^\star(r) F^\star(\omega_{\rm Polar}) \\ &= \frac 1 r \left( - \frac 1 {r^2}\omega_{\rm Polar} \right) \\ &= - \frac{1}{r^4} \left(r\omega_{\rm Polar} \right) \\ &= - \frac 1 {r^4} \omega_{\rm Cart} \end{align} which is consistent with the first calculation!

---

As for the application of the chain rule, we have: $$ (D\bar F)|_{(r, \theta)} = D(\phi^{-1})|_{F\circ \phi(r, \theta)} (DF)|_{\phi(r, \theta)} (D\phi)|_{(r, \theta)}$$

The key point is that you must evaluate $D(\phi^{-1})$ at the point $\left(\frac x { (x^2 +y^2)}, \frac y {(x^2 + y^2)}\right)$, not at the point $(x, y)$.

This is equal to

$$ D(\phi^{-1})|_{F\circ \phi(r, \theta)} = \begin{bmatrix} \frac{\frac{x}{x^2 + y^2}}{\sqrt{\left(\frac{x}{x^2 + y^2} \right)^2+\left( \frac{y}{x^2 + y^2}\right)^2}} & \frac{\frac{y}{x^2 + y^2}}{\sqrt{\left(\frac{x}{x^2 + y^2} \right)^2+\left( \frac{y}{x^2 + y^2}\right)^2}} \\ -\frac{\frac{y}{x^2 + y^2}}{\left(\frac{x}{x^2 + y^2} \right)^2+\left( \frac{y}{x^2 + y^2}\right)^2} & \frac{\frac{x}{x^2 + y^2}}{\left(\frac{x}{x^2 + y^2} \right)^2+\left( \frac{y}{x^2 + y^2}\right)^2}\end{bmatrix} = \begin{bmatrix}\cos\theta & \sin \theta \\ - r\sin \theta & r\cos\theta \end{bmatrix}$$ which is not the inverse of $(D\phi)|_{(r, \theta)}$.