How $dxdy$ becomes $rdrd\theta$ during integration by substitution with polar coordinates

The answers using the Jacobian are (of course) correct. You can get some intuition from this picture:

$\Delta A$ is (approximately) a rectangle with sides $r \Delta \theta$ and $\Delta r$. Its area is proportional to $r$ since it scales as $r$ increases.

Picture from http://citadel.sjfc.edu/faculty/kgreen/vector/block3/jacob/node4.html

We are using polar coordinates that is

• $x=r \cos \theta$
• $y=r \sin \theta$

and by the Jacobian determinant we have that

$$dxdy=\begin{vmatrix}\cos\theta&-r\sin\theta\\\sin \theta&r\cos\theta\end{vmatrix}drd\theta=rdrd\theta$$

Refer also to the related

• What is the Jacobian matrix?
• Origin of Jacobian determinant

Taylor series mandate $dx^2=0$, from which you can show $dxdy=-dydx$ etc. (Look up the wedge product on differential fotms.) Thus$$\begin {align}dxdy&=(\cos\theta dr-r\sin\theta d\theta)(\sin\theta dr+r\cos\theta d\theta)\\&=r(\cos^2\theta+\sin^2\theta)drd\theta.\end{align}$$In particular, the $drd\theta$ coefficient is a determinant called the Jacobian.

Note, however, the calculation $ds^2=dx^2+dy^2=dr^2+r^2d\theta^2$ takes infinitesimals as commuting, because this time there's no nilpotency axiom, so we're not working with the wedge product on differential forms.