Double integral bounds of integration polar change of coordinate

This question is a bit tricky because $R$ is not a circle quadrant, it's an ellipse quadrant. Therefore, the underlying ellipse needs to be transformed into a circle by a substitution – $u=\frac23x$ works: $$\iint_Rxy\,dA=\frac94\iint _Suy\,dA$$ $S$ is now a quarter-disc of radius $4$ around the origin, so polar coordinates can now be used: $$=\frac94\int_0^4\int_0^{\pi/2}r^3\cos\theta\sin\theta\,d\theta\,dr$$ $$=\frac94\int_0^4\frac12r^3\,dr$$ $$=\frac98×\frac14×4^4=72$$

The entire problem is that the domain is 1 quarter of an ellipse, and not a circle. Thus, we must parameterize the domain as we would parameterize an ellipse.

As $\frac{x^2}{36}+\frac{y^2}{16}=1$ can be parameterized as $x=6\cos\theta, y = 4\sin\theta$, the parameterization of the ellipse is $$x=6r\cos\theta, y = 4r\sin\theta$$ where $r\le 1$, $\theta \in [0, \frac{\pi} {2}]$

Let $x = 6r\cos\theta$ and $y =4r\sin\theta$

So, $\frac{x^2}{6^2}+\frac{y^2}{4^2}=r^2$

Now, $r^2\le1$

$x\ge0$ and $y\ge0\implies0\le r\le1$ and $0\le\theta\le\pi/2$

$dxdy = 24rdrd\theta$

$I = \int^{\frac{\pi}{2}}_0\int^{1}_{0}r^2(24\sin\theta\cos\theta)(24\ r)dr\ d\theta = 576 \int^{\frac{\pi}{2}}_0\int^{1}_{0}r^2(\sin\theta\cos\theta)(\ r)dr\ d\theta = 576\frac{1}{8} = 72$

(You've already found $\frac{1}{8}$, so I just multiplied it by $576$)

---

$x = 6r\cos\theta$ $y =4r\sin\theta$

Partial derivatives:

$x_r = 6\cos\theta$, $y_r = 4\sin\theta$

$x_{\theta} = - 6r\sin\theta$ , $y_{\theta} = 4r\cos\theta $

$J = \begin{vmatrix}6\cos\theta & 4\sin\theta \\ - 6r\sin\theta & 4r\cos\theta\end{vmatrix} = 24r\cos^2\theta+24r\sin^2\theta = 24r$

So, $dxdy = |J|drd\theta = 24r\ dr\ d\theta$