for $p>0$, when does this integration:$\int_0^{\infty} x^pe^{-x^8\sin^2x}dx$ converge?

HINT:

Using $2x/\pi\le\sin(x)\le x$ for $x\in [0,\pi/2]$, we can assert that

$$\begin{align} \int_0^{\pi/2} (x+n\pi)^pe^{-(x+n\pi)^8\sin^2(x)}\,dx&\le (\pi/2+n\pi)^p\int_0^{\pi/2} e^{-(n\pi)^8(2x/\pi)^2}\,dx\\\\ &\le (\pi/2+n\pi)^p\int_0^\infty e^{-4n^8\pi^6x^2}\,dx\\\\ &=\frac{(n+1/2)^p\pi^p}{4\pi^{5/2}n^4}\\\\ &=O\left(n^{p-4}\right) \end{align}$$

as $n\to \infty$.

Can you finish now?

Disclaimer: Do not be afraid of the constants. They come out naturally and it is not necessary to have them explicitly.

Since you asked for the values of $p>0$ for which the integral converges, I will show you that the integral converges if-f $0<p<3$. In doing so, I will have to prove the estimate of your teacher at some point. We begin with

$$\int_{0}^{+\infty}x^pe^{-x^8{(\sin x)}^2}dx=\int_0^{\pi}x^pe^{-x^8{(\sin x)}^2}dx + \sum_{n\geq1}\int_{0}^{1}(x+n\pi)^pe^{-(x+n\pi)^8(\sin x)^2}dx+\\\sum_{n\geq1}\int_{1}^{\pi-1}(x+n\pi)^pe^{-(x+n\pi)^8(\sin x)^2}dx.\ \ \ \ \ \ \ \ (*)$$

In the first integral the integrand is continuous on $[0,\pi].$ Consequently, the integrand is bounded there and so, the aforementioned integral is finite. This means that the integral of the right hand side will not affect the answer to our problem. Next, we show that the second series is convergent. On the interval $[1,\pi-1]$ we have that $(\sin x)^2\geq (\sin 1)^2=:c>0.$ Moreover, it is clear that $(x+n\pi)^8>n^8\geq n$ for $n\geq 1$ and $x>0.$ Thus,

$$e^{-(x+n\pi)^8(\sin x)^2}<e^{-nc}\ \ \ \ \ \ \ \ \ \ \ \ (1)$$

in the integrals of the second series. For a positive integer $n$ and an $x\in [1,\pi-1]$ we have that

$$x+n\pi<\pi(n+1)\leq 2\pi n=\frac{4\pi}{c}\cdot\frac{nc}{2}<\frac{4p\pi}{c}e^{\frac{nc}{2p}}.$$

Hence, $(x+n\pi)^p<(4p\pi/c)^pe^{\frac{nc}{2}}$ for positive integers $n$ and $x\in [1,\pi-1].$ We combine this inequality with $(1)$ and conclude that the second series is bounded from above by

$$\left(\frac{4p\pi}{c}\right)^p(\pi-2)\sum_{n\geq 1}\left(e^{-\frac{c}{2}}\right)^n=\left(\frac{4p\pi}{c}\right)^p\frac{(\pi-2)e^{-c/2}}{1-e^{-c/2}}<+\infty.$$

Of course, a series of positive terms, which is bounded, converges. So, the second series of $(*)$ is, indeed, convergent. This means that the answer to our problem depends only on the behavior of the first series of $(*)$. On $[0,1]$ (and more generally on $[0,\pi/2]$ by looking at the graphs) it is true that $\sin x\geq \frac{2}{\pi}x\geq 0$. As a result, for $x\in [0,1]$ and $n\geq 1$ we get

$$(x+n\pi)^pe^{-(x+n\pi)^8(\sin x)^2}\leq (1+n\pi)^pe^{-\frac{4}{{\pi}^2}n^8x^2}<5^pn^pe^{-c'n^8x^2},$$

where $c':=4/{\pi}^2>0.$ Consequently, the first series of $(*)$ is bounded from above by

$$5^p\sum_{n\geq 1}n^p\int_{0}^{1}e^{-c'n^8x^2}dx=\frac{5^p}{\sqrt{c'}}\sum_{n\geq 1}n^{p-4}\int_{0}^{\sqrt{c'}n^4}e^{-u^2}du\leq \frac{5^p\sqrt{\pi}}{2\sqrt{c'}}\sum_{n\geq 1}n^{p-4}.$$

At the last step we bounded all integrals by half of the Gaussian integral. Furthermore, $|\sin x|\leq |x|$ and so,

$$(x+n\pi)^pe^{-(x+n\pi)^8(\sin x)^2}>{\pi}^pn^pe^{-(1+n\pi)^8x^2}>{\pi}^pn^pe^{-c''n^8x^2}$$

for $c'=5^8>0,\ x\in [0,1]$ and $n$ a postive integer. Therefore, we deduce that the first series of $(*)$ is bounded from below by

$${\pi}^p\sum_{n\geq 1}n^p\int_{0}^{1}e^{-c''n^8x^2}dx=\frac{{\pi}^p}{\sqrt{c''}}\sum_{n\geq 1}n^{p-4}\int_{0}^{\sqrt{c''}n^4}e^{-u^2}du\\ \geq \frac{{\pi}^p}{\sqrt{c''}}\left(\int_{0}^{\sqrt{c''}}e^{-u^2}du\right)\sum_{n\geq 1}n^{p-4}.$$

The integral at the end is finite, because it has a finite interval of integration and its integrand is bounded by $1$. Summarizing, the series $\sum_{n\geq 1}n^{p-4}$ differs only by a constant from the first series of the right hand side of the first equality. Since that series determines the convergence of our integral, we deduce that the initial integral converges if-f the series $\sum_{n\geq 1}n^{p-4}$ converges. But, this is known. Thus, the integral of your question converges if-f $0<p<3.$