Difficult Integral $\int_0^{1/\sqrt{2}}\frac{\arcsin({x^2})}{\sqrt{1+x^2}(1+2x^2)}dx=$

Set $\displaystyle\,\, \cos \theta=\frac{1}{2x^2+1}.$

Then the intagral transforms into $$\frac12\int_0^{\large\frac{\pi}{3}} \sin^{-1}\left(\frac{1-\cos\theta}{2\cos\theta}\right)d\theta.$$

This type of integral is called a Coxeter integral. The user @Sangchul Lee (formerly, sos440) managed to obtain a closed-form formula for a certain family of these Coxeter integrals, by generalizing Ahmed's integral.

Here you can find his general approach, and in his paper "Coxeter's Integrals" ("Abstract. In this paper, we generalize the Ahmed’s integral and exhibit some identities involved in it.") you can found explicit computations.

Specifically, he proves there that $$I_6\equiv\int_0^{\large\frac{\pi}{3}} \cos^{-1}\left(\frac{1-\cos\theta}{2\cos\theta}\right)d\theta=\frac{11\pi^2}{72},$$

which, by recalling that whenever $x \in [0,1]:$ $\,\,\cos^{-1} x+\sin^{-1} x=\frac{\pi}{2} , \,\,\,$ gives the desired closed form of your integral.

$\displaystyle J=\int_0^{\tfrac{1}{\sqrt{2}}}\dfrac{\arcsin (x^2)}{\sqrt{1+x^2} (1+2x^2)}dx$

Perform change of variable $y=x^2$, one obtains:

$\displaystyle J=\int_0^{\tfrac{1}{2}}\dfrac{\arcsin x}{2\sqrt{x(1+x)}(1+2x)}dx$

$\displaystyle J=\left[\arcsin(x)\arctan\left(\sqrt{\dfrac{x}{1+x}}\right)\right]_0^{\frac{1}{2}}-\int_0^{\tfrac{1}{2}}\dfrac{\arctan\left(\sqrt{\dfrac{x}{1+x}}\right)}{\sqrt{1-x^2}}dx$

$\displaystyle J=\dfrac{\pi^2}{36}-\int_0^{\tfrac{1}{2}}\dfrac{\arctan\left(\sqrt{\dfrac{x}{1+x}}\right)}{\sqrt{1-x^2}}dx$

Define on $[0,1]$, $\displaystyle F(a)=\int_0^{\tfrac{1}{2}}\dfrac{\arctan\left(a\sqrt{\dfrac{x}{1+x}}\right)}{\sqrt{1-x^2}}dx$

For all $a$ in $[0,1]$, $\displaystyle F'(a)=\int_0^{\tfrac{1}{2}}\dfrac{x}{\sqrt{x(1-x)}(a^2x+x+1)}dx$

For all $a$ in $[0,1]$, $F'(a)=2\left[\dfrac{\sqrt{2+a^2}\arctan\left(\dfrac{\sqrt{x}}{\sqrt{1-x}}\right)-\arctan\left(\dfrac{\sqrt{2+a^2}\sqrt{x}}{\sqrt{1-x}}\right)}{(1+a^2)\sqrt{2+a^2}}\right]_0^{\tfrac{1}{2}}$

For all $a$ in $[0,1]$, $F'(a)=\dfrac{\pi}{2}\dfrac{1}{1+a^2}-\dfrac{2\arctan\left(\sqrt{2+a^2}\right)}{(1+a^2)\sqrt{2+a^2}}$

Therefore,

$\displaystyle \int_0^{\tfrac{1}{2}}\dfrac{\arctan\left(\sqrt{\dfrac{x}{1+x}}\right)}{\sqrt{1-x^2}}dx=\int_0^1 F'(a)da=\dfrac{\pi}{2}\int_0^1\dfrac{1}{1+a^2}da-2\int_0^1\dfrac{\arctan\left(\sqrt{2+a^2}\right)}{(1+a^2)\sqrt{2+a^2}}da$

Therefore,

$\displaystyle J=\dfrac{\pi^2}{36}-\dfrac{\pi^2}{8}+2\int_0^1\dfrac{\arctan\left(\sqrt{2+a^2}\right)}{(1+a^2)\sqrt{2+a^2}}da$

The last integral is the Ahmed's integral (see How to evaluate Ahmed's integral? )

Finally,

$J=\dfrac{\pi^2}{36}-\dfrac{\pi^2}{8}+2\times \dfrac{5\pi^2}{96}=\dfrac{\pi^2}{144}$