Integrating $\frac{\arctan x}{x\sqrt{\smash[b]{1-x^2}}}$

This can be solved using contour integration, which is overkill if this is truly a textbook problem, or it can be solved using the power of double integration (or nested integration, or whatever people call it).

Hopefully, you are aware that$$\frac {\arctan x}x=\int\limits_0^1dy\,\frac 1{1+x^2y^2}$$

If not, it's a good practice problem for you to try! Next, replace the $\arctan(\cdot)$ fraction with the above identity and switch the order of integration

$$\begin{align*}I & =\int\limits_0^1dy\,\int\limits_0^1 dx\,\frac 1{(1+x^2y^2)\sqrt{1-x^2}}\end{align*}$$

Now make the substitution $x\mapsto\sin x$ to get

$$\begin{align*}I & =\int\limits_0^1 dy\,\int\limits_0^{\pi/2}dx\,\frac 1{\cos^2 x+\sin^2 x(1+y^2)}\\ & =\int\limits_0^1dy\,\int\limits_0^{\pi/2}dx\,\frac {\sec^2x}{1+\tan^2x(1+y^2)}\\ & =\int\limits_0^1dy\,\int\limits_0^{\infty}dx\,\frac 1{1+x^2(1+y^2)}\end{align*}$$

Treating $1+y^2$ as a constant inside the second integral, we can pull a factor out so that the inner integral becomes a simple $\arctan(\cdot)$ question that we know how to evaluate!

$$\begin{align*}I & =\int\limits_0^1 dy\,\frac {\sqrt{1+y^2}}{1+y^2}\arctan\left(x\sqrt{1+y^2}\right)\,\Biggr\rvert_0^{\infty}\\ & =\frac {\pi}2\int\limits_0^1 dy\,\frac 1{\sqrt{1+y^2}}\\ & =\frac {\pi}2\operatorname{arcsinh} 1\end{align*}$$

Or, in a much more friendly notation free of hyperbolic functions, $I$ is also equal as

$$\int\limits_0^1 dx\,\frac {\arctan x}{x\sqrt{1-x^2}}\color{blue}{=\frac {\pi}2\log(1+\sqrt{2})}$$

Let $\displaystyle I(a)=\int^{1}_{0}\frac{\tan^{-1}(ax)}{x\sqrt{1-x^2}}dx$

Now $\displaystyle I'(a)=\int^{1}_{0}\frac{1}{(1+a^2x^2)\sqrt{1-x^2}}dx$

Put $x=\tan t$. Then $dx=dt$ and changing limits

So $$I'(a)=\int^{\frac{\pi}{2}}_{0}\frac{1}{1+a^2\sin^2 t}dt=\frac{1}{1+a^2}\int^{\frac{\pi}{2}}_{0}\frac{\sec^2 t}{k^2+\tan^2 t}dt$$

Where $\displaystyle k^2=\frac{1}{1+a^2}$

So $\displaystyle I'(a)=\frac{1}{(1+a^2)\cdot k}\tan^{-1}\bigg(\frac{t}{k}\bigg)\bigg|^{\infty}_{0}=\frac{\pi}{2}\cdot \frac{1}{\sqrt{1+a^2}}$

So $$I(a)=\frac{\pi}{2}\ln\bigg|a+\sqrt{1+a^2}\bigg|$$

put $a=1$. Then $$I(1)=\frac{\pi}{2}\ln\bigg|1+\sqrt{2}\bigg|.$$