Proving $\int_{\sqrt{\frac{3}{5}}}^1 \frac{\arctan (x)}{\sqrt{2 x^2-1} \left(3 x^2-1\right)} \, dx=\frac{3 \pi ^2}{160}$
Two proofs will be given. One proof is the remaining of this answer, along lines of Schläfli and Coxeter. Second proof, more direct, is given at this answer's remark.
Note that $$\int_{\sqrt {3/5} }^1 {\frac{{\arctan x}}{{\sqrt {2{x^2} - 1} (3{x^2} - 1)}}dx} = \int_1^{\sqrt {5/3} } {\frac{{x(\frac{\pi }{2} - \arctan x)}}{{\sqrt {2 - {x^2}} (3 - {x^2})}}dx} = \frac{{{\pi ^2}}}{{24}} - \frac{1}{2}\int_1^{5/3} {\frac{{\arctan \sqrt x }}{{\sqrt {2 - x} (3 - x)}}dx}$$ therefore your integral is equivalent to $$\int_1^{5/3} {\frac{{\arctan \sqrt x }}{{\sqrt {2 - x} (3 - x)}}dx} = \frac{{11{\pi ^2}}}{{240}}$$ I will prove this by establishing $$\tag{1}\int_0^1 {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = \frac{{{\pi ^2}}}{{48}}$$ $$\tag{2} \int_0^{5/3} {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = \frac{{{\pi ^2}}}{{15}}$$
The following ideas are an expounded version of Polylogarihm and Associated Functions by Leonard Lewin page 115-117, arguments there are very terse.
Let $$S(\alpha ,\beta ,\gamma ) = \sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{{{n^2}}}(\cos 2n\alpha - \cos 2n\beta + \cos 2n\gamma - 1)} - {\alpha ^2} + {\beta ^2} - {\gamma ^2}$$ where $0\leq \alpha,\gamma \leq \pi/2, 0\leq \beta < \pi$ and $k$ is a function of $\alpha,\beta,\gamma$ which will be determined soon. It is easy to check that $$\frac{{\partial S}}{{\partial \alpha }} = - 2\sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{n}\sin 2n\alpha } - 2\alpha = - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)$$ Therefore the differential $dS$ is $$\begin{aligned}dS &= \frac{1}{k}\sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{n}(\cos 2n\alpha - \cos 2n\beta + \cos 2n\gamma - 1)} dk - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)d\alpha+\\ & 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \beta } \right)d\beta - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \gamma } \right)d\gamma \end{aligned}$$ Now choose $k$ such that the coefficient of $dk$ vanishes, using $\sum\limits_{n = 1}^\infty {\frac{{{a^n}\cos nx}}{n}} = - \frac{1}{2}\ln ({a^2} - 2a\cos x + 1)$ one easily see such $k$ is $$ k = \frac{{\sqrt {{{\cos }^2}\alpha {{\cos }^2}\gamma - {{\cos }^2}\beta } - \sin \alpha \sin \gamma }}{{\sqrt {{{\cos }^2}\alpha {{\cos }^2}\gamma - {{\cos }^2}\beta } + \sin \alpha \sin \gamma }}$$ This completes the definition of $S(\alpha,\beta,\gamma)$. Note that in order for $k$ to be real, we need to assume the term inside radical is always $\geq 0$, we confine ourselves exclusive to this case. Now $dS$ becomes $$\tag{3}dS = - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)d\alpha+ 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \beta } \right)d\beta - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \gamma } \right)d\gamma $$
Four observations:
- $S(0,\beta,\gamma) = \pi(\beta-\gamma)$
- When $\sin^2 \alpha + \sin^2 \gamma = \sin^2 \beta$, $S(\alpha,\beta,\gamma) = -\alpha^2+\beta^2-\gamma^2$
- When $\cos \alpha \cos\gamma = \cos\beta$, $S(\alpha,\beta,\gamma)= 0$.
- $S(\alpha,\pi - 2\alpha,\alpha) = 6S(\alpha,\pi/3,\pi/6)$
Proof. For the first one, $\alpha = 0$ implies $k=1$, so $$S(0,\beta ,\gamma ) = \sum\limits_{n = 1}^\infty {\frac{1}{{{n^2}}}( - \cos 2n\beta + \cos 2n\gamma )} + {\beta ^2} - {\gamma ^2} = \pi (\beta - \gamma )$$ For second one, the condition implies $k=0$. For third one, the condition implies $k=-1$, then just evaluate the series. The fourth assertion is more crucial. We prove it carefully. By first bullet, it suffices to prove their derivative with respect to $\alpha$ are equal. When $\beta = \pi/3, \gamma = \pi/6$, one sees that $$\frac{{1 + k}}{{1 - k}}\tan \alpha = \sqrt{3 \cos^2 \alpha-1} \sec \alpha $$ Since $\beta,\gamma$ are constants, $d\gamma = d\beta = 0$, $(3)$ shows $$\tag{A}-\frac{1}{2}\frac{dS(\alpha,\pi/3,\pi/6)}{d\alpha} = \arctan(\sqrt{3 \cos^2 \alpha-1} \sec \alpha)$$
Now consider $S(\alpha,\pi - 2\alpha,\alpha)$, with $\beta = \pi - 2\alpha, \gamma = \alpha, d\beta = -2d\alpha$, $d\gamma = d\alpha$, one computes via $(3)$, $$\tag{B}-\frac{1}{2}\frac{dS(\alpha,\pi - 2\alpha,\alpha)}{d\alpha} = 2 \arctan \left(\sqrt{\cos ^4\alpha-\cos ^2 2\alpha} \csc \alpha \sec \alpha\right)-2 \arctan \left(\sqrt{\cos ^4 \alpha-\cos ^2 2 \alpha} \tan 2 \alpha \csc ^2 \alpha \right)+2\pi $$ To complete the proof, it suffices to differentiate RHS of $6\times (A), (B)$, and see whether they are equal. This becomes a trivial but computational heavy task.
Now let $a>b$, consider $$\sqrt {a - b} \int_0^x {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {b - t} (a - t)}}dt} = - 2\int_0^x {{{\tan }^{ - 1}}\sqrt t d({{\tan }^{ - 1}}\sqrt {\frac{{b - t}}{{a - b}}} )} $$ fix $\alpha = {\tan ^{ - 1}}\sqrt {\frac{{b - t}}{{a - b}}}$, we find $\beta,\gamma$ (depends on $a,b$ but not on $t$) such that $$\frac{{1 + k}}{{1 - k}}\tan \alpha = \sqrt t $$ One easily verifies one such pair $\beta,\gamma$ is $$\gamma = {\tan ^{ - 1}}\frac{1}{{\sqrt a }}\qquad \beta = {\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} $$ Since $\beta,\gamma$ are independent of $t$, $(3)$ implies $$dS(\alpha ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }}) = - 2{\tan ^{ - 1}}\left( {\frac{{1 + k}}{{1 - k}}\tan \alpha } \right)d\alpha = - 2{\tan ^{ - 1}}\sqrt t d\alpha $$ Hence $$\int_0^x { - 2{{\tan }^{ - 1}}\sqrt t d\alpha } = S({\tan ^{ - 1}}\sqrt {\frac{{b - x}}{{a - b}}} ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }}) + C$$ for a constant $C$ independent of $x$. Set $x=0$, then one checks the third bullet point applies for $$S({\tan ^{ - 1}}\sqrt {\frac{b}{{a - b}}} ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }})$$ therefore it is $0$, hence $C=0$. Thus we proved
$$\tag{4}\int_0^x {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{(a - t)\sqrt {b - t} }}dt} = \frac{1}{{\sqrt {a - b} }}S({\tan ^{ - 1}}\sqrt {\frac{{b - x}}{{a - b}}} ,{\tan ^{ - 1}}\sqrt {\frac{{b + 1}}{{a - b}}} ,{\tan ^{ - 1}}\frac{1}{{\sqrt a }})$$
Now let $a=3, b=2$, we deduce from $(4)$ $$\int_0^1 {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = S(\frac{\pi }{4},\frac{\pi }{3},\frac{\pi }{6})\qquad \int_0^{5/3} {\frac{{{{\tan }^{ - 1}}\sqrt t }}{{\sqrt {2 - t} (3 - t)}}dt} = S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6})$$ Now the second bullet applies for the former one, so $S(\frac{\pi }{4},\frac{\pi }{3},\frac{\pi }{6}) = \frac{{{\pi ^2}}}{{48}}$, this is $(1)$. For latter one, note that fourth bullet implies $$6S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6}) = S(\frac{\pi }{6},\frac{{2\pi }}{3},\frac{\pi }{6})$$ but directly from definition (the $k$ associated with these two pairs are equal), one sees that $$S(\frac{\pi }{6},\frac{{2\pi }}{3},\frac{\pi }{6}) - S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6}) = \sum\limits_{n = 1}^\infty {\frac{{{k^n}}}{{{n^2}}}(\underbrace{\cos \frac{{2\pi n}}{3} - \cos \frac{{4\pi n}}{3}}_{=0})} + {(\frac{{2\pi }}{3})^2} - {(\frac{\pi }{3})^2} = \frac{\pi^2}{3}$$ thus $S(\frac{\pi }{6},\frac{\pi }{3},\frac{\pi }{6}) = \frac{{{\pi ^2}}}{{15}}$, this is $(2)$. The integral claimed by OP is now established.
This is a partial solution which transformed the original integral into an Ahmed-like integral. Firstly, substitute $y\to \sqrt{2 x^2-1}$ and introduce a parameter $a$ into $\tan ^{-1}\left(a \sqrt{\frac{1}{2} \left(x^2+1\right)}\right)$. Differentiate w.r.t $a$, the integrand will be a rational function. After integration w.r.t $y$, we have $I=-A+B+\frac{C \left(\pi -3 \tan ^{-1}\left(\sqrt{\frac{3}{5}}\right)\right)}{\sqrt{3}}$ where $\small A=\int_0^1 \frac{x \tan ^{-1}\left(\frac{x}{\sqrt{x^2+2}}\right)}{\sqrt{x^2+2} \left(x^2+3\right)} \, dx=\frac{\pi ^2}{288},$ $\small B=\int_0^1 \frac{x \tan ^{-1}\left(\frac{x}{\sqrt{5} \sqrt{x^2+2}}\right)}{\sqrt{x^2+2} \left(x^2+3\right)} \, dx,$ $ C=\int_0^1 \frac{1}{x^2+3} \, dx=\frac{\pi }{6 \sqrt{3}}$. For $A$, integrate by parts using $\small \int \frac{x}{\sqrt{x^2+2} \left(x^2+3\right)} \, dx=\tan ^{-1}\left(\sqrt{x^2+2}\right)$ it reduces to the original Ahmed integral. Apply the same method to $B$, I arrive at corresponding $\small B'=\int_0^1 \frac{\tan ^{-1}\left(\sqrt{x^2+2}\right)}{\sqrt{x^2+2} \left(3 x^2+5\right)} \, dx$ whose value should be $\small\frac{1}{5} \sqrt{5} \left(\frac{\pi ^2}{30}-\frac{1}{6} \pi \tan ^{-1}\left(\sqrt{\frac{3}{5}}\right)+\frac{1}{3} \pi \tan ^{-1}\left(\sqrt{\frac{1}{15}}\right)\right)$ due to the conjectural result. So now we only need to justify the value of $B'$.
Update: According to the link FDP provided under pisco's answer, we are able to evaluate more nontrivial Coxeter integrals such as $\int_0^{\frac{\pi }{5}} \cos ^{-1}\left(\frac{\cos (x)}{2 \cos (x)+1}\right) \, dx=\frac{71 \pi ^2}{900}$. Based on $S$ function's representation as well as Feynman's trick, an elegant formula is found:
- $\small \int_{\frac{1}{\sqrt{y}}}^{\frac{1}{\sqrt{x}}} \frac{\arctan (x)}{\sqrt{2 x^2-1} \left(3 x^2-1\right)} \, dx=\frac{1}{2} \pi \left(\tan ^{-1}\left(\sqrt{2-y}\right)-\tan ^{-1}\left(\sqrt{2-x}\right)\right)+\frac{1}{2}\left(S\left(\tan ^{-1}\left(\sqrt{2-y}\right),\frac{\pi }{3},\frac{\pi }{6}\right)-S\left(\tan ^{-1}\left(\sqrt{2-x}\right),\frac{\pi }{3},\frac{\pi }{6}\right)\right)=\left(\sqrt{y (2-y)} A(y)-\sqrt{x (2-x)} A(x)\right)+\frac{1}{6} \pi \left(\tan ^{-1}\left(\sqrt{\frac{3 (2-y)}{y}}\right)-\tan ^{-1}\left(\sqrt{\frac{3 (2-x)}{x}}\right)\right)-\frac{1}{3} \pi \left(\tan ^{-1}\left(\sqrt{\frac{2-y}{3 y}}\right)-\tan ^{-1}\left(\sqrt{\frac{2-x}{3 x}}\right)\right)$
Where function $S$ is the one defined in pisco's answer, $A$ the generalized Ahmed integral:
- $A(t)=\int_0^1 \frac{\tan ^{-1}\left(\sqrt{x^2+2}\right)}{\sqrt{x^2+2} \left(t+x^2\right)} \, dx,A(1)=\frac{5 \pi ^2}{96}$
Using the original Ahmed integral and special value of $S$, one may let $x\to1$ and assign special values to $y$ to evaluate, say
- $\small \int_0^1 \frac{\tan ^{-1}\left(\sqrt{x^2+2}\right)}{\sqrt{x^2+2} \left(x^2+\frac{2 \sqrt{5}}{5}+1\right)} \, dx=\sqrt{5} \left(\frac{71 \pi ^2}{3600}+\frac{1}{3} \pi \tan ^{-1}\left(\sqrt{\frac{1}{3} \left(9-4 \sqrt{5}\right)}\right)-\frac{1}{6} \pi \tan ^{-1}\left(\sqrt{27-12 \sqrt{5}}\right)\right)$
- $\small \int_0^1 \frac{\tan ^{-1}\left(\sqrt{x^2+2}\right)}{\sqrt{x^2+2} \left(x^2-\frac{2 \sqrt{5}}{5}+1\right)} \, dx=\sqrt{5} \left(\frac{241 \pi ^2}{3600}+\frac{1}{3} \pi \tan ^{-1}\left(\sqrt{\frac{1}{3} \left(4 \sqrt{5}+9\right)}\right)-\frac{1}{6} \pi \tan ^{-1}\left(\sqrt{12 \sqrt{5}+27}\right)\right)$
From which we deduce the final one (via PFD), a remarkable quartic Ahmed integral:
- $\int_0^1 \frac{\tan ^{-1}\left(\sqrt{x^2+2}\right)}{\sqrt{x^2+2} \left(5 x^4+10 x^2+1\right)} \, dx=\frac{37 \pi ^2}{1440}$
Some substitutions which I think make the integral more simple.
$$I=\int_{\sqrt{\frac{3}{5}}}^1 \frac{\arctan (x)}{\sqrt{2 x^2-1} \left(3 x^2-1\right)} \, dx$$
$$x=\frac{1}{y}$$
$$I=\int_1^{\sqrt{\frac{5}{3}}} \frac{y \arctan \frac{1}{y}}{\sqrt{2 -y^2} \left(3-y^2\right)} \, dy$$
$$y=\sqrt{2} z$$
$$I=\frac{\sqrt{2}}{3} \int_{\frac{1}{\sqrt{2}}}^{\sqrt{\frac{5}{6}}} \frac{z \arctan \frac{1}{\sqrt{2} z}}{\sqrt{1 -z^2} \left(1-\frac23 z^2\right)} \, dz$$
$$z^2=u$$
$$I=\frac{\sqrt{2}}{6} \int_{\frac{1}{2}}^{\frac{5}{6}} \frac{\arctan \frac{1}{\sqrt{2 u} }}{\sqrt{1 -u} \left(1-\frac23 u\right)} \, du$$
$$u= \frac{1+s}{2}$$
$$I=\frac{1}{4} \int_{0}^{\frac{2}{3}} \frac{\arctan \frac{1}{\sqrt{1+s} }}{\sqrt{1 -s} \left(1-\frac12 s\right)} \, ds$$
So we need to prove that:
$$J=\int_{0}^{\frac{2}{3}} \frac{\arctan \frac{1}{\sqrt{1+s} }}{\sqrt{1 -s} \left(1-\frac12 s\right)} \, ds= \frac{3 \pi^2}{40}$$
Let's try integration by parts. Turns out that:
$$ \int \frac{ds}{\sqrt{1 -s} \left(1-\frac12 s\right)}=-4 \arctan \sqrt{1-s}$$
$$\frac{d}{ds} \arctan \frac{1}{\sqrt{1+s} }=-\frac{1}{4} \frac{ds}{\sqrt{1 +s} \left(1+\frac12 s\right)}$$
So our integral is equal to:
$$J=-4 \arctan\frac{1}{\sqrt{1+s} } \arctan \sqrt{1-s} \bigg|_0^{2/3}-\int_{0}^{\frac{2}{3}} \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(1+\frac12 s\right)} \, ds$$
$$J=\frac{\pi^2}{4}-\frac{2 \pi}{3} \arctan \sqrt{\frac{3}{5}} -\int_{0}^{\frac{2}{3}} \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(1+\frac12 s\right)} \, ds$$
Maybe this symmetry could help.
Substituting $s \to -s$ we have:
$$J=\int_{-\frac{2}{3}}^0 \frac{\arctan \frac{1}{\sqrt{1-s} }}{\sqrt{1 +s} \left(1+\frac12 s\right)} \, ds$$
$$\arctan \frac{1}{\sqrt{1-s} }= \frac{\pi}{2}-\arctan \sqrt{1-s}$$
$$J= \frac{\pi}{2}\int_{-\frac{2}{3}}^0 \frac{ds}{\sqrt{1 +s} \left(1+\frac12 s\right)} -\int_{-\frac{2}{3}}^0 \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(1+\frac12 s\right)} \, ds$$
$$J= \frac{\pi^2}{6} -\int_{-\frac{2}{3}}^0 \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(1+\frac12 s\right)} \, ds$$
Adding the two expressions for $J$ we obtain:
$$2J=\frac{5\pi^2}{12}-\frac{2 \pi}{3} \arctan \sqrt{\frac{3}{5}} -\int_{-\frac{2}{3}}^{\frac{2}{3}} \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(1+\frac12 s\right)} \, ds$$
$$J=\frac{5\pi^2}{24}-\frac{\pi}{3} \arctan \sqrt{\frac{3}{5}} -\int_{-\frac{2}{3}}^{\frac{2}{3}} \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(2+s\right)} \, ds$$
Again, the symmetry might help with the last integral.
So now we need to show:
$$Y=\int_{-\frac{2}{3}}^{\frac{2}{3}} \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(2+s\right)} \, ds=\frac{2\pi^2}{15}-\frac{\pi}{3} \arctan \sqrt{\frac{3}{5}}$$
Note that a related integral (from numerical results):
$$\int_{-1}^{1} \frac{\arctan \sqrt{1-s}}{\sqrt{1 +s} \left(2+s\right)} \, ds= \frac{\pi^2}{6}$$
Update:
Turning arctangent into an integral, substituting $s=\sin \theta= \frac{2t}{1+t^2}$ and then integrating a rational integral w.r.t. $t$ (with Mathematica's help), I have made yet another form of the conjecture:
Prove that: $$\int_0^1 \frac{\arctan \frac{2 \sqrt{1+2 p^2}}{\sqrt{5} (1+p^2)}}{\sqrt{1+2 p^2} (1+3 p^2)} dp= \frac{\pi}{2} \arctan \sqrt{\frac{3}{5}}- \frac{\pi^2}{15}$$
This one looks more complicated, but at least the limits are nice.
The integral is similar to $B$ from Fengshan Xiong's solution.