How find this integral $I=\int_{0}^{\frac{\pi}{2}}(\ln{(1+\tan^4{x})})^2\frac{2\cos^2{x}}{2-(\sin{(2x)})^2}dx$

Here is an approach. I'll continue from the step you have already reached with the correction suggested in Fabien comment which is considering the integral

$$ I = \int_{0}^{\infty} \frac{\ln^2(1+x^4)}{1+x^4} dx. $$

To evaluate the above integral we consider the integral

$$ F = \int_{0}^{\infty} (1+x^4)^{\alpha} dx = \frac{\pi}{2\sqrt{2}\Gamma( 3/4 )}{\frac {\Gamma ( -1/4-\alpha ) }{ \Gamma \left( -\alpha \right) }},$$

which can be evaluated using the beta function techniques (use the substitution $(1+x^4)=\frac{1}{t}$). Now $I$ follows from $F$ as (see related techniques )

$$ I = \lim_{\alpha \to -1}\frac{d^2F(\alpha)}{d\alpha^2}=\frac{\pi \,\sqrt {2}}{48}\left({\pi}^{2}-36\,\pi \,\ln \left( 2 \right) +108 \ln^2( 2 )+12\,\psi'\left( 3/4 \right) \right) .$$

Hint: $\quad~I(n)~=~\displaystyle\int_0^\infty\Big(1+x^k\Big)^n~dx\quad~=>\quad~I''(-1)~=~\displaystyle\int_0^\infty\frac{\ln^2\big(1+x^k\big)}{1+x^k}~dx.~$ But, at

the same time, $I(n)$ can be shown to equal $~\dfrac1k\cdot B\bigg(\dfrac1k~,-\dfrac1k-n\bigg),~$ with the help of the simple

substitution $t=\dfrac1{1+x^k}$ . Then, expressing it in terms of the $\Gamma$ function, differentiating, and

using the relationship between the polygamma function $\psi_0$ and harmonic numbers, along with

Euler's formula for their generalization to non-natural arguments, we are thus finally able to

arrive at an expression whose only “mysterious” quantity is $\psi_{_1}\Big(\frac34\Big).~$ Since this function has

been studied for several centuries, I'm sure you will be able to find its closed form expression

somewhere, along with the proof behind it. Also, Euler's reflection formula for the $\Gamma$ function

might be of some assistance.

After performing the initial arctangent substitution, the resultant integral may be rewritten as the second derivative of a beta function and evaluated accordingly:

$$\begin{align} I &=\int_{0}^{\frac{\pi}{2}}\frac{2\cos^2{\theta}}{2-\sin^2{(2\theta)}}\log^2{\left(1+\tan^4{\theta}\right)}\,\mathrm{d}\theta\\ &=\int_{0}^{\infty}\frac{\log^2{(1+x^4)}}{1+x^4}\mathrm{d}x\\ &=\frac{d^2}{d\alpha^2}\bigg{|}_{\alpha=1}\int_{0}^{\infty}\frac{\mathrm{d}x}{(1+x^4)^{\alpha}}\\ &=\frac{d^2}{d\alpha^2}\bigg{|}_{\alpha=1}\int_{0}^{\infty}\frac{\frac14u^{-\frac34}}{(1+u)^{\alpha}}\mathrm{d}u\\ &=\frac14\frac{d^2}{d\alpha^2}\bigg{|}_{\alpha=1}\operatorname{B}{\left(\frac14,\alpha-\frac14\right)}\\ &=\frac14\Gamma{\left(\frac14\right)}\Gamma{\left(\frac34\right)}\left[-\zeta{(2)}+\left(\gamma+\Psi{\left(\frac34\right)}\right)^2+\Psi^\prime{\left(\frac34\right)}\right]\\ &=\frac{\sqrt{2}\,\pi}{4}\left[-\frac{\pi^2}{6}+\left(\frac{\pi}{2}-\log{8}\right)^2+\pi^2-8C\right]\\ &=-2\sqrt{2}\,\pi\,C+\frac{13\pi^3}{24\sqrt{2}}+\frac{\pi\log^2{8}}{2\sqrt{2}}-\frac{\pi^2\log{8}}{2\sqrt{2}}.~~~\blacksquare \end{align}$$