Closed-form expression for $F(x,y) = \int_0^1 \frac{\sqrt{t(1-t)}}{(t+x)^2 (t+y)} \mathrm{d}t$?

Let's change notation to $\displaystyle F(a,b) = \int_0^1 \frac{\sqrt{t(1-t)}}{(t+a)^2 (t+b)}\, \mathrm{d}t$ to emphasise the constant variables.

Let $\displaystyle t = \frac{u-b}{u+a} \implies \mathrm{d}t = \frac{a+b}{(u+a)^2}\,\mathrm{d}u$; moreover, this maps $[0, 1]$ to $[b, \infty)$. We've

$$\displaystyle F(a,b) = (a+b)\sqrt{a+b}\int_b^{\infty} \frac{\sqrt{u-b}}{(a b + b u - b + u) (a^2 + a u - b + u)^2}\,\mathrm{d}u $$

Now, letting $u = v^2+b$ we obtain integral of a rational function

$$\displaystyle F(a,b) ={2(a+b)\sqrt{a+b}}\int_0^{\infty} \frac{v^2}{(a^2 + a b + v^2 + a v^2)^2 (a b + b^2 + v^2 + b v^2)}\mathrm{d}v $$

So using partial fractions on the integrand we find

$$\displaystyle \frac{F(a,b)}{2(a+b)\sqrt{a+b}} = \frac{a}{a-b}I+\frac{(a+1)b}{(a-b)^2(a+b)}J -\frac{b(b+1)}{(a-b)^2(a+b)}K ~~~~~~~~~ (1)$$

where $I, J, K$ are the integrals $$\displaystyle I= \int_0^\infty \frac{1}{(a^2+ab+av^2+v^2)^2} \,\mathrm{d}v = \frac{\pi}{2} \cdot\frac{\sqrt{a (a + b)}}{2 a^2 \sqrt{1 + a} (a + b)^2}$$

$$J = \int_0^\infty \frac{1}{(a^2+ab+av^2+v^2)} \,\mathrm{d}v = \frac{\pi}{2} \cdot \frac{1}{\sqrt{a (1 + a) (a + b)}}.$$

$$K = \int_0^\infty \frac{1}{(ab+b^2+bv^2+v^2)} \, \mathrm{d}v = \frac{\pi}{2} \cdot \frac{1}{\sqrt{b (1 + b) (a + b)}}. $$

Which routinely fall-out as standard arctangent integrals. Putting these values in $(1)$ we get:

$$ F(a,b) = \frac{\pi}{2}\cdot \frac{a + b + 2 a b - 2 \sqrt{ab (1 + a) (1 + b)}}{2 \sqrt{a (1 + a)} (a - b)^2}.$$


NoName's approach is more elegant, but the integral can be evaluated using contour integration.

Consider the complex function $$g(z) = \frac{\sqrt{z} \sqrt{z-1}}{(z+x)^{2}(z+y)}$$ where $x$ and $y$ are positive parameters, $y \ne x$ , and $0 < \arg(z), \arg(z-1) < 2 \pi.$

The function is well defined on $\mathbb{C} \setminus [0,1]$ and real-valued on the real axis to the right of $z=1$.

Since $g(z) \sim \mathcal{O} \left(\frac{1}{z^{2}}\right) $ as $|z| \to \infty$, the residue of $g(z)$ at complex infinity is $0$.

If we integrate clockwise around a dog-bone/dumbbell contour, we get $$ \begin{align} \int_{0}^{1} \frac{\sqrt{t} \sqrt{(1-t)e^{i \pi}}}{(t+x)^{2}(t+y)} \, \mathrm dt + \int_{1}^{0}\frac{\sqrt{te^{ 2 \pi i}} \sqrt{(1-t)e^{i \pi}}}{(t+x)^{2}(t+y)} \, \mathrm dt &= 2i \int_{0}^{1} \frac{\sqrt{t} \sqrt{1-t}}{(t+x)^{2}(t+y)} \, \mathrm dt \\ &=2 \pi i \left(\operatorname{Res}[g(z), -x] + \operatorname{Res}[g(z), -y] \right), \end{align}$$

where $$\operatorname{Res}[g(z), -y] = \frac{\sqrt{y e^{ i \pi}} \sqrt{(y+1) e^{ i \pi }}}{(-y+x)^{2}} = -\frac{\sqrt{y(1+y)}}{(x-y)^{2}} $$

and $$ \begin{align} \operatorname{Res}[g(z), -x] &= \lim_{ z \to -x} \frac{\mathrm d}{\mathrm dz} \frac{\sqrt{z}\sqrt{z-1}}{z+y} \\ &= \lim_{ z \to -x} \frac{\left(\frac{\sqrt{z-1}}{2 \sqrt{z}} + \frac{\sqrt{z}}{2\sqrt{z-1}}\right)(z+y) - \sqrt{z} \sqrt{z-1} }{(z+y)^{2}} \\ &= \frac{\left(\frac{\sqrt{(x+1)e^{ i \pi}}}{2 \sqrt{xe^{ i \pi}}} + \frac{\sqrt{xe^{ i \pi}}}{2\sqrt{(x+1)e^{ i \pi}}}\right)(-x+y) - \sqrt{xe^{i \pi}} \sqrt{(x+1)e^{ i \pi}} }{(-x+y)^{2}}\\ &= \frac{\left(\frac{\sqrt{x+1}}{2 \sqrt{x}} + \frac{\sqrt{x}}{2 \sqrt{x+1}} \right)(y-x) + \sqrt{x}\sqrt{x+1}}{(x-y)^{2}} \\ &=\frac{(2x+1)(y-x) + 2x(1+x) }{2 \sqrt{x(1+x)}(x-y)^{2}} \\&= \frac{x+y+2xy}{2 \sqrt{x(1+x)}(x-y)^{2}}. \end{align} $$

Therefore, $$\int_{0}^{1} \frac{\sqrt{t} \sqrt{1-t}}{(t+x)^{2}(t+y)} \, \mathrm dt = \frac{\pi}{2} \frac{x+y+2xy-2 \sqrt{xy(1+x)(1+y)}}{\sqrt{x(1+x)}(x-y)^{2}}. $$


The result should hold for $y=0$, but the case $y=x$ has to be done separately.