Simplifying this long expression

The result is 0, and we can show it by a smart choice of transformation. expr is what I define as the expression that OP wants to simplify:

expr=Sin[t] (b^2 Cos[t] (1/Sqrt[(Sqrt[a^2-b^2]-a Cos[t])^2+b^2 Sin[t]^2]+1/Sqrt[(Sqrt[a^2-b^2]+a Cos[t])^2+b^2 Sin[t]^2])-a ((-Sqrt[a^2-b^2]+a Cos[t])/Sqrt[(Sqrt[a^2-b^2]-a Cos[t])^2+b^2 Sin[t]^2]+(Sqrt[a^2-b^2]+a Cos[t])/Sqrt[(Sqrt[a^2-b^2]+a Cos[t])^2+b^2 Sin[t]^2]))

Clearly FullSimplify does not help even with OP's assumptions:

FullSimplify[expr, a > b > 0 && t > 0]

$\sin(t) \left(b^2 \cos (t) \left(\frac{1}{\sqrt{\left(\sqrt{(a-b) (a+b)}-a \cos (t)\right)^2+b^2 \sin ^2(t)}}+\frac{1}{\sqrt{\left(\sqrt{(a-b) (a+b)}+a \cos (t)\right)^2+b^2 \sin ^2(t)}}\right)-a \left(\frac{a \cos (t)-\sqrt{(a-b) (a+b)}}{\sqrt{\left(\sqrt{(a-b) (a+b)}-a \cos (t)\right)^2+b^2 \sin ^2(t)}}+\frac{\sqrt{(a-b) (a+b)}+a \cos (t)}{\sqrt{\left(\sqrt{(a-b) (a+b)}+a \cos (t)\right)^2+b^2 \sin ^2(t)}}\right)\right)$

We first use the fact that $a>b>0$ to reparametrize $a$ as $b/\sin(k)$ for $\pi/2>k>0$:

expr2=FullSimplify[expr /. a -> b/Sin[k], b > 0 && Pi/2 > k > 0 && t > 0]

$b \cot (k) \sin (t) (\text{sgn}(\csc (k)-\cos (t) \cot (k))-\text{sgn}(\cos (t) \cot (k)+\csc (k)))$

We observe that the expression simplified significantly. Let us now insert back the parameter $a$:

FullSimplify[expr2 /. k -> ArcSin[b/a], a > b > 0 && t > 0]

0


Not an answer, as I could not figure out why, just to confirm that it should be zero

Clear[a, b, c, t, e, r]
L[vektor_] := Sqrt[Total[vektor^2]];
r = {a Cos[t], b Sin[t]}
e = {Sqrt[a^2 - b^2], 0};
c = ((r - e)/L[r - e] + (r + e)/L[r + e]).D[r, t]

Mathematica graphics

Manipulate[
 Plot[c /. {a -> a0, b -> b0}, {t, -200, 200}],
 {{a0, 1, "a"}, -100, 100, 1},
 {{b0, 1, "b"}, -100, 100, 1},
 TrackedSymbols :> {a0, b0}
 ]

Mathematica graphics

And using Chop shows it is zero for any choice of a,b. Reduce also not able to help. FullSimplify did not help either. I think you got Mathematica stumbled on this one.

the only way I could get it to give zero, it to give it bad assumption

 Simplify[c, Sqrt[a^2 - b^2] < 0]
 (* 0 *)

But the above assumption is not correct, since Sqrt[a^2 - b^2] < 0 means complex number is less than 0. But < does not apply to complex numbers, only to real numbers.

Tried Maple's version of Reduce and Maple says it can be zero. Copied the expression to Maple first.

Mathematica graphics


You say, you know, that c should be zero. Let Reduce test whether it can be unequal zero.

Reduce[{a > b, b > 0, t > 0, c != 0}, {a, b, t}]

(*   False   *)

Edit Another way to show c == 0

Substitute the a-b-squareroute by d and take the b-solution that is > 0.

sol = Solve[Sqrt[a^2 - b^2] == d, b]

(*   {{b -> -Sqrt[a^2 - d^2]}, {b -> Sqrt[a^2 - d^2]}}   *)

c /. sol[[2]] // FullSimplify[#, {a > d > 0, t > 0}] &

(*   0   *)