Factoring the square of this polynomial?

Try

Simplify[1 - (2 b^2)/a^2 + b^4/a^4 /. a ->  b/λ] /. λ -> b/a
(*(-1 + b^2/a^2)^2*)

Explanation of the problem

The reason it never reaches the form you want despite of being minimal by LeafCount is that that form is never tried, so its LeafCount is never measured nor compared. Look at the attempts with a fresh kernel (as Simplify is cashed).

So the solution is to find a way for Simplify to explore the form you want using TransformationFunctions and score it as optimal as per your definition ComplexityFunction -> LeafCount.

{sol, {attempts}} = Reap@Simplify[
   1 - (2 b^2)/a^2 + b^4/a^4
   , ComplexityFunction -> ((Sow[#]; LeafCount[#]) &)
   ];

Solution requested: Simplify

We use TransformationFunctions to tell Simplify to explore the form you want, and ComplexityFunction to define how these alternatives are scored.

One way is to use the CompleteSquare described below

Simplify[
 1 - (2 b^2)/a^2 + b^4/a^4
 , TransformationFunctions -> {Automatic, CompleteSquare[#, b^2] &}
 , ComplexityFunction -> LeafCount
 ]

(-1 + b^2/a^2)^2

Or based on the answer by Ulrich Neumann, also

transf[expr_] := Module[
  {vars, tvar},
  vars = Variables[expr];
  ReplaceAll[
   Simplify@ReplaceAll[expr, vars[[1]] -> vars[[2]]/tvar]
   , tvar -> Divide @@ vars[[{2, 1}]]
   ]
  ]

Simplify[
 1 - (2 b^2)/a^2 + b^4/a^4
 , TransformationFunctions -> {Automatic, transf}
 , ComplexityFunction -> LeafCount
 ]
(* (-1 + b^2/a^2)^2 *)

PolynomialForm

The undocumented PolynomialForm allows placing the terms in the desired order with the option TraditionalOrder->True.

PolynomialForm[(-1+b^2/a^2)^2,TraditionalOrder->True]
(* (b^2/a^2-1)^2 *)

or

Format[bda] = DisplayForm@FractionBox["b", "a"];

PolynomialForm[
 (-1 + b^2/a^2)^2 /. b -> bda a
 , TraditionalOrder -> True
 ]

Alternative Solution: CompleteSquare

Other people have suggested way to complete the square. You can force that form by

CompleteSquare[f_, x_] := Module[
  {a, b, c},
  {c, b, a} = CoefficientList[f, x];
  Assuming[
   Sqrt[a] > 0,
    (FullSimplify[Sqrt[a] x] + 
       FullSimplify[b/(2 Sqrt[a])])^2 + (FullSimplify[(a c - b^2/4)])
   ]]

CompleteSquare[1 - (2 b^2)/a^2 + b^4/a^4, b^2]
(* (-1 + b^2/a^2)^2 *)

You can do

Factor[1 - (2 b^2)/a^2 + b^4/a^4] // FullSimplify

or (as suggested by @Thies Heidecke)

FullSimplify[1 - (2 b^2)/a^2 + b^4/a^4]

with identical output: $$\frac{\left(a^2-b^2\right)^2}{a^4}$$

which is close.