How do I recursively calculate this equation and generate a list of iteration?

You were correct. NestList is exactly the function you want to use.

NestList[Dot[A, #]&, x0, 5]

(* {{2, 0}, {1., 1.5}, {-0.4, 2.4}, {-1.64, 2.34}, {-2.224, 
   1.344}, {-1.9184, -0.1896}} *)

Note that the first argument of NestList must be a function.

Your can use MatrixPower for this example:

f[n_] := MatrixPower[{{.5, -.6}, {.75, 1.1}}, n].{2, 0}
f /@ Range[0, 5]

yields:

{{2., 0.}, {1., 1.5}, {-0.4, 2.4}, {-1.64, 2.34}, {-2.224, 
  1.344}, {-1.9184, -0.1896}}

You can also do this recursively, very nearly as you wrote it:

a[k_] := a[k] = A.a[k - 1];
a[1] = x0;

Now you can calculate any desired iterate by asking for a[5] or a[10]. Or calculate a range of values:

a[#] & /@ Range[5]
{{2, 0}, {1., 1.5}, {-0.4, 2.4}, {-1.64, 2.34}, {-2.224, 1.344}}

Related