Differentiation and series expansion of dot product - inconsistent results

This has been confirmed as a bug by Wolfram support.

(Bug still present in 11.0.0)

You can muck about with an internal function to get Series to work a bit better on Dot (and Cross etc) products. For example:

protect = Unprotect[System`Private`InternalSeries];
System`Private`InternalSeries[a_Dot, {x_, x0_, n_Integer?NonNegative}] := Module[
    {d = NestList[D[#, x]&, a, n], res},

    res = Quiet @ Check[d /. x->x0, $Failed];
    SeriesData[x, x0, TensorExpand @ res, 0, n+1, 1] /; res =!= $Failed
]
Protect @@ protect;

Now, your example works correctly:

SeriesCoefficient[a.b[x],{x,0,1}]

a.b'[0]

A more complicated example:

Series[Exp[a[x].b[x]+x^2], {x, 0, 2}] //TeXForm

$e^{a(0).b(0)}+x e^{a(0).b(0)} \left(a'(0).b(0)+a(0).b'(0)\right)+\frac{1}{2} x^2 e^{a(0).b(0)} \left(\left(a'(0).b(0)+a(0).b'(0)\right)^2+2 \left(a''(0).b(0)+2 a'(0).b'(0)+a(0).b''(0)+1\right)\right)+O\left(x^3\right)$

Where this modification won't help is when the series is not a Taylor or MacLaurin series, e.g., a Laurent or Puiseaux series.