Why is Gauss credited with this connection?

I've heard from a reliable source that Grothendieck decided on this name because he wanted to include the first person who discovered the thing and the last person who made an important contribution:-) (Of course, Grothendieck knew that there were many in between).

However, as it frequently happens, this attribution of the first discovery is not correct. The fact that periods of an elliptic integral satisfy a hypergeometric equation was discovered by Legendre. Actually several important discoveries of Legendre are attributed to Gauss, (for example, quadratic reciprocity law, and the method of least squares) and Legendre bitterly complained about this himself.

Also the discovery of the arithmetic geometric mean is universally credited to Gauss, while Lagrange discovered it and published 6 years earlier, and Legendre used it to compute elliptic integrals.

The short answer is we call it the Gauss-Manin connection because that's what Grothendieck called it. The name is attributed to Grothendieck in two early, seminal pieces: namely, Katz's thesis and a subsequent article of his ("On the differentiation of De Rham cohomology classes with respect to parameters", Katz & Oda).

If you can read (some) French, look through Yves Andre's chapter in "Geometric Aspects of Dwork Theory". If not, check through the start of this arxiv paper (through the first couple paragraphs of 1.2).

The slightly less short answer is the one Gerhard Paseman alluded to above; quoting from the aforelinked arxiv paper ("Towards a nonlinear Schwarz’s list", Philip Boalch):

"One reason hypergeometric equations are interesting is that they provide the simplest explicit examples of Gauss–Manin connections. Indeed this is one reason Gauss was interested in them: he observed that the periods of a family of elliptic curves satisfy a (Gauss) hypergeometric equation. (The modern interpretation of this is as the explicit form of the natural flat connection on the vector bundle of first cohomologies over the base of the family of elliptic curves, written with respect to the basis given by the holomorphic one forms—and their derivatives—on the fibres.) Nowadays there is still much interest in such linear differential equations “coming from geometry”.

Thus the nonlinear analogue of the Gauss hypergeometric equation should be the explicit form of the simplest nonabelian Gauss–Manin connection (i.e the explicit form of the natural connection on the bundle of first nonabelian cohomologies of some family of varieties). The simplest interesting case corresponds to taking the universal family of four punctured spheres and taking cohomology with coefficients in $SL_2(\mathbb{C})$ (one needs a non-trivial family of varieties with nonabelian fundamental groups)."