Why are variance and expected value all we care about?

The expected value and the variance describe the central trend and the spread. These are the minimal quantities one cares about for a first grasp of the data. They are sufficient to define a linear transformation of the data to normalize it (nonlinear transformations aren't so prized).

Higher moments (classically up to order $4$) can be used for a finer description (symmetry and resemblance to a normal law) but they are much less useful, and the tools for their estimation are much less developed.

Also think that the mean and standard deviation are all it takes to describe a Normal distribution, and by the CLT, this distribution is the "destiny" of all samples.