Intuition on variance of linear combination of variables

Other answers have addressed derivation, but you asked for intuition:

Variance is a measure of uncertainty. The higher the variance the greater the uncertainty. Uncertainty is additive. If $X_1$ and $X_2$ are uncorrelated, the uncertainty in their sum is the sum of their uncertainties: $Var(X_1 + X_2) = Var(X_1) + Var(X_2)$. But what if they are correlated? Two things can happen. If they are positively correlated this will increase the uncertainty since this makes it more likely that observations of the two variables will pull away from the means in the same direction, pulling the sum even farther away from its mean. On the other hand, if they are negatively correlated, they will tend to pull in opposite directions with larger than average observations of one variable tending to be offset by smaller than average observations of the other. This will lead to a cancellation of errors whereby the sum will be even closer to its average than either of the variables are close to theirs. The covariance term $Cov(X_1,X_2)$ shows how the correlation between the variables effects the overall uncertainty. Similar intuitions hold with more than 2 variables. Every pair of variables $X_i,X_j$ can either increase or decrease the overall uncertainty of the sum. By adding all such covariances together you get the net change of uncertainty from all such pairs.

Intuitions are not theorems. The above paragraph doesn't apply to all cases. For example, one way that you could have $Cov(X_1,X_2) > 0$ is by having a large number of negative values be offset by rarer but larger positive values. $X_1$ and $X_2$ might actually tend to pull in opposite directions in the sense that $P((X_1 - \mu_1)(X_2 - \mu_2) < 0) > 0.5$ but nevertheless $Cov(X_1,X_2) > 0$ since when they pull in the same direction they really pull in the same direction. The formula itself takes all such considerations into account.


This formula is actually not at all deep and works if you replace $\mathrm{Cov}$ with any symmetric bilinear form. Imagine we have an inner product $(-,-)$ and we want to find $(a+b,a+b)$. Then $$(a+b,a+b)=(a,a)+(b,b)+(a,b)+(b,a)=(a,a)+(b,b)+2(a,b)$$ You can find a similar formula for $(a+b+c,a+b+c)$, and the general form is exactly the same. It just depends on covariance being symmetric and bilinear and the variance of a variable being the covariance with itself.


It is very simple. It is just like opening up brackets in a $(a+b)^2$. In that case you will have $a^2+2ab+b^2$.

Hopefully that can guide you to figure out how it's derived. Let me know if you need in depth explanation though.

Tags:

Statistics