Is the binomial theorem actually more efficient than just distributing

The binomial theorem allows you to write out the expansion of your polynomial immediately.

It also allows you to answer such questions as "What is the coefficient of $x^{20}$ in $(1+x)^{100}$?"

Its generalisation to non-integer exponents allows you to get the expansion of $(1-x)^{-1/2}$.

It is a good thing.

There are all sorts of reasons. First of all, you'll learn when you've moved forward a bit that the binomial theorem can actually be applied in cases of non-integer exponents, by defining $\binom{n}{k} = \dfrac{n^{\underline{k}}}{k!}$, where the underline represents a falling power (i.e., $n(n-1)(n-2)\cdots$). Another that comes to mind immediately is the series expansion of $e$...

$$e^x = \lim_{n\to\infty}\left(1+\frac{x}{n}\right)^n = \lim_{n\to\infty}\left(1 + n\frac{x}{n} + \frac{n^\underline{2}}{2!}\frac{x^2}{n^2} + \frac{n^\underline{3}}{3!}\frac{x^3}{n^3} + \cdots + \frac{x^n}{n^n}\right)$$

As $n$ gets very large, for fixed $k$, $n^\underline{k}$ approaches $n^k$, and $\frac{x^n}{n^n}$ vanishes since $x$ is fixed, so you end up with the familiar series...

$$e^x = 1 + x + \frac{x^2}{2} + \frac{x^3}{6} + \frac{x^4}{24} + \cdots$$

If you're not seeing the utility of the binomial theorem, then I think you're missing an important observation:

The coefficients of every term can be calculated extremely efficiently (your example of $(x-4)^6$ is easy to expand by hand with the binomial theorem, without, it's doable but very tedious).

A useful shorthand for lower powers of the exponent is to remember the first few rows of Pascal's Triangle along with the rule for adding more if you need them. Example of first few rows:

$$1$$ $$1\qquad 1$$ $$1\qquad 2\qquad 1$$ $$1\qquad 3\qquad 3\qquad 1$$ $$1\qquad 4\qquad 6\qquad 4\qquad 1$$ $$1\qquad 5\qquad 10\qquad 10\qquad 5\qquad 1$$ $$1\qquad 6\qquad 15\qquad 20\qquad 15\qquad 6\qquad 1$$

The last row gives the coefficients of $x^k\cdot(-4)^{(6-k)}$ for $k$ running from $0$ to $6$. Computing $(-4)^{(6-k)}$ is easy enough by hand, and multiplying it by the relevant entry in the last row above is also fairly easy (give it a try).

One more thing: The binomial theorem admits an interesting generalization to powers of general polynomials of a given length sometimes called the multinomial theorem. (Challenge: try investigating the coefficients of the first few powers of $(x+y+z)^n$ and perhaps you can figure out the pattern for trinomials? How does it relate to the pattern for binomials? Can you generalize it to general polynomials?)