How to prove that the derivative of Heaviside's unit step function is the Dirac delta?

This is a place where physicists and mathematicians would phrase the question differently. A mathematician would say that $d \theta/dx$ is undefined at $0$, and that $\delta$ is not a function. However, as Mariano says, the statement is true "in the sense of distributions".

What does that mean? A distribution is a gadget which takes as input a smooth function $g(x)$, which is zero for $|x|$ sufficiently large, and returns some real number. When we think of an ordinary function as a distribution, that makes we think of $f$ as corresponding to the gadget $F: g \mapsto \int f(x) g(x) dx$. Notice that changing $f$ at a finite number of points leaves $F$ unaltered. From a physical point of view, if $f$ is something like the value of an electric field at a point, we would never know if it had a finite discontinuity at some point, so the gadget $F$ captures everything that is physically measurable about $f$.

Now, how can we see differentiation in terms of distributions? By integration by parts, we have $\int f'(x) g(x) dx = - \int f(x) g'(x)$. So, for any distribution $F$, we define the derivative of $F$ to be the gadget $g \mapsto -F(g')$.

Now, let $F$ correspond to $\theta$, so $F(g) = \int_{- \infty}^0 g(x) dx$. The Dirac delta distribution is $\delta(g) = g(0)$. I leave it to you to show that $F'(g) = \delta(g)$, with the definitions above.

I'd be curious to see how a physicist would answer this question. I suspect that I have acted like a Frenchman.

By definition, $\delta(x)$ satisfies $$\int_{-\infty}^\infty f(x)\delta(x) dx=f(0)$$ for any continuous function $f$ on $\mathbb{R}$.

By definition of derivative in distribution theory, $$ \int_{-\infty}^\infty f(x)\theta'(x) dx=-\int_{-\infty}^\infty f'(x) \theta(x) dx$$ for any $C^1$ function $f(x)$ which vanishes outside of a bounded interval.

Now $$-\int_{-\infty}^\infty f'(x) \theta(x) dx=-\int_0^\infty f'(x) dx= -f(\infty)+f(0)=f(0)=\int_{-\infty}^\infty f(x)\delta(x) dx.$$

So $\theta'(x)=\delta(x)$.

I'd like to add an answer that will try to justify the statement with an example. Consider the function sequence $$ f_n = \left\{\begin{align} 0 \ \ &\text{ if } |x|>1/n \\ n^2x^2/2 + nx + 1/2 &\text{ if } -1/n\leq x \leq 0\\ -n^2x^2/2 + nx + 1/2 &\text{ if } 0<x\leq 1/n \end{align}\right.. $$
These functions are differentiable everywhere, and they approximate the unit step function better and better as $n$ gets larger. If we look at the derivatives of these functions, we'll see that they are a sequence of tent functions that are getting narrower and taller, as Griffith uses to introduce the delta function.

Actually, with an appropriate mode of convergence, when a sequence of differentiable functions converge to the unit step, it can be shown that, their derivatives converge to the delta function. That's why, one may take the derivative of the unit step function to be defined as the limit of the derivatives, which is the delta function.