Why does integrating a complex exponential give the delta function?
The integral is not meant to be taken in the space of functions; it is meant to be taken over the space of distributions.
If I write $[f(x)]$ when I mean to view $f(x)$ as a distribution rather than a function, then this integral is saying
$$ \int_{-\infty}^{\infty} [e^{ikx}] \, \mathrm{d}k = 2 \pi \delta(x) $$
(note that $\delta(x)$ is a distribution, not a function, so I don't need to put brackets around it)
Two distributions are equal if and only if they have the same value when convolved with test functions. The above equation is asserting, for every test function $f(x)$, you have
$$ \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} [e^{ikx}] \, \mathrm{d}k \right) f(x) \, \mathrm{d}x = \int_{-\infty}^{\infty} 2 \pi \delta(x)f(x) \, \mathrm{d} x$$
What the test functions are precisely can vary with context; in this case they are probably meant to be the rapidly decreasing functions (aka Schwartz functions).
IIRC, distributions coming from functions of two variables satisfy
$$ \int_{-\infty}^{\infty}\left( \int_{-\infty}^{\infty} [g(x,y)] \, \mathrm{d}y \right)f(x) \mathrm{d}x = \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} g(x,y) f(x)\, \mathrm{d}x \right) \mathrm{d}y $$
(note the change of order of the integration variables)
Consequently, The above equation is asserting
$$ \int_{-\infty}^{\infty} \left( \int_{-\infty}^{\infty} e^{ikx} f(x) \, \mathrm{d}x \right) \mathrm{d}k =2 \pi f(0)$$
Note, in particular, that the inner integral is that of ordinary functions, and the integrand is integrable.
Following Sylvain's comment look up the formulas for Fourier transform $F(w)=\int f(t) e^{-iwt}{\rm {d}}t$ and the inverse transform $ f(t)=1/{2\pi}\int F(w) e^{iwt} { \rm {d}}w$ and combine them to write \begin{equation} \begin{split} F(\hat w)=\int_{-\infty}^\infty f(t) e^{-i\hat{w}t}\rm {d}t & = \int_{-\infty}^\infty \frac{1}{2\pi}\int_{-\infty}^\infty F(w) e^{iwt} {\rm {d}} w\, e^{-i\hat{w}t}\,\rm {d}t \\ & = \frac{1}{2\pi}\int_{-\infty}^\infty F(w)\,{\rm {d}} w\int_{-\infty}^\infty e^{iwt} \, e^{-i\hat{w}t}\,\rm {d}t\\ & = \frac{1}{2\pi}\int_{-\infty}^\infty F(w)\,{\rm {d}} w\int_{-\infty}^\infty e^{i(w-\hat{w})t}\,\rm {d}t \end{split} \end{equation} Knowing the definition of the delta-function $f(y)=\int f(x) \delta(x-y) {\rm d} x$ one can see that in this case $$ F(\hat w) = \int_{-\infty}^{\infty} F(w) \delta (w-\hat w) {\rm d}w $$ and thus identifying the right integral with $$\frac{1}{2\pi}\int_{-\infty}^\infty e^{i(w-\hat{w})t}\,\rm {d}t= \delta (w- \hat w) $$
The answer of @Statics attacks with the argument that "if you think Fourier Transformation is correct, then you should accept this definition of Dirac Delta Function." But why the Fourier Transformation works at the first place, is because we have this Dirac Delta definition. So the argument using FT isn't sound to me.
The distribution explanation is correct. And the delta function makes sense under an integral sign. We can arrive to this formula by constructing a sequence of distribution $\delta_n(x)$, such that: $$\lim_{n\rightarrow\infty}\int^\infty_{-\infty}\delta_n(x-a)f(x)dx = f(a)$$ Then the limit of the sequence is $\delta(x)$, i.e. $$\lim_{n\rightarrow\infty}\delta_n(x) = \delta(x)$$
There are many ways to construct $\delta_n(x-a)$. One common example is: $$\delta_n(x-a)=\sqrt{\frac{n}{\pi}}e^{-n(x-a)^2}$$ $$=\frac{1}{2\pi}\int^\infty_{-\infty}e^{-\frac{k^2}{4n}}e^{i(x-a)k}dk$$
As you can see, the integral converge to $$\frac{1}{2\pi}\int^\infty_{-\infty}e^{i(x-a)k}dk$$ as $n\rightarrow\infty$. And integrate $f(x)$ with a Gaussian distribution will give the value $f(a)$ (for a rigorous proof, check textbooks on functional analysis). Therefore, we can safely use $$\frac{1}{2\pi}\int^\infty_{-\infty}e^{i(x-a)k}dk=\delta(x-a)$$
This is a sketch of proof, and definitely not a rigorous one. But hopefully, you can get some flavor out of it.