A rigorous definition of the exponential of an operator in QM?
Even if there is already a good accepted answer I would like to say something further to completely fix some details.
Is this definition correct for unbounded operators, too?
No, it does not work essentially because of the used wrong notion of convergence.
However, it is possible to prove that, if $A$ — with dense domain $D(A)$ — is closed and normal (*) — which includes the selfadjoint and the unitary case — then there is a dense subspace $D_A\subset D(A)$ of vectors, called analytic vectors where the formula is still valid with the crucial changes that
(a) the operators have to be applied to these vectors, and
(b) the topology of the Hilbert space has to be used (the series is now of vectors rather than operators), $$e^{tA}\psi = \sum_{n=0}^{+\infty} \frac{t^n}{n!}A^n \psi\:, \quad \forall \psi \in D_A .$$
(The parameter $t\in \mathbb{C}$ can be taken in a sufficiently small neighborhood of $0$, independent of $\psi\in D_A$.)
I stress that the series is not the definition of the exponential, the identity above is an identity of two independently defined mathematical objects.
However that series can be used to equivalently define the exponential on the said domain and this definition coincides with the definition for unbounded operators below.
If not, what is the correct definition?
If $A: D(A) \to H$, densely defined, is closed and normal, then it admits a spectral measure $P: B(\mathbb{C}) \to B(H)$, where $B(\mathbb{C})\ni E$ is the Borel $\sigma$-algebra on $\mathbb{C}$ and each $P(E)$ is an orthogonal projector in $H$.
We can eventually define (on a suitably dense domain defined below) $$f(A) := \int_{\mathbb{C}} f(\lambda) dP(\lambda)\tag{1}$$ for every Borel-measurable function $f: \mathbb{C}\to \mathbb{C}$.
The exponential of the said $A$ is defined in this way simply replacing $f$ for the exponential map.
If $A$ is selfadjoint, $B(\mathbb{C})$ can be repalced by $B(\mathbb{R})$ since outside $\mathbb{C}$ the spectral measure vanishes.
Actually the support of the spectral measure of $A$ (densely-defined, closed, and normal) always coincides with the spectrum $\sigma(A)$ of $A$.
What properties does $\hat A$ have to have in order to have a well-defined exponential?
We have two cases which actually coincide where both definitions are suitable.
(a) If $A$ is everywhere defined and bounded, the exponential is automatically well-defined by its series expansion — with respect to the operator norm — and this expansion can be used as the very definition.
(b) If $A$ is not everywhere defined / bounded, the previous definition (1) based on the Borel functional calculus applies when $A$ is densely defined, normal, and closed, in particular selfadjoint.
This latter definition, (b), coincides with the former, (a), when $A$ is everywhere defined, bounded and normal, for instance if $A$ is unitary.
As declared at the beginning, the series expansion is however valid for densely-defined, closed, normal operators working on analytic vectors and using the norm of $H$ (technically the strong operator topology).
As far as I know these (densely-defined, closed, normal) are the minimal requirements producing a consistent theory for unbounded operators.
If $\hat A$ is defined on $D(\hat A)$ what is the domain of $\mathrm{e}^{\hat A}$?
The domain of $f(A)$ as in (1) is
$$D(f(A)) = \left\{\psi \in H \:\left|\: \int_{\mathbb{c}} |f(x)|^2 d \mu^A_\psi(x)< +\infty \right.\right\}\tag{2}$$ where $$\mu^A_\psi(E) := \langle \psi |P(E) \psi\rangle$$ is a standard positive finite Borel measure.
If $A$ is selfadjoint $\mu^A_\psi$ is supported in $\mathbb{R}$ actually on $\sigma(A)$. There, $f(x) = \exp x$ is not bounded (unless $\sigma(A)$ is bounded which menas that $A$ is bounded), so that $D(f(A)) \subsetneq H$.
However if you instead consider $f(x)= \exp ix$ and $A$ is selfadjoint, then $f$ is boundend by $1$ on $\mathbb{R}$. Since $\mu^A_f(\mathbb{R}) = ||\psi||^2 < +\infty$, it turns out from (2) that $$D(f(A)) = H\:.$$
If $E \subset \mathbb C$ is Borel set and $\chi_E(x)=1$ for $x\in E$ and $\chi_E(x)=0$ otherwise, then $$P_E := \int_{\mathbb C} \chi_E(x) dP^{(A)}(x)$$ is an orthogonal projector onto a closed subspace $H_E$.
A family of analytic vectors $\psi$ thus satisfying $$e^{tA}\psi = \sum_{n=0}^{+\infty} \frac{t^n}{n!}A^n \psi$$ whose (finite) span is dense is obtained as follows. Take a class of Borel sets $E_N\subset \mathbb C$, where $N\in \mathbb N$, requiring that every $E_N$ is bounded and $\cup_N E_N = \mathbb C$. The said family of analytic vectors consists of all vectors $\psi \in H_{E_N}$ for every $N \in \mathbb N$.
As I final remark, I stress that almost all operators with some relevance in QM are both densely defined and closed.
(*) $A: D(A) \to H$ is closed if the set of pairs $(\psi, A\psi)$ with $\psi \in D(A)$ is a closed set in $H \times H$.
$A: D(A) \to H$ densely defined and closed is normal if $A^\dagger A= A A^\dagger$ on the natural domains of both sides which are required to coincide.
The mathematical name for the theory of applying functions to operators is functional calculus, and the one employed usually when one wants to rigorously talk about e.g. the exponential of the position and momentum operators - for instance in the context of Stone's theorem - is Borel functional calculus. It works for all normal operators, i.e. all operators to which you can apply some version of the spectral theorem to get the spectral measure that a physicist would write as $\int a \lvert a\rangle\langle a\rvert\mathrm{d}a$ for $\lvert a\rangle$ the "eigenstates" of some normal operator $A$.
Applying the operator $A$ is the same as applying $\int a\lvert a\rangle\langle a\rvert \mathrm{d}a$, so applying $f(A)$ is the same as applying $\int f(a)\lvert a\rangle\langle a\rvert\mathrm{d}a$. The difficulty mathematically is proving the existence and uniqueness of the operator described by this modified spectral measure. E. g. the book of Simon and Reed should have a proof of the well-definedness of the functional calculus required for physical applications.
The domain of the resulting operator is the entire Hilbert space if $f$ is bounded, and if $f$ is unbounded, then it is whatever subset of the Hilbert space the expression $\int f(a)\lvert a\rangle \langle a\vert \psi\rangle\mathrm{d}a$ converges on. Note again that, rigorously, things like $\lvert a\rangle$ don't really exist within the Hibert space and $a \lvert a\rangle\langle a\rvert \mathrm{d}a$ is just one indivisible expression denoting the spectral measure.