The formal solution of the time-dependent Schrödinger equation
I) The solution to the time-dependent Schrödinger equation (TDSE) is
$$ \Psi(t_2) ~=~ U(t_2,t_1) \Psi(t_1),\tag{A}$$
where the (anti)time-ordered exponentiated Hamiltonian
$$\begin{align} U(t_2,t_1)~&=~\left\{\begin{array}{rcl} T\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! dt~H(t)\right] &\text{for}& t_1 ~<~t_2 \cr\cr AT\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! dt~H(t)\right] &\text{for}& t_2 ~<~t_1 \end{array}\right.\cr\cr ~&=~\left\{\begin{array}{rcl} \lim_{N\to\infty} \exp\left[-\frac{i}{\hbar}H(t_2)\frac{t_2-t_1}{N}\right] \cdots\exp\left[-\frac{i}{\hbar}H(t_1)\frac{t_2-t_1}{N}\right] &\text{for}& t_1 ~<~t_2 \cr\cr \lim_{N\to\infty} \exp\left[-\frac{i}{\hbar}H(t_1)\frac{t_2-t_1}{N}\right] \cdots\exp\left[-\frac{i}{\hbar}H(t_2)\frac{t_2-t_1}{N}\right] &\text{for}& t_2 ~<~t_1 \end{array}\right.\end{align}\tag{B} $$
is formally the unitary evolution operator, which satisfies its own two TDSEs
$$ i\hbar \frac{\partial }{\partial t_2}U(t_2,t_1) ~=~H(t_2)U(t_2,t_1),\tag{C} $$ $$i\hbar \frac{\partial }{\partial t_1}U(t_2,t_1) ~=~-U(t_2,t_1)H(t_1),\tag{D} $$
along with the boundary condition
$$ U(t,t)~=~{\bf 1}.\tag{E}$$
II) The evolution operator $U(t_2,t_1)$ has the group-property
$$ U(t_3,t_1)~=~U(t_3,t_2)U(t_2,t_1). \tag{F}$$
The (anti)time-ordering in formula (B) is instrumental for the (anti)time-ordered expontial (B) to factorize according to the group-property (F).
III) The group property (F) plays an important role in the proof that formula (B) is a solution to the TDSE (C):
$$\begin{array}{ccc} \frac{U(t_2+\delta t,t_1) - U(t_2,t_1)}{\delta t} &\stackrel{(F)}{=}& \frac{U(t_2+\delta t,t_2) - {\bf 1} }{\delta t}U(t_2,t_1)\cr\cr \downarrow & &\downarrow\cr\cr \frac{\partial }{\partial t_2}U(t_2,t_1) && -\frac{i}{\hbar}H(t_2)U(t_2,t_1).\end{array}\tag{G}$$
Remark: Often the (anti)time-ordered exponential formula (B) does not make mathematical sense directly. In such cases, the TDSEs (C) and (D) along with boundary condition (E) should be viewed as the indirect/descriptive defining properties of the (anti)time-ordered exponential (B).
IV) If we define the unitary operator without the (anti)time-ordering in formula (B) as
$$ V(t_2,t_1)~=~\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! dt~H(t)\right],\tag{H}$$
then the factorization (F) will in general not take place,
$$ V(t_3,t_1)~\neq~V(t_3,t_2)V(t_2,t_1). \tag{I}$$
There will in general appear extra contributions, cf. the BCH formula. Moreover, the unitary operator $V(t_2,t_1)$ will in general not satisfy the TDSEs (C) and (D). See also the example in section VII.
V) In the special (but common) case where the Hamiltonian $H$ does not depend explicitly on time, the time-ordering may be dropped. Then formulas (B) and (H) reduce to the same expression
$$ U(t_2,t_1)~=~\exp\left[-\frac{i}{\hbar}\Delta t~H\right]~=~V(t_2,t_1), \qquad \Delta t ~:=~t_2-t_1.\tag{J}$$
VI) Emilio Pisanty advocates in a comment that it is interesting to differentiate eq. (H) w.r.t. $t_2$ directly. If we Taylor expand the exponential (H) to second order, we get
$$ \frac{\partial V(t_2,t_1)}{\partial t_2} ~=~-\frac{i}{\hbar}H(t_2) -\frac{1}{2\hbar^2} \left\{ H(t_2), \int_{t_1}^{t_2}\! dt~H(t) \right\}_{+} +\ldots,\tag{K} $$
where $\{ \cdot, \cdot\}_{+}$ denotes the anti-commutator. The problem is that we would like to have the operator $H(t_2)$ ordered to the left [in order to compare with the TDSE (C)]. But resolving the anti-commutator may in general produce un-wanted terms. Intuitively without the (anti)time-ordering in the exponential (H), the $t_2$-dependence is scattered all over the place, so when we differentiate w.r.t. $t_2$, we need afterwards to rearrange all the various contributions to the left, and that process generate non-zero terms that spoil the possibility to satisfy the TDSE (C). See also the example in section VII.
VII) Example. Let the Hamiltonian be just an external time-dependent source term
$$ H(t) ~=~ \overline{f(t)}a+f(t)a^{\dagger}, \qquad [a,a^{\dagger}]~=~\hbar{\bf 1},\tag{L}$$
where $f:\mathbb{R}\to\mathbb{C}$ is a function. Then according to Wick's Theorem
$$ T[H(t)H(t^{\prime})] ~=~ : H(t) H(t^{\prime}): ~+ ~C(t,t^{\prime}), \tag{M}$$
where the so-called contraction
$$ C(t,t^{\prime})~=~ \hbar\left(\theta(t-t^{\prime})\overline{f(t)}f(t^{\prime}) +\theta(t^{\prime}-t)\overline{f(t^{\prime})}f(t)\right) ~{\bf 1}\tag{N}$$
is a central element proportional to the identity operator. For more on Wick-type theorems, see also e.g. this, this, and this Phys.SE posts. (Let us for notational convenience assume that $t_1<t_2$ in the remainder of this answer.) Let
$$ A(t_2,t_1)~=~-\frac{i}{\hbar}\int_{t_1}^{t_2}\! dt~H(t) ~=~-\frac{i}{\hbar}\overline{F(t_2,t_1)} a -\frac{i}{\hbar}F(t_2,t_1) a^{\dagger} ,\tag{O}$$
where
$$ F(t_2,t_1)~=~\int_{t_1}^{t_2}\! dt ~f(t). \tag{P}$$
Note that
$$ \frac{\partial }{\partial t_2}A(t_2,t_1)~=~-\frac{i}{\hbar}H(t_2), \qquad \frac{\partial }{\partial t_1}A(t_2,t_1)~=~\frac{i}{\hbar}H(t_1).\tag{Q} $$
Then the unitary operator (H) without (anti)time-order reads
$$\begin{align} V(t_2,t_1)~&=~e^{A(t_2,t_1)} \\ ~&=~\exp\left[-\frac{i}{\hbar}F(t_2,t_1) a^{\dagger}\right]\exp\left[\frac{-1}{2\hbar}|F(t_2,t_1)|^2\right]\exp\left[-\frac{i}{\hbar}\overline{F(t_2,t_1)} a\right].\tag{R} \end{align}$$
Here the last expression in (R) displays the normal-ordered for of $V(t_2,t_1)$. It is a straightforward exercise to show that formula (R) does not satisfy TDSEs (C) and (D). Instead the correct unitary evolution operator is
$$\begin{align} U(t_2,t_1)~&\stackrel{(B)}{=}~T\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! dt~H(t)\right] \\~&\stackrel{(M)}{=}~:\exp\left[-\frac{i}{\hbar}\int_{t_1}^{t_2}\! dt~H(t)\right]:~ \exp\left[\frac{-1}{2\hbar^2}\iint_{[t_1,t_2]^2}\! dt~dt^{\prime}~C(t,t^{\prime})\right] \\ ~&=~ e^{A(t_2,t_1)+D(t_2,t_1)}~=~V(t_2,t_1)e^{D(t_2,t_1)}\tag{S}, \end{align}$$
where
$$ D(t_2,t_1)~=~\frac{{\bf 1}}{2\hbar}\iint_{[t_1,t_2]^2}\! dt~dt^{\prime}~{\rm sgn}(t^{\prime}-t)\overline{f(t)}f(t^{\prime})\tag{T}$$
is a central element proportional to the identity operator. Note that
$$\begin{align} \frac{\partial }{\partial t_2}D(t_2,t_1)~&=~\frac{{\bf 1}}{2\hbar}\left(\overline{F(t_2,t_1)}f(t_f)-\overline{f(t_2)}F(t_2,t_1)\right) \\ ~&=~\frac{1}{2}\left[ A(t_2,t_1), \frac{i}{\hbar}H(t_2)\right]~=~\frac{1}{2}\left[\frac{\partial }{\partial t_2}A(t_2,t_1), A(t_2,t_1)\right].\tag{U} \end{align}$$
One may use identity (U) to check directly that the operator (S) satisfy the TDSE (C).
References:
- Sidney Coleman, QFT lecture notes, arXiv:1110.5013; p. 77.