Representation of $e$ as a descending series
You can derive the following telescoping sum for $k\ge2$:
\begin{align*} \frac{1}{k!}+\frac{1}{k!(k-1)k} &=\frac{1}{k!}\left(1+\frac{1}{(k-1)k}\right)\\ &=\frac{1}{k!}\left(1+\frac{1}{k-1}-\frac{1}{k}\right)\\ &=\frac{k-1}{kk!}+\frac{1}{(k-1)k!}\\ &=\frac{k-1}{kk!}+\frac{1}{(k-1)k(k-1)!}\\ &=\frac{k-1}{kk!}+\frac{1}{(k-1)(k-1)!}-\frac{1}{k(k-1)!}\\ &=\frac{1}{k!}-\frac{1}{kk!}+\frac{1}{(k-1)(k-1)!}-\frac{1}{k!}\\ &=\frac{1}{(k-1)(k-1)!}-\frac{1}{kk!} \end{align*}
Hence, if you go back to the original series representation of $e$ as you did:
\begin{align*} e+\sum_{k=2}^{\infty}\frac{1}{k!k(k-1)} &=2+\sum_{k=2}^{\infty}\frac{1}{k!}+\frac{1}{k!k(k-1)}\\ &=2+\sum_{k=2}^{\infty}\frac{1}{(k-1)(k-1)!}-\frac{1}{kk!}\\ &=2+1-\lim_{n\rightarrow\infty}\frac{1}{nn!}=3 \end{align*}
As an alternative approach, we have
$$ 3-e=\int_{0}^{1}x(1-x)e^x\,dx \tag{1}$$
by IBP and the RHS of $(1)$ is clearly positive, hence $e<3$.
Since $\int_{0}^{1}x(1-x)\frac{x^k}{k!}\,dx = \frac{1}{k!(k+2)(k+3)}$ we also have
$$ 3-e = \sum_{k\geq 0}\frac{1}{k!(k+2)(k+3)}\tag{2} $$
by termwise integration of a Taylor series.
An equivalent integral
Plugging a relationship similar to the one used by Jack D'Aurizio, namely $$\frac{1}{k!(k-1)k}=\int_0^1 \frac{x^{k-2}(1-x)}{k!} dx$$ into this integral
$$\int_0^1 \frac{(1-x)(e^x-1-x)}{x^2}dx = 3-e$$
with non-negative integrand in $(0,1)$ that proves $e<3$, yields the series in the question.
$$\sum_{k=2}^\infty \frac{1}{k!(k-1)k}=3-e$$
This relates the series with the inequality $1+x \leq e^x$.
Similar approximations
This integral is similar to the one used to explain why $e$ is close to the eighth harmonic number.
$$\frac{1}{14} \int_0^1 x^2(1-x)^2(e^x-1-x)dx = e-\frac{761}{280}=e-H_8\approx 0$$
with corresponding series $$e=\frac{761}{280}+\frac{1}{7}\sum_{k=2}^\infty \frac{1}{k!(k+3)(k+4)(k+5)}$$
and $$\frac{1}{2}\int_0^1 (1-x)^2\left(e^x-1-x-\frac{x^2}{2}\right)dx = e-\frac{163}{60}$$
used to explain the observation by Lucian that $2\pi+e$ is close to $9$.
Another series for $3-e$
Yet another series to prove $e<3$ is related to the integer sequence http://oeis.org/A165457.
$$\frac{1}{e}=\frac{1}{3}+\sum_{k=1}^\infty \frac{1}{(2k+1)!(2k+3)}$$