Strange interaction between unicodemath and siunitx

Something weird is happening when \text is processed in the argument of \SI. Here's a fix:

\documentclass{article}

\usepackage{amsmath}
\usepackage{unicode-math}% this makes e appear straight (not math)
\usepackage{siunitx,xparse}

\NewDocumentCommand{\mathnormalsymbol}{m}{\text{$\mathnormal{#1}$}}
\DeclareSIUnit{\elementarycharge}{\mathnormalsymbol{e}}

\begin{document}

\SI{1.234}{\planckbar^{-4}\elementarycharge^{2}}

$1.234\, \hbar^{-4} e^2$ %this is always ok

\end{document}

The answer by egreg provides a perfectly good solution, but it might be useful to understand why this is happening.

Internally, siunitx resolves input into constructs of the form

$\mathrm{<unit-output>}$

so here

$\mathrm{\text{\ensuremath{e}}}$

With classical TeX, switching math-text-math gets us 'back to normal' in font terms so we can the normal italic font for letters without any explicit formatting. However, with unicode-math this doesn't seem to happen: nested font changes stay active. As as result, the usual set up gives incorrect results.

I'll see whether this can be addressed at the unicode-math end or whether a change in siunitx is needed.