What is the standard notation for a multiplicative integral?

This type of construction also arises in topology and algebraic geometry as "iterated integrals" or "Chen's iterated integrals". There are many sources of which a famous one by Chen himself is: http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.bams/1183539443 .

Path-ordered (or time-ordered) exponential, as suggested in the other answer, is the most common term, or at least would get the most hits in a search, but this is due to the usage in physics.

ADDED: this paper by Hain (Chen's student) calls the construction "iterated line integrals". http://arxiv.org/abs/math.AG/0109204 . Another paper calls it "iterated integrals" in a more specific context matching that of the question: p.21 of http://www.math.toronto.edu/drorbn/papers/OnVassiliev/OnVassiliev.pdf .

As I mentioned in my comment, this concept is the time-ordered exponential (as I remember from Quantum Field Theory lectures, long ago). Alternatively, the path-ordered exponential. I'm no expert here, so I can't really point you to any definitive sources that you couldn't find with a bit of searching yourself anyway.

The function f(t) does not have to be particularly 'nice', just locally integrable should be enough. You can go even further and replace f(t)dt by dF(t) for a continuous finite-variation function F. In fact, F can be any continuous semimartingale, as long as you use Stratonovich integration in the associated SDE (as Ito integration is not coordinate independent). This is the method used by Rogers & Williams (Diffusions, Markov Processes and Martingales) to construct Brownian motions on Lie groups and referred to there as the product-integral injection. Actually, it's a bijection from the continuous semimartingales X in the Lie algebra starting at $X_0=0$ to the continuous semimartingales Y in the Lie group starting at the point $Y_0=1$ satisfying the Stratonovich SDE $$ \partial Y = Y\,\partial X. $$

Also, why restrict to Lie groups/algebras? Any manifold with an affine connection will do, where f maps to the tangent space at some base point, and is moved along the generated curve by parallel transport. There is the possibility of the solution exploding in finite time though. In the case of Lie groups, there is a standard invariant connection which gives you the time-ordered exponential.

In the physics world, the notation is $P\exp(\int_a^b f(t)\,dt)$ or $T\exp(\int_a^b f(t)\,dt)$, where the "$P$" and "$T$" stand for "path ordered" and "time ordered". The idea of time-ordered arithmetic I think is originally due to Feynman:

• R.P. Feynman (1951). An operator calculus having applications in quantum electrodynamics. Physical Review. vol. 84 (1) pp. 108-128.

In the UC Berkeley 2008 course on Lie theory by Mark Haiman (my edited lecture notes are available as a PDF), we called it just $\int$, which was a bit of an abuse of notation. Or rather, for any ODE, we referred to the corresponding "flow" as $\int$: $\int_p(\vec x)(t)$ was the point that you get to by starting at a point $p$ and flowing via the vector field $\vec x$ by $t$ seconds. I'm not a fan of this notation, myself, whereas I'm reasonably happy with "$T\exp$".