Alternative notation for inverse function

If anything, I think $f^{-1}(x)$ is absolutely the correct notation for an inverse function. Correspondingly, I think $f^2(x)$ is absolutely the correct notation for $(f\circ f)(x) = f(f(x))$, not for $(f(x))^2$. But this is definitely a matter of taste, as well as context, and other people will disagree with me. For example, if $f$ isn't an endomorphism then $(f\circ f)$ makes no sense, whereas if the codomain of $f$ doesn't have any kind of multiplicative structure then $(f(x))^2$ doesn't make sense. As a reader, you should always keep in mind that the author may use symbols in a way more appropriate to their work (or even just their aesthetic sensibilities) and be willing to do some interpretative work!

There might be some sense in defining e.g. $f^{\circ 2}(x) = (f\circ f)(x)$, $f^{\cdot 2}(x) = (f(x))^2$, similarly to how we define $f^{\otimes 2}(x) = (f\otimes f)(x)$ on a tensor square of vector spaces $V^{\otimes 2}$, etc. I've seen this notation before. This would make the inverse function $f^{\circ(-1)}(x)$ and the reciprocal $f^{\cdot(-1)}(x)$. But this is pretty ugly, and I've almost never seen it. Unless you have a whole bunch of binary operations like $\circ, \cdot, \otimes$ lying around and you need to explicitly disambiguate between them, I'd avoid it. Just use words. (Besides which, you'd probably need to use words to tell readers what your notation means, because it's probably new to them.)

Good notation is like gold dust, but keep in mind that most notation is bad. Don't add to it, if possible. Just make your own work as readable as you can.

I was reading Tensor Geometry not too long ago and I encountered a good notation for the reverse of a map,

The reason for the two conventional uses of $^{-1}$ is that what they invert are written very similarily. In fact, in many settings, composition of two functions is actually written exactly like multiplication.

For instance, multiplying matrices corresponds to composing linear maps, so we often see composed linear maps written next to one another without a symbol between, as though we're multiplying. And when working with symmetry groups, a composition of two permutations is also often written by writing them next to one another without a symbol in-between.

So, seing as how composition of functions is often written like multiplication, it is not that difficult to see why the compositional inverse is also written the same way as the multiplicative inverse.

I would love for there to be a consistent way to differentiate between the two, but there isn't. At least not any widely used convention. Also, in some cases, like in my two examples above, it would mean you have to decide, every time you want to invert an object, whether that object is a function to be composed, or an element of a group or monoid to be multiplied, when really, you want it to be both at the same time.