a continuous function satisfying $f(f(f(x)))=-x$ other than $f(x)=-x$

The only solution is $f(x)=-x$ for all odd $n$.

We have $f(0)=0$ because if $f(0)=a$, then $f(f(a)))=0$ and $-a=f(f(f(a)))=f(0)=a$, so $a=0$.

Since $f^{2n}(x)=x$ for all $x$, the function $f$ cycles sets of at most $2n$ elements $\{\pm a_1,\pm a_2,\ldots,\pm a_n\}$. Fix one of these cycles, and assume without loss of generality that all the $a_i$ are nonzero, and denote by $a>0$ the least positive number in this cycle. Then $f((0,a))$ is an open interval with the endpoints $f(0)=0$ and $f(a)=b$. Assume that $b\neq \pm a$. Since $|b|>|a|$, the interval $f((0,a))$ must contain either $a$ or $-a$, which means there is an $x, |x|<a$ such that $f(x)=a$ or $f(x)=-a$, which is a contradiction, because $f$ is a bijection and $\pm a$ are the images of some $a_i, |a_i|\geq a$. Thus it must hold that $|f(a)|=a$ for all $a$, and consequently $f(a)=-a$ for all $a$.

As Henry points out in the comments, when $n$ is odd $f(x)=-x$ obviously works.

When $n$ is even, there is no such function. To see this, note the following properties must hold:

• $f$ is a bijection, and hence stricly monotonic.
• $f(0)=0$. Indeed, if $f(0)=a$, then $f^n(a) = f^{n+1}(0) = f(f^n(0)) = f(0) = a$, so $a=0$ since $0$ is the only fixed point of $-x$.
• From the previous two properties, we must have that $f$ is either increasing, in which case $f(x)$ has the same sign as $x$ for every $x$, or decreasing, in which case $f(x)$ has the opposite sign as $x$ for every $x$. Either way, $f^n(x)$ and $x$ share the same sign, and so $f^n(x) \neq -x$.

I have seen an example of a piecewise continuous function $f$ satisfying $f^2(x)=-x$. It's best described with a picture, but I don't know how to make those in this forum. Instead, I can describe it for nonnegative $x$, and then I'll ask you to give it odd symmetry.

EDIT: I did indeed mess up the formula. I think this is what I wanted. \begin{align} f(0) &= 0 \\\\ f(x) &= x+1 &&\mbox{for $x\in(2n-2,2n-1]$}\\\\ f(x) &= 1-x &&\mbox{for $x\in(2n-1,2n]$}\\\\ \end{align}

Again, this is for nonnegative $x$ only. Extend to negative $x$ with odd symmetry.

Sketch this for $n=1,2,3$ and you'll get the idea. Assuming I haven't messed up the formula, you get a graph that reminds me of TIE fighters from Star Wars.

EDIT: I uploaded a picture - much easier than I thought.