Determine all functions satisfying $f(x + f(x + y)) + f(xy) = x + f(x + y) +yf(x)$

Complete solution in the case $f(0) = 2$

Roots of $f$

As you point out, letting $y=0$ yields $$f(x+f(x)) + f(0) = x+f(x)$$ If $f(x)$ is ever $0$, then, we have $f(x+0) + f(0) = x+0$, so $$f(0) = x$$

So $f$ only hits $0$ at most once, and that is at $x = f(0)$; that is, if $f$ ever hits $0$ then $f(f(0)) = 0$ and that is the only time $f$ hits $0$.

Letting $x=y=0$, we see $f(f(0)) + f(0) = f(0)$, so $f(f(0)) = 0$.

Therefore $f$ does indeed hit $0$, so $f$ has exactly one root and it is at $f(0)$.

Moreover, as Batominovski points out, letting $x=0$ and $y=f(0)$ yields $f(0)^2 = 2 f(0)$, so $f(0) = 0$ or $f(0) = 2$.

From this point on, I'll trust Batominovski's proof that if $f(0) = 0$ then the identity is the only possible option; I'll look only at $f(0) = 2$.


Fixed points of $f$

We will do this by interchanging $x$ and $y$.

We'll follow Batominovski's shorthand of writing $P(x,y)$ for the statement $f(x+f(x+y)) + f(xy) = x+f(x+y)+y f(x)$.

We have $P(x,y)$ and $P(y, x)$, so subtracting the two equalities, we obtain the symmetrised expression $$f(x+f(x+y)) - f(y+f(x+y)) = x-y+yf(x)-xf(y)$$

Setting $x=0$ we get $$f(f(y)) + y = f(y+f(y)) + y f(0)$$

Now we will use that $f(0) = 2$.

In this case, $f(2) = 0$ because $f(0)$ is a root of $f$; it is the only root.

If $y$ is a fixed point of $f$, then $2y = f(2y) + 2y$, so $2y$ is a root of $f$, so $y = 1$. (And $1$ is indeed a fixed point.)

So in the case $f(0)=2$, $f$ has exactly one fixed point and it is at $1$.

Hence (by OP's $y=1$ work, which showed that $x+f(x+1)$ is always a fixed point) we have $f(x+1) + x = 1$ for all $x$, so $$f(t) = 2-t$$ for all $t$.


Addendum (ignore, it's a work in progress): Case $f(0) = 0$

(I discovered most of this independently of Batominovski, but I was much too slow with it! I did it as an exercise after Batominovski produced their solution.)

Letting $x=0$, as OP did, we see that $f(f(y)) = f(y)$ for all $y$. Also the root was unique, so $f(x) = 0$ if and only if $x=0$.

By OP's work, $x+f(x)$ is fixed by $f$, so if $x$ is a fixed point then so is $2x$.

Also letting $y=1$, if $x+1$ is fixed then so is $2x+1$. (That is, if $x$ is fixed then so is $2x-1$.)

Inductively, therefore, all nonpositive integers are fixed, since we know $0$ is fixed already.

Now letting $x=y=-1$, obtain $$f(f(-2) - 1) + f(1) = -1+f(-2)-f(-1)$$ which means $-3+f(1) = -1-2+1$ and so $f(1) = 1$.

Hence again inductively all integers are fixed.


Partial Solution

Let $P(x,y)$ denote $$f\big(x+f(x+y)\big)+f(xy)=x+f(x+y)+y\,f(x)$$ for any $x,y\in\mathbb{R}$. Write $c:=f(0)$. Then, $P(0,0)$ gives $f(c)=0$. Hence, $P(0,c)$ yields $$2c=2\,f(0)=f\big(f(c)\big)+f(0)=f(c)+c\,f(0)=c^2\,.$$ That is, $c=0$ or $c=2$. In this work, we shall assume that $c=0$.

Note that $P(x,-x)$ leads to $$f(x)+f\left(-x^2\right)=x-x\,f(x)\,.\tag{1}$$ With $x:=\pm1$ in the equation above, we conclude that $f(n)=n$ holds for $n\in\{-1,0,+1\}$. Now, we can utilize $P(1,y)$, namely, $$f\big(1+f(1+y)\big)+f(y)=1+f(1+y)+y\,,$$ to show that $f(n)=n$ for all nonnegative integers $n$. Similarly, $P(-1,y)$, which is given by $$f\big(-1+f(-1-y)\big)+f(-y)=-1+f(-1+y)-y\,,$$ implies that $f(n)=n$ for all negative integers $n$. Consequently, $f(n)=n$ holds true for any $n\in\mathbb{Z}$.

Now, plug in $x:=-x$ into (1) and compare the new equation with (1) itself. We see that $$f(-x)=\frac{2x-(x+1)\,f(x)}{x-1}\tag{2}$$ for all $x\in\mathbb{R}\setminus\mathbb{Z}$. Fix $n\in\mathbb{Z}$. Then, $P(x,n-x)$ gives us $$f(x+n)+f\big(x(n-x)\big)=(x+n)+(n-x)\,f(x)\,.\tag{3}$$ Replacing $x$ and $n$ in (3) by $-x$ and $-n$, respectively, yields $$f(-x-n)+f\big(x(n-x)\big)=-x-n-(n-x)\,f(-x)\,.\tag{4}$$ Utilize (2) in (3) and (4), and then remove the term $f\big(x(n-x)\big)$ to get $$f(x+n)=\frac{\left(x^2-x-n^2+n\right)\,f(x)+n\,\left(2x^2+(2n-3)x-n\right)}{(x-1)(x+n)}\tag{5}$$ holds for all $x\in\mathbb{R}\setminus\mathbb{Z}$. In particular, $$f(x+1)=\frac{x\,f(x)+(2x+1)}{x+1}\tag{6}$$ for all $x\in\mathbb{R}\setminus\mathbb{Z}$, and $$f(x+2)=\,\frac{\left(x^2-x-2\right)\,f(x)+2\,\left(2x^2+x-2\right)}{(x-1)(x+2)}\tag{7}$$ for all $x\in\mathbb{R}\setminus\mathbb{Z}$. From (6), we have $$f(x+2)=\frac{(x+1)\,f(x+1)+(2x+3)}{x+2}=\frac{x\,f(x)+4(x+1)}{x+2}$$ for all $x\in\mathbb{R}\setminus\mathbb{Z}$. Comparing the last equation with (7), we have $$f(x)=x$$ for all $x\in\mathbb{R}\setminus\mathbb{Z}$. However, as $f(x)=x$ holds for $x\in\mathbb{Z}$, we conclude that the identity function $f(x)=x$ for all $x\in\mathbb{R}$ is the sole solution to the condition $P(x,y)$ with the additional restraint $f(0)=c=0$.

From now on, assume that $f(0)=c=2$. Then, $P(1,1)$ implies that $f(1)=1$. Also, $P(x,2-x)$ gives us $$f(x)+f\big(x(2-x)\big)=x+(2-x)\,f(x)\,.$$ Replace $x$ by $2-x$ in the equation above, we obtain $$f(2-x)+f\big(x(2-x)\big)=(2-x)+x\,f(2-x)\,.$$ Comparing these two equations, we conclude that $$f(x)+f(2-x)=2\tag{8}$$ for all $x\in\mathbb{R}$ (where the case $x=1$ must be done separately, but then $f(1)=1$).

The condition $P(x,1-x)$ yields $$f(x+1)+f\big(x(1-x)\big)=x+1+(1-x)\,f(x)\,.\tag{9}$$ Substitute $1-x$ for $x$ in the equation above to get $$f(2-x)+f\big(x(1-x)\big)=2-x+x\,f(1-x)\,.$$ As $f(2-x)=2-f(x)$, we then see that $$f\big(x(1-x)\big)=f(x)-x+x\,f(1-x)\,.\tag{10}$$ Comparing (9) and (10), we have $$f(x+1)=2x+1-x\,f(x)-x\,f(1-x)\,.$$ From (8), we have $$ f(x+1)=2x+1-x\,f(x)-x\,\big(2-f(x+1)\big)\,,$$ or $$f(x+1)=\frac{x\,f(x)-1}{x-1}\tag{11}$$ for all $x\in\mathbb{R}\setminus\{1\}$. From this equation, it follows easily that $f(n)=2-n$ for every $n\in\mathbb{Z}$. Consequently, $$f(x+n)=\frac{(x+n-1)\,f(x)-n}{x-1}\tag{12}$$ for all $x\in\mathbb{R}\setminus\{1\}$ and $n\in\mathbb{Z}$.

From (12) with $P(x,n-x)$, we see that $$f\big(x(n-x)\big)=\frac{-\left(x^2-nx+1\right)\,f(x)+\left(x^2-(n-1)x\right)}{x-1}$$ for all $x\in\mathbb{R}\setminus\{1\}$ ans $n\in\mathbb{Z}$. This shows that $f(x)=2-x$ for all real quadratic algebraic integers $x$. I have no idea how to proceed further than this.


Here's a clean solution invoking fixed points.

Denote $P(x,y)$ as the assertion that $f(x+f(x+y))+f(xy)=x+f(x+y)+yf(x)$.

$P(0,0) \implies f(f(0))=0$

$P(x,1) \implies$ $x+f(x+1)$ is a fixed point.

$P(x,0) \implies f(x+f(x))+f(0)=x+f(x)$.

$P(0,x) \implies f(f(x))+f(0)=f(x)+xf(0)$.

Take $P(0,y+f(y+1))$. Using that $y+f(y+1)$ is a fixed point, we get $f(y+1)+y-1=0$ or $f(0)=0$.

Therefore, $f(0)=0$ or $f(x)=2-x$ for all $x$.

If $f(0)=0$, we now have by $P(x,0)$ that $x+f(x)$ are fixed points.

$P(x,-x)$ gives $f(x)+f(-x^2)=x-xf(x)$ and swapping $x$ and $-x$ gives $f(-x)+f(-x^2)=-x+xf(-x)$.

Subtract these two to get $f(x)-f(-x)=2x-xf(x)-xf(-x)$.

Plug $x=1$, get $f(1)=1$. Then $P(1,-1)$ gives $f(-1)=-1$.

Now $P(1,y)$ gives $f(1+f(y+1))+f(y)=1+f(y+1)+y$.

Set $y=x+f(x+1)$. Then $y, y+1$ are all fixed points.

This gives us $f(y+2)=y+2$, so $x+f(x-1)$ is a fixed point.

$P(x,-1)$ gives $f(x+f(x-1))+f(-x)=x+f(x-1)-f(x)$ and using that $x+f(x-1)$ is a fixed point gives $f$ is odd.

Since $f(x)-f(-x)=2x-xf(x)-xf(-x)$ and $f(x)=-f(-x)$, we get $f(x)=x$.

Therefore $f(x) \equiv x$ or $f(x) \equiv 2-x$.