How to find $f$ and $g$ if $f\circ g$ and $g\circ f$ are given?

If it is given that $f$ and $g$ are polynomials then the product of their degrees must be $2$ (the degree of the composition of two polynomials is the product of their individual degrees), therefore one of the two must be quadratic and the other linear. I do not think that you can assume $\deg f=2$ without loss of generality.

If $\deg f=2$ and $\deg g=1$ then $g$ must be increasing, i.e., its leading coefficient is positive. The graph of $f,$ like the graph of $f\circ g,$ is tangent to the $X$ axis (discriminant $0$) and $f\circ g$ reaches its minimum $0$ when $x=-\frac12.$ The minimum of $g\circ f$ is $g(f(-1))=1$ and, because $g$ is strictly increasing, is reached when $f(x)$ reaches its minimum $0.$ Therefore the $Y$-intercept of $g$ is 1. This also tells us that $f$ reaches its minimum at $x=-1$ and we already know that $g(-\frac12)=-1$. Therefore

$$g(x)=4x+1;\ f(x)=a(x+1)^2$$

Plugging this into $g(f(x))=x^2+2x+2$ gives $a=\frac14,$ and then into $f(g(x))=4x^2+4x+1$ verifies this.

Now suppose $\deg g=2$ and $\deg f=1.$ Then $f$ must increasing (positive leading coefficient). The graph of $g$, like the graph of $g\circ f,$ has minimum $1.$ The function $g\circ f$ reaches this minimum when $x=-1.$ The minimum of $f\circ g$ is $f(g(-\frac12))=0$ and, because $f$ is strictly increasing, is reached when $g(x)$ reaches its minimum $1.$ Therefore $f(1)=0.$ This also tells us that $g$ reaches its minimum at $x=-\frac12$ and we already know that $f(-1)=-\frac12.$ Therefore

$$f(x)=\frac14x-\frac14;\ g(x)=a(2x+1)^2+1$$

Plugging this into $f(g(x))=4x^2+4x+1$ gives $a=4,$ and then into $g(f(x))=x^2+2x+2$ verifies the result.

Without any restriction on $f$ and $g$ the solutions can be very wild. If $(f_1,g_1)$ and $(f_2,g_2)$ are distinct solutions where all 4 functions leave a proper subset $A\subset\mathbb R$ and its complement intact, then two new solutions can be obtained by applying one solution to the numbers in $A$ and the other to its complement. As an example, the two polynomial solutions above map algebraic numbers to algebraic numbers and transcendental numbers to transcendental numbers.


If you disregard the requirement that $f(x)$ be real, there is a non-polynomial solution

$$ f(x) = \dfrac{\sqrt{x-1}}{4} -\dfrac{1}{2},\ g(x) = 1 + 4 (8 x^2 + 8 x + 3)^2$$