What exactly are fractals
First, a distinction should be made: a fractal is one thing, and certain methods for constructing particular fractals are another.
Loosely, a fractal can be described as an object which is self similar at different scales, that is, "zooming in" repeatedly leads to the same curve. An interesting property of fractals which is sometimes used to define them, is that one can assign a non integer dimension to them. For instance, a smooth curve has dimension 1, but a Koch snowflake is in a certain sense closer to being a two dimensional object, and we can assign it a non integer dimension of $\sim1.26$. Intuitively, A Sierpinski carpet is even closer to a 2D object, and indeed we assign it a higher fractal dimension, of $\sim1.89$.
As to Mandelbrot's famous set, the idea is as follows: To check whether or not a (complex valued) point $c$ is in the set, start with $z_0=0$, and iterate. When the series stays bound, $c$ is in the set. When the series diverges, $c$ is not in the set. (try $c=-1,0,1$ for yourself, and see what you get). For instance, is the point $i$, i.e. $(0,1)$ in the Mandelbrot set? $$z_1 = 0^2 + i = i,\quad z_2 = i^2 + i = i - 1,\quad z_3 = (i-1)^2+i= -i$$ $$z_4 = (-i)^2+i = i-1$$ Thus, the point $i$ leads to a bound repeated loop, and is therefore in the Mandelbrot set (i.e. the black area in most drawings).
A fractal is perhaps best defined as a set that is complex at an arbitrarily small scale. Whereas a smooth curve looks like a line if you zoom in close enough, a fractal never looks like anything so simple no matter how far you zoom in. See, for instance, this video.
As a special case, for many fractals, zooming in on a small region (and clipping it) will produce a set identical to the original fractal, and this holds true no matter how far you zoom in. Such fractals are called self-similar.
Concerning the definition of the Mandelbrot set: For functional notation, let $f^n(z)$ denote $$\underbrace{f \circ f \circ \cdots \circ f}_{n\text{ times}}$$ For a fixed point $c \in \mathbb C$, let $f_c$ be the function $f_c(z) = z^2 + c$. Then $c$ is a member of the Mandelbrot set if and only if $\lvert f_c^n(0)\rvert\not\to \infty$ as $n \to \infty$. If $c$ happens to be a real number, then it is true that $f^n_c(0)$ is real for all values of $n$. There are, roughly, two reasons to bring complex numbers into the picture:
- You care about what happens to complex numbers.
- You want to get a pretty picture. (In less glib language, you find the Mandelbrot set intrinsically interesting, even if it is considered only as a subset of $\mathbb R^2$, and complex numbers are the easiest way to define it.)
Update: Certain geometric figures are differentiable (i.e., $C^1$) and yet seem to be complex at arbitrarily small scales: see the bottom of the page in the Hevea project. Should these be considered fractals? If one defines fractals based on the fractal dimension, then the answer is "no." But I'm inclined to think this highlights a deficiency in the fractal dimension-based definition. The authors of the Hevea project call these objects "$C^1$ fractals."
Mathematically, the term is "fractal set". A fractal set is a set with a non-integer Hausdorff dimension, which is a generalization of what you normally think of "dimension" (e.g. the Hausdorff dimension of a line is 1; of a plane is 2).
The most simple example of a fractal set is the Cantor set:
The image represents the construction of the set and the set is defined in the limit as this construction goes to infinity (i.e. as it goes downwards). It is possible to compute the dimension of this set, which is $\log(2)/\log(3)\notin \mathbb{N}$.
Any other way to define a fractal is ambiguous and prone to magicality. Given the amount of subtleties in this topic, sticking to some mathematical rigor helps.
Relation with chaos
The equation you wrote, more generally written as $x_{n+1}=F(x_n)$, is a dynamical system in discrete time (also called a map).
The reason why they normally appear related with fractal sets is that chaotic dynamical systems generally have fractal sets. Skipping a full undergrad book, the most notable relation between fractals and chaotic systems is that the dimension of the attractor is related with the Lyapunov exponent of the system, i.e. a topology property (dimension) is related with a dynamical property (Lyapunov exponent).
To better illustrate this point, consider the following chaotic map:
$$x_{n+1} = 3 x_n\ \ for\ \ 0 < x_n < 1/2;\ \ \ \ \ \ x_{n+1} = 3 (1 - x_n)\ \ for\ \ 1/2 < x_n < 1$$
On each iteration of the map there are some points that stay inside the interval [0,1], and some points that leave the interval.
The points that stay up to time $n^*$ are exactly the construction of Cantor set in the $n^*$ iteration. Those that never leave are the Cantor set.
This and much more can be found in the fascinating (and old) topic of Chaos... ^_^