What causes bifurcation?

To start with, it's worth being really clear on what the equation $$x_{n+1}=rx_n(1-x_n)$$ actually defines. If you have some fixed value of $r$ and $x_0$, this expression gives some infinite sequence of further values $x_1,\,x_2,\,x_3,\ldots$ via a recurrence. The tricky thing about a bifurcation diagram is that it tries to show how the behavior of this infinite sequence varies with $r$ - which is a lot of information to pack into a single plot. Let's first look at some simpler ways to visualize the behavior of this recurrence. For simplicity, I'll just start at $x_0=0.5$ throughout this post, although this is another parameter you could vary (it turns out that almost all values of $x_0$ lead to similar behavior in the long run - although this is not that easy to prove).

Let me first present several pictures that display this equation from a more familiar angle, where we just start at $x_0=0.5$ and plot the sequence $x_0,x_1,x_2,x_3,\ldots$ for some fixed values of $r$.

At $r=2.9$, before the bifurcation, we see that successive terms in the sequence rather quickly approach some single value (a fixed point of the recurrence).

The long term behavior of this sequence can be described as converging to a single value.

At $r=3$, the terms get closer together, but do so way more slowly - from the first $50$, it is not even apparent that they converge:

If we look at the next $5000$ terms, we can see they continue to get closer together:

And, if we do a bit of mathematics, we can find out that the sequence eventually converges to $2/3$. The sketch of the argument is to observe that if $x_n < 2/3$ some algebra can prove that $x_n < x_{n+2} < 2/3$ and a bit of analytical work can show actual convergence of $x_n$ - and we can even estimate the rate of convergence if we like.

If we keep increasing $r$, strange things start happening. At $r=3.05$, for instance, the terms don't get closer together:

Rather quickly, the terms settle into a pattern of alternation between a high value and a low value - and then don't move any more. The long term behavior here would be described as a two-cycle, where two values keep alternating - and you could find explicitly where that occurs by using the recurrence to solve the quadratic equation $x_{n+2}=x_n$.

Continue increasing more, say to $r=3.5$, and you get a pattern of four values:

And if you sort of start working in smaller and smaller increments of $r$, you find a pattern of eight values, then sixteen and so on - followed by, eventually, cycles of every length (appearing in the same order - and not by coincidence - as described in Sharkovskii's theorem). The last kind of cycle to appear (and of the major events one sees on a bifurcation diagram) is a three-cycle, which appears near $r=3.83$ and gives behavior like:

Where we see three values. There are also values of $r$ that display more complicated behavior that never becomes cyclic - although this poses certain problems in trying to get a computer to accurately plot it. One can get a taste of this sort of thing at $r=3.675$, which gives the first 500 terms as:

There's certainly some patterns - we can see a few shapes repeating irregularly in the middle - but it's not clear that a cycle is emerging (or will emerge). In fact, somewhere near this $r$ value there are some points that are easy-ish to prove do not tend to periodic behavior ever for most starting points $x_0$. This fact is also easy to show at $r=4$, but I promised to start my plots at $x_0=0.5$, and that starting point is not interesting at all at $r=4$.

Now, let me get to your question: a bifurcation diagram tries to display the information contained in these plots over a lot of $r$ all at once. Basically, it reduces each plot to a single vertical line, and plots on that line the values that appear in the long term behavior*. So, for $r\leq 3$, only a single point belongs on the bifurcation diagram, meaning that most points** tend there in the long term. In the next region, we see two points, meaning most points end up alternating between those two points in the long run. So, each vertical slice tells us what the sequence does after a lot of time for a fixed value of $r$ - and, as you can see, this is quite complex. A crucial point here is that the bifurcation diagram isn't trying to show a function - each vertical slice is essentially a distribution*** saying things like "the sequence will spend half of its time on the upper branch and half on the lower branch" or other such statements about behavior more complicated than identifying a single value of interest.

As to why it does that, it helps to define $f_r(x)=rx(1-x)$, which is a function defined to run one step of the iteration. There's a lot to say, but the basic idea is that if you plot this function for $r=2$, you get something like this, where I also plot the line $y=x$:

For this value of $r$, the iteration converges very quickly to $1/2$ - and we can see that, if we started at exactly $1/2$, we would end up exactly back there - which is called a fixed point. More strongly, however, this fixed point is attractive meaning that if we start near 1/2, we end up closer after one iteration - one can see this by the fact that the value of the function changes very slowly near the fixed point - it's just a big flat top, all of it mapping closer and closer to $1/2$ than we started. This leads to very fast convergence of the sequence.

The plot at $r=3$ looks different in substantial ways:

Note that there still is a fixed point - at $2/3$ now - but the behavior of things near $2/3$ is different. The slope of our function $f_3$ near that fixed point is $-1$ - meaning that if we start at $2/3+e$, the next iterate is around $2/3-e$, meaning we didn't get meaningfully closer - although if we look at a quadratic approximation, we can see that we are getting closer, just way more slowly.

At $r=3.1$, the slope near the fixed point is now greater than $1$ in absolute value - meaning if we start to near the fixed point, we actually end up further away after one iteration - this is called a repulsive fixed point.

This is why the behavior changes - it can't be that most points end up at a repulsive fixed point****, so something else must happen. On the plot, we see two branches emerge, so let's consider that.

If we increase $r$ a little more to $3.3$ and plot $f_3(f_3(x))$ as a third line, we get to see a bit more about why now a two-cycle prevails:

While the orange curve ($f_3(x)$) continues to get steeper and steeper near the fixed point - preventing the simple convergent behavior, the green curve ($f_3(f_3(x))$) crosses the $y=x$ line in two other places - and is fairly flat near those locations, meaning that, if we start near one of those places and advance two iterations, we get closer to where those lines cross the $y=x$ line - leading to an attractive two-cycle between the values of those intersections. As we increase $r$ further, what happens is that these new intersections get steeper and steeper until they become repulsive - leading to larger cycles (and eventually to chaotic behavior - but that's really hard to explain in a short post).

(*A common way to create such a plot is to iterate through each $r$, following the following procedure: first calculate $x_{1000}$, then plot the values $x_{1001},\,x_{1002},\ldots,x_{2000}$ on the vertical strip corresponding to that $r$. This basically just waits for the sequence to get close to where it converges, then plots what happens next. Obviously, more terms discarded at the start and plotted at the end give finer results, although at some point you run into the issue that computers do arithmetic with finite precision, which is insufficient to correctly represent the exact mathematical behavior)

(**For $r\leq 3$, this is actually true of every point, but this is stronger than what the bifurcation diagram wishes to communicate)

(***These slices really are distributions - while the really clear bits only involve finite cycles of varying lengths, where it's just as well to talk about some finite set or sequence associated to various values of $r$, if you get a really good quality plot of this, you can see things like, "the iteration behavior chaotically here, but it spends more time near some points than near others" - where there's no simple description of what's happening, yet there's still some structure. This sort of thing explains why, in high-quality plots, one can see various dark curves near things like repulsive cycles, where a value might spend a lot of time slowly getting further away, leading to more time spent near certain points even when those points fail to dominate the long term behavior)

(****It is worth noting that some points do end up at the repulsive fixed point - for instance, the fixed point itself will never leave, even if everything nearby gets pushed further. There is also a countable set of points with the property that, after finitely many iterations, they end up at the fixed point. Similarly, even after the two cycle stops attracting, some countably many points end up there - but this is exceptional behavior if you choose a point at random from $(0,1)$ for $x_0$ this kind of behavior almost never occurs)

First of all, congratulations for you work and your observations.

Since @Milo Brandt gave very nice explanations, I shall not repeat any of them.

You can "simplify" the quadratic recurrence equation $$x_{n+1}=r\,x_{n}(1-x_{n})$$ letting $$x_n=\frac{r-2 z_n}{2 r}\implies z_{n+1}=\frac r 4+\frac 1r z_n^2$$ and you enter in the domain of quadratic maps which, at leat for me, is not the simplest problem.

For example, the simplest $$c_{n+1}=c_n^2-k$$ with $c_0=0$ used to generate the Mandelbrot Set.

I shall not enter into more details but I suggest you have a look at this paper which, even if too complex for you at the time being, gives very good explanations about the bifurcation diagram. There are a few plots which are very illustrative.