What's the relationship between $SL(2,\mathbb{C})$, $SU(2)\times SU(2)$ and $SO(1,3)$?

Here's my two cents worth.

Why Lie Algebras?

First I'm just going to talk about Lie algebras. These capture almost all information about the underlying group. The only information omitted is the discrete symmetries of the theory. But in quantum mechanics we usually deal with these separately, so that's fine.

The Lorentz Lie Algebra

It turns out that the Lie algebra of the Lorentz group is isomorphic to that of $SL(2,\mathbb{C})$. Mathematically we write this (using Fraktur font for Lie algebras)

$$\mathfrak{so}(3,1)\cong \mathfrak{sl}(2,\mathbb{C})$$

This makes sense since $\mathfrak{sl}(2,\mathbb{C})$ is non-compact, just like the Lorentz group.

Representing the Situation

When we do quantum mechanics, we want our states to live in a vector space that forms a representation for our symmetry group. We live in a real world, so we should consider real representations of $\mathfrak{sl}(2,\mathbb{C})$.

A bit of thought will convince you of the following.

Fact: real representations of a Lie algebra are in one-to-one correspondence (bijection) with complex representations of its complexification.

That sounds quite technical, but it's actually simple. It just says that we can have complex vector spaces for our quantum mechanical states! That is, provided we use complex coefficients for our Lie algebra $\mathfrak{sl}(2,\mathbb{C})$.

When we complexify $\mathfrak{sl}(2,\mathbb{C})$ we get a direct sum of two copies of it. Mathematically we write

$$\mathfrak{sl}(2,\mathbb{C})_{\mathbb{C}} = \mathfrak{sl}(2,\mathbb{C}) \oplus \mathfrak{sl}(2,\mathbb{C})$$

So Where Does $SU(2)$ Come In?

So we're looking for complex representations of $\mathfrak{sl}(2,\mathbb{C}) \oplus \mathfrak{sl}(2,\mathbb{C})$. But these just come from a tensor product of two representations of $\mathfrak{sl}(2,\mathbb{C})$. These are usually labelled by a pair of numbers, like so

$$|\psi \rangle \textrm{ lives in the } (i,j) \textrm{ representation of } \mathfrak{sl}(2,\mathbb{C}) \oplus \mathfrak{sl}(2,\mathbb{C})$$

So what are the possible representations of $\mathfrak{sl}(2,\mathbb{C})$? Here we can use our fact again. It turns out that $\mathfrak{sl}(2,\mathbb{C})$ is the complexification of $\mathfrak{su}(2)$. But we know that the real representations of $\mathfrak{su}(2)$ are the spin representations!

So really the numbers $i$ and $j$ label the angular momentum and spin of particles. From this perspective you can see that spin is a consequence of special relativity!

What about Compactness?

This tortuous journey shows you that things aren't really as simple as Ryder makes out. You are absolutely right that

$$\mathfrak{su}(2)\oplus \mathfrak{su}(2) \neq \mathfrak{so}(3,1)$$

since the LHS is compact but the RHS isn't! But my arguments above show that compactness is not a property that survives the complexification procedure. It's my "fact" above that ties everything together.

Interestingly in Euclidean signature one does have that

$$\mathfrak{su}(2)\oplus \mathfrak{su}(2) = \mathfrak{so}(4)$$

You may know that QFT is closely related to statistical physics via Wick rotation. So this observation demonstrates that Ryder's intuitive story is good, even if his mathematical claim is imprecise.

Let me know if you need any more help!

Firstly, what book is this? It will help greatly if I can reference it myself.

It is highly likely that when he says $\mbox{SO}(1,3)$ [or $\mbox{SO}(3,1)$!] that he means $\mbox{SO}(1,3)_\uparrow$, which is absolutely not the same! But most people are very lazy about this.

Here you're picking out the region of $\mbox{O}(1,3)$ path-connected to the identity element, where $\mbox{O}(1,3)$ consists of four disconnected regions, labeled by

$$ \det(L) = \pm1$$

and

$$ L^{00} > 1 \space \mbox{ or } \space L^{00} < -1 $$

Then we have

$$ (\mbox{SU}(2) \times \mbox{SU}(2))/ \mathbb{Z_2} \simeq \mbox{SO}(4) $$

You can show this by considering the action of each of $\mbox{SU}(2)$ and $\mbox{SO}(4)$ on 2-complex-dimensional and 4-dimensional vectors respectively. You'll find that

$$ (x^1)^2 + (y^1)^2 + (x^2)^2 + (y^2)^2 = 1 $$

and

$$ (x^1)^2 + (x^2)^2 + (x^3)^2 + (x^4)^2 = 1 $$

respectively, up to normalisation. Here we have to quotient out $\mathbb{Z_2}$ since we want only those $U \in \mbox{SU}(2) $ which have

$$ \det(U) = 1 $$

Then $\mbox{SO}(4)$ is to Euclidean space as $\mbox{SO}(1,3)$ is to Minkowski space (using the Russian metric). This is why he says that $\mbox{SO}(1,3)$ is essentially $\mbox{SU}(2) \times \mbox{SU}(2)$, but shys away of declaring that the former is the latter (which would be an incorrect statement).

Next, you need to recall that $\mbox{SL}(2,\mathbb{C})$ is the real part of the complexification of $\mbox{SU}(2) \times \mbox{SU}(2)$. That is, $\mbox{SL}(2,\mathbb{C})$ is a double cover of $\mbox{SU}(2)$. This is because, when you complexify, and then take the real parts, you get two copies of $\mbox{SU}(2)$. Think of the way when we 'complexify' $\mathbb{R}$ and get $\mathbb{C}$, and we know that we can always write, for $z \in \mathbb{C}$

$$ z = x + i y $$

where $x, y \in \mathbb{R}$. So if we took the real parts of $\mathbb{C}$ we would get two copies of $\mathbb{R}$, the $x$ and the $y$, $$ \mathbb{C} \simeq \mathbb{R^2} $$

We can do the same thing with Lie Algebras, since they are just vector spaces after all, just like $\mathbb{R}$ and $\mathbb{C}$ are (maybe a little less trivial however!).