Why are symmetry transformations connected to the identity necessarily represented by linear unitary operators?

Well, the point is not continuity of the representation (if continuity is referred to the representation, see the final comment), but a related deeper fact concerning the group itself (though some vague argument based on continuity also of the representation is recurrent in the literature).

One has also to remember that (Wigner's theorem) symmetries are represented by unitary/antiunitary operators up to phases. This is in agreement with the basic fact of QM in Hilbert spaces that (pure) states are represented by unit vectors up to arbitrary phases.

Groups of symmetries are therefore represented by maps $G \ni g\mapsto U_g$ where

(a) $U_g$ is defined up to a phase and

(b) the requirement concerning preservation of the group structure is made weaker ($e$ is the neutral element of $G$): $$U_e= e^{i\gamma(e)} I\:,\quad U_g U_f =e^{i\gamma(g,f)} U_{g\circ f}$$ for some phases related with the choices of the unitary/antiunitary operators $U_h$.

I stress that, in view of Wigner's theorem, the unitary/antiunitary character of $U_g$ depends on $g$ only (if the Hilbert space has dimension greater than $1$) so, the remaining ambiguity in fixing the map $g \mapsto U_g$ concerns only the arbitrary phase and not the charcter of the operator $U_g$.

A map satisfying (a) and (b) (with a precise choice of the operators $U_g$) is called unitary projective representation of $G$.

In some cases, especially when $G$ is equipped with further structures as the Lie-group one, it is possible to redefine the operators $U_g$ by means suitable phases, $U'_g = \omega_g U_g$, in order to end up with a standard group representation $$U'_e= I\:,\quad U'_g U'_f = U'_{g\circ f}\:.$$ This is a difficult co-homological problem with important results, like Bargmann's theorem, I will not address here (Varadarajan's book of the geometry of QM include a list of relevant and very delicate results in this area of the theory of group representation).

Coming back to the main question, consider a group $G$ such that every element $g\in G$ can be decomposed into the product $g= h^2 = h\circ h$. A trivial but physically fundamental example is $\mathbb R$ as additive group.

Next, consider a unitary projective representation $G \ni g \to U_g$ associating every $g$ with a unitary/antiunitary operator $U_g$.

The relation $g=h\circ h$, taking (b) into account, implies $$U_g = e^{i\gamma(h,g)} U_h U_h\:.$$

No matter if $U_h$ is unitary or antiunitary, the composition $U_hU_h$ is unitary and the phases do not play any role. $U_g$ is necessarily unitary here!

In particular, every unitary projective representation of $\mathbb R$ is necessarily made of unitary operators (even if phases are not fixed and if the representation has no continuity property). This is one of the starting points to formulate the quantum version of Noether theorem.

In summary, if $G$ is such that $g= h^2$ for every $g\in G$ and an associated $h\in G$, then every representation up to phases must be made of unitary operators only.

The same argument extends to the more complicated case where $g = h_1^2 \circ \cdots \circ h_N^2$.

Now consider a Lie group $G$.

There is only one connected component $G_0$ which is a Lie subgroup as well, the one containing the neutral element $e$. For instance, this component is $SU(2)$ if the full group is $U(2)$, the orthochronous special Poincaré group for the Poincaré group, $SO(3)$ for $O(3)$, and so on.

In a sufficiently small neighborhood $A$ (which can be fixed to satisfy $A=A^{-1}$) of $e$ in $G_0$, every element $g \in A$ can be written as $g = \exp(t T)$ for some element $T$ of the Lie algebra of $G_0$ and some $t\geq 0$. Therefore $g= h^2$ where $h = \exp((t/2) T)$.

Finally, as for every topological connected group, it is possible to prove that $g \in G_0$ is the product of a finite number (depending on $g$) of elements in every neighborhood $O$ of the neutral element, so that, taking $A=O$, $$g = h_1^2 \circ \cdots \circ h_N^2$$

Therefore we can conclude that

THEOREM. Representing on a Hilbert space of dimension $>1$ a Lie group $G$ in terms of unitary/antiunitary operators up to phases according with Wigner's theorem -- i.e. by means of a unitary projective representation $G\ni g \mapsto U_g$ -- the elements of the connected component of $G$ containing the identity can only be represented by unitary operators.

Antiunitary operators may arise when representing the full Lie group if it has more than a connected component. In particular, the elements connecting different components may be (but also may not) represented by antiunitary operators. A typical example is the time reversal operation in $O(3,1)$ which does not belong to $SO(3,1)_+$.

Perhaps the continuity property mentioned by Weinberg is not referring to the representation but to the elements of the group. The connected component $G_0$ including the neutral element $e$ is exactly made of the elements of $G$ which can be connected to $e$ by means of a continuous curve of elements of $G$.


Wigners theorem says:

any symmetry transformation of ray space is represented by a linear and unitary or antilinear and antiunitary transformation of Hilbert space. The representation of a symmetry group on Hilbert space is either an ordinary representation or a projective representation.

It seems likely that the space of symmetries of ray space splits into two connected pieces, one consisting of unitary transformations and the other of anti-unitary transformations. But this is, to be honest, a restatement of what Weinberg says in the extract shown. The actual proof of this probably relies on a close examination of Wigner's theorem.

Here's Bargman's paper proving Wigner's theorem; the original theorem is proven in Wigner's book, Group theory and its application to the the quantum mechanics of atomic spectra.