What is the physical significance of the determinants of orthogonal matrices having the value of $\pm 1$?
It means that orthogonal transformations preserve volumes. That is so because, if you have an object $O$ and if $A$ is a linear transformation, then the volume of $A.O$ is the volume of $O$ times the absolute value of $\det A$.
Let $A$ be an orthogonal $n\times n$-matrix. Thus, $A^T A = I_n$ (where $I_n$ denotes the $n\times n$ identity matrix). Thus, $\det\left(A^T A\right) = \det\left(I_n\right) = 1$. But any two $n\times n$-matrices $X$ and $Y$ satisfy $\det\left(XY\right) = \det X \cdot \det Y$ (this is the famed multiplicativity property of the determinant). Applying this to $X = A^T$ and $Y = A$, we obtain $\det \left(A^T A\right) = \det A^T \cdot \det A = \det A \cdot \det A$ (since yet another known property of determinants says $\det A^T = \det A$). Hence, $1 = \det\left(A^T A\right) = \det A \cdot \det A = \left(\det A\right)^2$. Subtracting $1$ from this equality, we obtain $0 = \left(\det A\right)^2 - 1 = \left(\det A - 1\right)\left(\det A + 1\right)$. But a product of two (complex or real or rational) numbers can only be $0$ if one of them is $0$. Hence, from $\left(\det A - 1\right)\left(\det A + 1\right) = 0$, we obtain that either $\det A - 1 = 0$ or $\det A + 1 = 0$. In other words, either $\det A = 1$ or $\det A = -1$.
This holds for any matrix $A$ with complex (or real or rational) entries (or generally with entries in any field). The geometric intuition ("orthogonal matrix = congruence transformation") does not hold in this generality.
OK, let me expand on my (now deleted) comment. To give a complete answer to your question, let me introduce a few concepts. Let us work in $V=\mathbb{R}^n$. I will use the notation $(x,y)$ for the inner product of $x,y\in V$ (which is $x^Ty$ in $\mathbb{R}^n$).
Householder transformations: A householder transformation is defined as follows. Fix a unit vector $v\in V$ and define the linear transformation $T:V\to V$ as $$ T_v(x) = x-2(x,v)v $$ What this transformation does is as follows: The vector $v$ defines a $(n-1)$-dimensional hyperplane $H_v$ passing through the origin which is perpendicular to $v$. Our $T_v$ reflects any other vector $x$ with respect to this hyperplane.
I leave as an exercise to show that $T_v$ is orthogonal. Furthermore, note that if $x$ lies in the hyperplane $H_v$, i.e. $(x,v)=0$, then $T_v(x)=x$. However, $T_v(v)=-v$. This means that $v$ is the unique eigenvector of $T_v$ with eigenvalue $-1$. The other eigenvalue of $T$ is $+1$ with multiplicity $n-1$ (as the dimension of $H_v$ is $n-1$). As such $\det T_v=-1$.
Another exercise you can do is to show that the combination of two householder transformations is a rotation (note that in this case $\det = +1$). HINT: Let $u,v$ are the two vectors determining these householder transformations. Define $W=\mathrm{span}\{u,v\}$. Then $V=W\oplus W^\perp$. Note that if your transformation is $U$, then $U$ leaves $W^\perp$ invariant. So your problem is essentially 2-dimensional.
Finally, consider a diagonal matrix of the form $$ M=\mathrm{diag}(\underbrace{-1, \cdots,-1}_{m\text{ times}}, \underbrace{+1, \cdots,+1}_{n-m\text{times}})$$ if this is the matrix of a transformation $U$ in the standard basis $e_1, \cdots, e_n$, and $T_i=T_{e_i}$ is aforementioned householder transformation, then $$ U= T_m\circ \cdots \circ T_2\circ T_1 $$ More generally, suppose $U$ is any orthogonal transformtion, $M$ its matrix. Define the new orthonormal basis $b_i=U(e_i)$. Note that we can always find a rotation $R$, which sends $e_i$ either to $b_i$ or $-b_i$ (in other words the $x_i$ axis goes to $b_i$ axis). In the new basis $b_i$, the matrix is of the above form. In other words, combining everything we learned:
Any orthogonal transformation, is a combination of rotations and householder transformations (i.e. reflections). One can take this even further to: if $\det =+1$, the transformation has even number of reflections. If $\det =-1$ then the transformation has odd number of reflections. You can even work this further and show that all of this simplifies to $\det =+1\Longrightarrow$ one rotation, $\det =-1\Longrightarrow$ one reflection followed by one rotation.