Normal Order of Normal Order

  1. Short explanation: Polchinski's eq. (1) is not a formula that transforms no normal order into normal order: The expression ${\cal F}$ on the right-hand side of eq. (1) is implicitly assumed to be radially ordered. In fact, eq. (1) is a Wick theorem for changing radial order into normal order, cf. e.g. this Phys.SE post.

  2. Longer explanation: When dealing with non-commutative operators, say $\hat{X}$ and $\hat{P}$, the "function of operators" $f(\hat{X},\hat{P})$ does not make sense unless one specifies an operator ordering prescription (such as, e.g., radial ordering, time-ordering, Wick/normal ordering, Weyl/symmetric ordering, etc.). A more rigorous way is to introduce a correspondence map $$\begin{array}{c} \text{Symbols/Functions}\cr\cr \updownarrow\cr\cr\text{Operators}\end{array}\tag{A}$$ (E.g. the correspondence map from Weyl symbols to operators is explained in this Phys.SE post.) To define an operator $\hat{\cal O}$ on operators, one often give the corresponding operator ${\cal O}$ on symbols/functions, i.e., $$ \begin{array}{ccc} \text{Normal-Ordered Symbols/Fcts}&\stackrel{\cal O}{\longrightarrow} & \text{Radial-Ordered Symbols/Fcts} \cr\cr \updownarrow &&\updownarrow\cr\cr \text{Normal-Ordered Operators}&\stackrel{\hat{\cal O}}{\longrightarrow} & \text{Radial-Ordered Operators}\end{array}\tag{B}$$ E.g. Polchinski's differential operator ${\cal O}$ does strictly speaking only make sense if it acts on symbols/functions. The identification (A) of symbols and operators is implicitly implied in Polchinski.

  3. Concerning idempotency of normal ordering, see also e.g. this related Phys.SE post.

Applying normal ordering twice is not a valid operation. In fact, "normal ordering" is not a proper operator at all, and only defined on a single product of operators. It is undefined when acting on sums of operators, since trying to extend it linearly fails: $$ :a a^\dagger: = : a^\dagger a + 1 : = :a^\dagger a: + :1: = a^\dagger a + 1 \neq a^\dagger a = :a a^\dagger,$$ which is a contradiction. However, $\mathcal{O}$ clearly is a linear operator, and therefore the equation $: F : = \mathcal{O} F$ is only meant to hold when $F$ is a product of operators, but not their sum.

If you examine the counterexample to linearity above, you will realize this means that normal ordering isn't defined as a function in the mathematical sense on operators at all, since in $a a^\dagger = a^\dagger a +1$ the l.h.s. and the r.h.s. are the exact same mathematical object, yet normal ordering can only be applied to one of them. It is best to think of normal ordering as acting on symbols, by which I mean strings composed of $a$ and $a^\dagger$. Every operator has, if it can be written as a product of $a$ and $a^\dagger$ without any sum, exactly one such representation, so there is a well-defined map from operators to such symbols, to which you then can apply normal ordering, and then turn them back into operators again.