How exactly is "normal-ordering an operator" defined?

An axiomatic definition of normal order can be found in the book "Solitons: Differential Equations, Symmetries and Infinite Dimensional Algebra" by T. Miwa, M. Jimbo and E. Date at page 44-46. This is the best definition I have found so far and goes as follow for products of bosonic operators. Call $\mathcal{A}$ the set of linear combinations of formal finite products of bosonic operators $b_i,\,b_i^\dagger$. The normal order ${:} a {:}$ of $a \in\mathcal{A}$ is a notation defined inductively by the properties

1. Linearity, $${:} z_1a_1+z_2a_2 {:}= z_1{:} a_1 {:} + z_2 {:} a_2 {:}\quad \text{for} \quad z_1,\,z_2\in \mathbb{C}\quad \text{and}\quad a_1,\,a_2 \in \mathcal{A}$$
2. ${:} 1 {:} = 1$, with $1$ the identity operator in $\mathcal{A}$
3. within the dots all the operators $b_i,\,b_i^\dagger$ commute among themselves
4. the annihilation operators can be taken out of the columns on the right $${:}a b_i{:} = {:} a{:}\,b_i$$
5. the creation operators can be taken out of the columns on the left $${:}b_i^\dagger a{:} = b_i^\dagger\,{:} a{:}$$

The definition for fermionic operators is the same with the only difference that the fermionic operators anticommute among themselves (Property 3). It is important to stress that usually annihilation operators are those operators that annihilate a specified vacuum state, therefore the normal ordering depends on the choice of the vacuum.

The normal order is a notation and not a function that acts on operators (i.e. a superoperator). This means that, whereas $b_ib_i^\dagger$ and $b_i^\dagger b_i +1$ are the same operators according to the canonical commutations relations, they are represented as different elements of $\mathcal{A}$ which is the set of linear combinations of strings of symbols generated by $b_i,b_i^\dagger$. In mathematical terms the normal order is a function defined on the elements of of the free algebra generated by $b_i,b_i^\dagger$, but is not a well defined function on the CCR-algebra (Canonical Commutation Relations algebra). The wrong step that leads to the paradoxical result

$$b_ib_i^\dagger = b_i^\dagger b_i +1 \Rightarrow {:}b_ib_i^\dagger {:} = {:}b_i^\dagger b_i +1{:} = {:}b_i^\dagger b_i{:} + 1 = {:}b_ib_i^\dagger{:} + 1 \Rightarrow 0 = 1$$

is actually the first equality since $b_ib_i^\dagger \neq b_i^\dagger b_i +1$ in $\mathcal{A}$. In another post it was suggested that the normal product is undefined when acting on linear combinations. However this would seriously limit the usefulness of the normal order. For example it is common to take the normal order of infinite series such as exponentials. The definition as a function acting on the free algebra $\mathcal{A}$ is in fact the one employed in practice.

This is a good example of how being mathematically rigor should not be regarded as a nuisance in the physics community, but rather as an important tool to avoid misunderstanding and confusion.

1. The main point is that the normal ordering procedure $:~:$ does not take operators to operators, but symbols/functions to operators. This important point resolves various paradoxes created by abuse of language.

2. That is, if $a$ and $a^{\ast}$ denotes the symbols/functions corresponding to the operators $\hat{a}$ and $\hat{a}^{\dagger}$, respectively, then the normal order satisfies $$:aa^{\ast}:~=~:a^{\ast}a:~=~\hat{a}^{\dagger}\hat{a},$$ and so forth.

3. For a full explanation, see e.g. this Phys.SE post and links therein.

In quantum mechanics with finitely many degrees of freedom, normal ordering always means applying the commutation rules to move creators to the left of annihilators, resulting in a unique final expression equal to the original one in the operator sense.

In quantum field theory (i.e., quantum mechanics with infinitely many degrees of freedom) proceeding this way usually is impossible as it leads to ill-defined coefficients. Therefore, in quantum field theory, normal ordering always means permuting creators to the left of annihilators without any consideration of commutation rules (i.e., on the classical expression). However the sign is changed when permuting two fermionic operators. In this way otherwise ill-defined expressions make sense at least as quadratic forms. In spacetime dimension $>2$, further renormalization is needed to turn the expressions into true operators.