Why Faddeev-Popov ghost cannot exist in external line?
On one hand, the S-matrix does not depend on the gauge-fixing condition. On the other hand, there exist a unitary gauge, where Faddeev-Popov ghosts decouple from the theory.
References:
M.D. Schwartz, QFT and the Standard Model, 2014; Section 28.4.
C. Itzykson & J.B. Zuber, QFT, 1985; Subsection 12-5-5.
Qmechanic is right, but his answer doesn't explain why we can't just consider the ghosts as physical and be done with it.
There are two main reasons why ghosts can't be considered physical.
- They violate spin-statistics (ghosts are scalar fermions).
- The S-matrix operator as it stands isn't unitary.
The problem can be traced back to the kinematics of gauge-fixing. Remember how in the $U(1)$ case we had a gauge-fixing constraint (e.g. Lorentz gauge condition), which after implementing it as a quantum operator constraint $C$ selected a unique subspace $\text{ker} C$ of physical states? Well, here we deal with a similar situation, plagued by additional technical difficulties because the group is non-abelian.
The Fock space of the gauge+ghost system of the Lagrangian mentioned in your question isn't physical. It contains negative-norm states (just like in the $U(1)$ case). As an example of the negative-norm state, consider a timelike polarized gauge boson $$ a_0^{\alpha\,\dagger} \left| 0 \right>.$$
Just like in the $U(1)$ case this can be solved by implementing a gauge condition constraint as a quantum operator and solving. However, we run into the following complication:
Solutions of the constraint no longer decompose into the physical and spurious (zero-norm) subspaces which can be treated separately, because dynamics of the theory mixes these two subspaces.
This can be traced back to the following fact: the current-conservation law contains a covariant derivative instead of the ordinary one, while the Lorentz gauge condition still operates with an ordinary partial derivative.
This difficulty can be successfully solved with help of BRST quantization technique. Existence of ghosts is essential for the BRST to work.
In conclusion: the S-matrix given by the quantization of the Lagrangian from your question gives the correct quantum dynamics of the quantum gauge field, but only when projected to a subspace of the naive Fock space given by the BRST cohomology. It also has a nice property of not mixing physical and unphysical degrees of freedom, meaning that we can use its full form in practical computations and only project down to the physical subspace afterwards.
That we can't use the extended Fock space is already obvious because of the two reasons given in the beginning of my answer.