Why does it make sense to talk about the first-quantized BRST formulation of a relativistic point particle?
You are not right. There is no orthogonal basis $|x,t\rangle$ for fixed $t$ in relativistic quantum mechanics, but there is a state $|x,t\rangle=|x^{\mu}\rangle$. In quantum field theory this state is expressed by $\phi(x)|0\rangle$, but this is a one-particle state, so can be expressed in the one-particle space, spanned by the momenta eigenstates $|\vec{p}\rangle$.
$$ |x^{\mu}\rangle=(2\pi)^{-3/2}\int \frac{d^{3}\vec{p}}{\sqrt{2E_p}}e^{ip_{\mu}x^{\mu}}|\vec{p}\rangle $$
The main point is that the state $|x\rangle$ does not imply the existence of a hermitian operator $x$, so this state can exist without the existence of $x$-operator. In fact, this operator can't exist since the Hamiltonian is bounded from bellow. You can see more about it here.
If the state $|x^{\mu}\rangle$ exist, makes perfect sense to have path integrals over trajectories $x_{\mu}(\tau)$, if we gauge away the parametrization at the end.
$$ \langle x_2^{\mu}|x_1^{\mu}\rangle=\int_{x(0)=x_1}^{x(1)=x_2}\frac{\left[dx(\tau)\right]}{\mathbb{diff}}\,e^{-S[x]} $$
What you will discover from that path integral is that $\langle x_2|x_1\rangle \neq 0$ even for space-like intervals, and by Lorentz symmetry you may deduce that the same happens for hyper-surface of simultaneity. This means that we cannot take $|x\rangle \rightarrow |\vec{x},t\rangle$ as an orthogonal basis for $t$ fixed.
This is why there is no wave function $\psi(x)$ description for the states. For the momentum wave-function there will be no problem, and a wave-function $\phi(p)$ is totally acceptable. For instance, $P$ is a hermitian operator for the one-particle Hilbert space in the free theory (free means no mixing with multi-particles states).
Actually the BRST-approach is far from be just pedagogical. You can obtain the field theory description by defining a field operator $\Psi(x^{\mu})$ and impose the equation of motion:
$$ Q\Psi=0 $$
This is useful to obtain non-linear generalization (i.e. how to put interactions) for particles in the presence of supersymmetry and/or gauge symmetry. The gauge transformations of the field will be described by the generalization of the BRST-exact fields $\delta\Psi=Q\Lambda+g\left[\Psi,\Lambda\right]$ and the equation of motion will be:
$$ Q\Psi + g\Psi^2=0\,. $$
You can see more about it here.
I believe there maybe a misunderstanding that is greatly clarified by the BRST method as presented in Polchinsky. It avoids the discussion in Ticciati's book since it contains a position operator in 4 rather than 3 dimensions. The Kinematical Hilbert space obtained by quantizing the action
$$S=\int d\lambda\left(\frac{1}{2}(\dot{x}^2+m^2)-\dot{b}c\right)$$
yields the operators $X^\mu$, $P_\mu$, $B$ and $C$ satisfying the canonical commutation relations $[X^\mu,P_\nu]=\delta^\mu_\nu$ and $\{B,C\}=1$. From Stone-von Neumann theorem, we know that there is a unique irreducible representation of these on the Hilbert space $L^2(\mathbb{R}^4)$ where $(X^\mu\psi)(x)=x^\mu\psi(x)$ and $(P_\mu\psi)(x)=-i\hbar\partial_\mu\psi(x)$. In particular, the Dirac delta $|x\rangle=\delta_{x}$, satisfying $(X^\mu\delta_x)(y)=y^\mu\delta(x-y)=x^\mu\delta(x-y)=x^\mu\delta_x(y)$, i.e. $X^\mu|x\rangle=x^\mu|x\rangle$, is in the theory. The set of such states are orthogonal.
Now, the canonical quantization procedure also yields the Hamiltonian $H=\frac{1}{2}(p^2+m^2)$ and the BRST operator $Q=cH$. Physical states are those for which $Q\psi=0$, i.e. those for which $H\psi=0$. Via Schrödinger's equation, a state $\phi:\mathbb{R}\rightarrow L^2(M)$ evolving in the parameter $\lambda$ which is physical would then satisfy $i\hbar\frac{d\phi(\lambda)}{d\lambda}=H\phi(\lambda)=0$, which is precisely what we want! States should not depend on the unphysical parameter $\lambda$. Indeed, a state $\psi\in L^2(M)$ already corresponds to the whole worldline of the particle. Now, we see that eventhough we do have a position operator and vectors $|x\rangle$, the latter are unphysical since they are not BRST closed.
Incidentally, $H\psi=0$ is the Klein-Gordon equation. Usually, it is claimed in QFT classes that this is not a satisfactory equation for a wave function since the norm obtained by integrating over three space is not conserved. However, we now see that the real norm must be obtained by integrating over the whole four dimensional space, which doesn't lead to any trouble. On the other hand, the conserved quantity given by the Klein-Gordon Lagrangian doesn't have to be interpreted as a probability density but rather as a charge density. Then there is no problem with the fact that it is not positive definite.
Let me note that BRST quantization does seem to be important for this analysis. If one tries to quantize a gauge fixed action like $$S=-m\int\sqrt{1-\|\vec{v}\|^2}$$ directly, one gets to a Schrödinger equation $$i\hbar\frac{d\phi}{dt}=\sqrt{-\hbar^2\Delta+m^2}\phi(t)$$ which is non-local. I don't really know what is happening here but I guess this is why people use BRST quantization.
I hope I didn't miss any obvious details of something that makes my arguments invalid hahahah. Let me finalize by stating that
- I don't know of a way to introduce interactions into this constructions. If there is, it should only be possible for very specific interactions which wouldn't violate particle number conservation. This is why QFT is very important to describe the interesting part of the relativistic theory of particles. One can however do perturbative QFT in a "first quantized" point of view using the action above and Feynman diagrams, thus bypassing the need of say the Klein-Gordon action.
- This "first quantized" point of view is very important for string theory where the string field theory (the analogue of Klein-Gordon for the spinless relativistic particle) is difficult to obtain, but the analog of the action above (the Polyakov action) is readily available.
I don't really understand much of string theory but those two remarks where made in my class.