What makes the Cauchy principal value the "correct" value for a integral?
The Cauchy principal value is very important, especially in cases where the Lebesgue integral (which it seems you refer to as the improper integral) does not exist. The issue is that the Lebesgue integral doesn't deal too well with really big oscillations. Indeed, a measurable function $f$ is Lebesgue integrable if and only if $|f|$ is, so oscillations don't matter - only the magnitude does. This becomes problematic when dealing with functions such as $\frac{\sin x}{x}$ on $(0,\infty)$ or $\frac{1}{x}$ on $(-1,1)\setminus\{0\}$.
A purpose of the Cauchy principal value is to rectify this problem, to take into account oscillations like the Riemann integral does and give a meaningful number that represents the integral (i.e. scaled average) of the function in question. The Cauchy p.v. of $\frac{1}{x}$ is $\lim_{\epsilon \to 0} \int_{-1}^{-\epsilon} \frac{1}{x}dx+\int_{\epsilon}^1 \frac{1}{x}dx = 0$, which coincide with our intuition for what the average value of $\frac{1}{x}$ should be.
The most prominent use of the Cauchy principal value is the Hilbert transform, in which we study $\int \frac{f(x-y)}{y}dy$, which of course, needs to be defined properly. It is critical here that we don't just say the improper integral exists, but rather get a quantitative sense of the oscillation present.
A physics related example may also be useful. Consider a potential of the form $V=-Log(\left|x\right|)$. Then the force is given by $$ F=-{d V \over d x}={1\over x}. $$ The work done on a particle which moves from $-a$ to $a$ (where $a>0$) is $$ W=\int_{-a}^a{1\over x} dx. $$ From conservation of energy, we would expect $W=0$ as $V(a)=V(-a)$ and so, in that case, the Cauchy principal value gives the correct answer. However, in general, we would expect that it is not physically possible to have an infinite potential and so some new physics would come in near to $x=0$ which would change the shape of the potential in that region. But, provided that energy is still conserved we would still get $W=0$ when moving the particle from $-a$ to $a$ and so the Cauchy principle value would still have given the correct answer even without accounting directly for the new physics.
However, if you examine the case of $V=1/x$, which gives $F=1/x^2$ then it can be seen that Cauchy principle value gives infinity whereas a contour based integration in the complex plane will give a result consistent with the conservation of energy. So, in general, the contour integration approach is the one best suited for physics.
I like Abhimanyu Pallavi Sudhir answer: we can assign the integral $\int_{-1}^1\tfrac1x\,dx$ the value $\infty$, $-\infty$ or any other number, and we would be equally right. There is no criterion to select a result over the other. Sometimes this is presented as a flaw of the Lebesgue integral, which I think is nonsense; this integral just does not have an unambiguous meaning and it is not the fault of Lebesgue or of Riemann.
If, however, we consider the integral operator $$ T(f):=\int_{-1}^1 \frac{f(x)}{x}\, dx, $$ where $f$ is an arbitrary smooth function, then things are different. Let us develop $f$ in a Taylor series; we get, formally, $$ \int_{-1}^1 \frac{f(x)}{x}\, dx = f(0)\int_{-1}^1\frac{dx}{x} + f'(0)\int_{-1}^1\, dx + \ldots$$ The only ambiguous term is the zeroeth-order one; all the others make perfectly sense as standard integrals.
Conclusion. We need a renormalization procedure that removes the zeroeth-order term. This procedure amounts to exploit symmetry, prescribing that $\int_{-1}^1 \frac{1}{x}\, dx$ should be interpreted as $$ \lim_{\epsilon \to 0}\int_{\epsilon<\lvert x \rvert <1} \frac{dx}{x}=0,$$ that is, considering integrals in the principal value sense.
This is the starting point of the Calderón-Zygmund theory of singular integrals. In general, given $$ K(x)=\frac{\Omega(\tfrac{x}{|x|})}{|x|^n}, $$ where $\Omega$ is a function on the unit sphere $\mathbb S^{n-1}$ with zero integral, that is, $$ \int_{\mathbb S^{n-1}}\Omega\, dS =0, $$ we can define an integral operator $$ Sf(x):=\int_{\mathbb R^n} K(x-y)f(y)\, dy, $$ where the integral is interpreted in the principal value sense, exactly as before. Some conditions must be put on $K$ to ensure that this operator is continuous in appropriate function spaces. But most importantly, this definition based on the principal value is not crazy, or arbitrary, as it might seem at first sight. It is the only possible choice that removes the singular term from the integral, while leaving all the other terms untouched.
A final remark. I think that "principal value" is a gallicism, which obscures the true meaning. "Renormalization" would be better, as Gerry Folland suggests in his book "Real Analysis".