# Underbrace under square root sign plain TeX

You could (a) apply a \smash instruction to the \underbrace{...}_{...} part and (b) insert a \vphantom directive after the \sqrt{...} part.

$$\sqrt{ \pmatrix{A^{-1} \cr -1}^{\!T} \smash{\underbrace{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}}% _{\textstyle A_{\Phi, X\cup\{x\}}}} \pmatrix{A^{-1} b(x) \cr -1} } \vphantom{\underbrace{\pmatrix{.\cr .}}_{\textstyle A_{\Phi}}}$$
\bye


Observe that I've also dropped some \displaystyle directives and have converted others to \textstyle.

TeX can measure the object:

$$\def\contentsA{\pmatrix{A^{-1} b(x) \cr -1}} \def\contentsB{\pmatrix{A & b(x) \cr b(x)^T & \Phi(x,x)}} \setbox0=\hbox{\displaystyle \contentsA^{\!T} {\underbrace{\!\contentsB\!}_{\displaystyle A_{\Phi, X\cup\{x\}}}} \contentsA } \sqrt{\vphantom{\contentsA^T}\hphantom{\copy0}} \kern-\wd0 \box0$$

\bye


I made a couple of changes: braces like {\underbrace{...}_{...}} in order to avoid it being considered a math operator with an excess of spacing; for the same reason I introduced a couple of \! tokens. And also one for the exponent in order to place it nearer the top of the parenthesis.