How to have overlapping under-braces and over-braces

\documentclass[12pt]{article}
\usepackage{amsmath}
\begin{document}
\[
  a+b+\rlap{$\overbrace{\phantom{c+d+e+f+g}}^x$}c+d
     +\underbrace{e+f+g+h+i}_y +k+l=e^2
\]
\end{document}

enter image description here


In Herbert's Mathmode 63.2 Overlapping braces, he uses a similar method with align, \hphantom and \\[-11pt].

I think it is better to use LaTeX's \ooalign:

\[
\ooalign{
  $a+b+\overbrace{c+d+e+f+g}^{x}+h+i+k+l=e^2$\cr
  $\phantom{a+b+c+d+{}} {\underbrace{\phantom{e+f+g+h+i}}_{y}} $\cr
}
\]

Note: \ooalign is defined in LaTeX kernel, and is used in definition of some special text accents and math symbols. It is defined with primitive \halign. Similar approach is widely used in LaTeX kernel to define overlapping symbols — like \angle, \cong, \notin, \overrightarrow and \overbrace, etc.


Looking at the definitions of the \overbrace and \underbrace macros reveals that the are using \ialign which \halign with a zero \tabskip and empty \everycr.

So the braces are on there own "table rows". It is possible to define a macro which does take three arguments: the first part only under the overbrace, the middle part between both and the last part only over the underbrace. Then place this into a similar \ialign structure with three cells and three rows. The braces are then put into the plainTeX equivalent of \multicolumn{2} (\span\omit as I understand it).

However, the super- and subscript doesn't work as normal and must be read manual and placed at the correct positions. Getting them to the correct vertical position was the most difficult thing to get right.

The code:

\documentclass{article}

\iffalse
% The definitions of the existing macros for study:

\overbrace:
macro:#1->\mathop {\vbox {\m@th \ialign {##\crcr 
\noalign {\kern 3\p@ }\downbracefill \crcr \noalign {\kern 3\p@ \nointerlineskip }$\hfil \displaystyle {#1}\hfil $\crcr 
}}}\limits 

\underbrace:
macro:#1->\mathop {\vtop {\m@th \ialign {##\crcr 
$\hfil \displaystyle {#1}\hfil $\crcr \noalign {\kern 3\p@ \nointerlineskip }\upbracefill \crcr \noalign {\kern 3\p@ }%
}}}\limits 

\fi

\makeatletter
\def\overunderbrace#1#2#3{%
    \begingroup
    \let\overunderbrace@sup\empty
    \let\overunderbrace@sub\empty
    \@ifnextchar^%
        {\@overunderbracesup{#1}{#2}{#3}}%
        {\@ifnextchar_%
            {\@overunderbracesub{#1}{#2}{#3}}%
            {\@overunderbrace{#1}{#2}{#3}}%
        }%
}

\def\@overunderbracesup#1#2#3^#4{%
    \def\overunderbrace@sup{#4}%
    \@ifnextchar_%
        {\@overunderbracesub{#1}{#2}{#3}}%
        {\@overunderbrace{#1}{#2}{#3}}%
}

\def\@overunderbracesub#1#2#3_#4{%
    \def\overunderbrace@sub{#4}%
    \@ifnextchar^%
        {\@overunderbracesup{#1}{#2}{#3}}%
        {\@overunderbrace{#1}{#2}{#3}}%
}

\def\@overunderbrace#1#2#3{%
\mathop {\vcenter {\m@th \ialign {##&##&##\crcr
    \noalign {\kern 3\p@}%
    \span\omit\hfil\hbox{\vbox to 0pt{\vss\hbox{\vbox{\hbox{$\m@th\scriptstyle\overunderbrace@sup$}\vspace{0pt}}}}}\hfil
    &%
    \crcr \noalign {\kern 5\p@\nointerlineskip}%
    \span\omit\downbracefill&%
    \crcr \noalign {\kern 3\p@\nointerlineskip}%
        $\hfil \displaystyle {#1}\hfil $&%
        $\hfil \displaystyle {#2}\hfil $&%
        $\hfil \displaystyle {#3}\hfil $%
    \crcr \noalign {\kern 3\p@\nointerlineskip}%
      & \span\omit  \upbracefill
    \crcr \noalign {\kern 5\p@\nointerlineskip}%
    &\span\omit\hfil\hbox{\vbox to 0pt{\hbox{\vbox{\vspace{0pt}\hbox{$\m@th\scriptstyle\overunderbrace@sub$}}}\vss}}\hfil
    \crcr \noalign {\kern 3\p@ }%
}}}%
    \endgroup
}

\makeatother


\begin{document}

% The extra `\mathord{}` is used to get the spacing right, otherwise
% the +'s before f and h think they are signs not operators. 
% This is content specific and can't be added to the \overunderbracemacro.
\[
a+b+\overunderbrace{c+d+\mathord{}}{e+f+g}{\mathord{}+h+i}^{x}_{y} =e^2
\]

\end{document}

Result

Result