Defining a macro for "real" colored delimiters

The macros \left and \right put their content in its own group.

You will notice this in the following example if you leave out the \normalcolor in the \rightcolor macro:

\leftcolor{blue}( \frac{a}{b} \rightcolor{red})^n

The superscript n (and everything following it in the same group) turns blue!

The internal macro \set@color that is called by \color as well as \normalcolor contains a \aftergroup\reset@color which gets called after the group of \left and \right. (So the problem is the \normalcolor in \leftcolor and the \color in \rightcolor.) The following ^n doesn’t see the right math atom to get correctly aligned.

Adding another group around it reverses this affect and ^2 is aligned correctly (see the fractions 4 and 5). But the fact that \left and \right is an Inner Atom is destroyed hereby, too, as {} construct an Ordinary Atom.

TeXbook, page 155:

There's also an eighth classication, \mathinner, which is not normally used for individual symbols; fractions and \left...\right constructions are treated as “inner” subformulas, which means that they will be surrounded by additional space in certain circumstances.

The correct usage of re-colored delimiters would need instead of the second group a \mathinner to get the correct spacing.

I have implemented this in the \p macro that takes an additional optional argument if you want to use different colors for the delimiters.

The difference in spacing is shown with the a.

Although it has been said on the linked question, I want to repeat the fact that \normalcolor does not switch back to the current color before the \left but to the default color. A \color{green} a \p{red}(){b} c will still show a black b and a green c.

Code

\documentclass{article}
\usepackage{mathtools}
\usepackage{xcolor}
\makeatletter
\newcommand*{\p}[5][]{%
  \edef\@tempa{#1}%
  \ifx\@tempa\@empty
    \edef\@tempa{#2}%
  \fi
  \mathinner{\begingroup\color{\@tempa}\left#3\normalcolor#5\color{#2}\right#4\endgroup}
}
\makeatother
\newcommand*\leftcolor[2]{%
 \color{#1}\left#2\normalcolor%
}
\newcommand*\rightcolor[2]{%
 \color{#1}\right#2%
}

\begin{document}
\begin{gather*}
a \p{red}(){\frac{a}{b}}^n \\
a \p[blue]{red}(){\frac{a}{b}}^n \\ 
%
a \left(\frac{a}{b}\right)^n \\
%
%
a \mathinner{{\leftcolor{blue}( \frac{a}{b} \rightcolor{red})}}^n \\
a \leftcolor{blue}( \frac{a}{b} \rightcolor{red})^n \\ % the 'n' is blue not red!
a {{\leftcolor{blue}( \frac{a}{b} \rightcolor{red})}}^n \\ % the spacing is bad
%
a \left(\frac{a}{b}\right)^n
\end{gather*}
\end{document}

Output

For comparison, ConTeXt provides a \definemathfence command for defining colored delimiters. math-fen.mkiv predefines a few math fences:

\definemathfence [parenthesis] [\c!left=0x28,\c!right=0x29]
\definemathfence [bracket]     [\c!left=0x5B,\c!right=0x5D]
\definemathfence [braces]      [\c!left=0x7B,\c!right=0x7D]
\definemathfence [bar]         [\c!left=0x7C,\c!right=0x7C]
\definemathfence [doublebar]   [\c!left=0x2016,\c!right=0x2016]
\definemathfence [angle]       [\c!left=0x3C,\c!right=0x3E]

which you can use using \fenced[parenthesis]{ \frac {a}{b} }. To add color, overload one of the predefined fences (or just start from scratch), and add a color attribute as well. For example:

\definemathfence[colorbracket][parenthesis][color=red, command=yes]

The command=yes part defines a command \colorbracket that can be used in place of \fenced[colorbracket]. Complete example:

\definemathfence[colorbracket][parenthesis][color=red, command=yes]

\starttext

\startformula
  \colorbracket{\frac {a}{b}}^n
  \qquad
  \left( \frac {a}{b} \right)^n
\stopformula

\stoptext

which gives

Like all ConTeXt \define.... commands, you can use \setup... command to change the behavior. For example, to change the color from red to blue, you can use:

\setupmathfence[colorbracket][color=blue]

This command does not support different colors for left and right delimiters. For implementation, see math-fen.mkiv.

After considering the useful comments above (I never imagined that people like Heiko Oberdiek would spend their time on my only second question!), I added the implementation of the case where the argument of color is expanded from \@empty to the standard black.

The final MWE is as follows:

\documentclass{article}
\usepackage{amsmath}
\usepackage{mathtools}
\usepackage{xcolor}
%
\makeatletter
%
\newcommand\leftcolor[2]{%
  \edef\@tempa{#1}%
  \ifx\@tempa\@empty
    \edef\@tempa{black}%
  \fi
 \mathinner\bgroup\begingroup\color{\@tempa}\left#2\normalcolor%
}
\newcommand\rightcolor[2]{%
  \edef\@tempa{#1}%
  \ifx\@tempa\@empty
    \edef\@tempa{black}%
  \fi
  \color{\@tempa}\right#2\endgroup\egroup%
}
\makeatother
%
\begin{document}
%
\begin{gather*}
a\leftcolor{red}(\frac{a}{b} \rightcolor{red})^n \left(\frac{a}{a}\right)^n \\
a\leftcolor{}(\frac{a}{b}\rightcolor{})^n \left(\frac{a}{a}\right)^n \\
a\leftcolor{red}(\frac{a}{b}\rightcolor{blue})^n \left(\frac{a}{a}\right)^n \\
\end{gather*}
%
\end{document}

And the final output is now shown correct:

I also noticed that the function of \mathinner\begingroup\bgroup and \endgroup\egroup in each definition makes the code working as \mathinner{{<argument>}}^n, but inside my package LyXbasic I'll keep the command that Qrrbrbirlbel suggested, because the syntax is equally fluid with:

\p{<left delim color>}{<right delim color>}{<left delim>}{<right delim>}{<argument>}