Stacked math with lefthand braces around selected lines

I'm not quite sure of what the intended output should be, but in this case I'd just go with an array, and maybe also the cases environment from amsmath.

\documentclass{article}

\usepackage{amsmath}

\usepackage{xcolor}
\renewcommand{\vec}[1]{\mathbf{#1}}

\begin{document}
\[
\mathcal{Z} =
\begin{array}[t]{l}
\multicolumn{1}{c}{foo}\\
\begin{cases}
\vec{a}-\vec{b},\vec{a}+\vec{b}\\
\vec{a}+2\vec{b},-\vec{a}\\
\textcolor{red}{2\vec{a}+\vec{b},\vec{b}}
\end{cases}
\\
\begin{cases}
\vec{a}-\vec{b},\vec{a}+\vec{b}\\
\vec{a}+2\vec{b},-\vec{a}\\
\textcolor{red}{2\vec{a}+\vec{b},\vec{b}}
\end{cases}
\end{array}
%
= % second version
%
\begin{array}[t]{l}
\multicolumn{1}{c}{foo}\\
\left\{\begin{array}{@{}c@{}}
\vec{a}-\vec{b},\vec{a}+\vec{b}\\
\vec{a}+2\vec{b},-\vec{a}\\
\textcolor{red}{2\vec{a}+\vec{b},\vec{b}}
\end{array}\right.
\\[4ex]
\left\{\begin{array}{@{}c@{}}
\vec{a}-\vec{b},\vec{a}+\vec{b}\\
\vec{a}+2\vec{b},-\vec{a}\\
\textcolor{red}{2\vec{a}+\vec{b},\vec{b}}
\end{array}\right.
\end{array}
\]

\end{document}

I've written "foo" in the first line because I didn't understand whether c2mm was a typo or not.

You can use \Shortunderstack with two cases environments or \stackunder with two blocks in a blockarray environment:

\documentclass{article}
\usepackage{mathtools}
\usepackage{blkarray}
\usepackage{stackengine}
\usepackage{xcolor}
\renewcommand{\vec}[1]{\mathbf{#1}}

\begin{document}

\stackMath
\begin{equation*}
    \mathcal{G} = \stackunder{p6mm}{\vec{a},\vec{b}} \overset{t[\mathit{3}]}{>}
    \mathcal{Z} = \Shortunderstack{{c2mm} {\begin{dcases}\begin{matrix}\vec{a}-\vec{b},\vec{a}+\vec{b}\\ \vec{a}+2\vec{b},-\vec{a}\\ \textcolor{red}{2\vec{a}+\vec{b},\vec{b}}\end{matrix}\end{dcases}} { \begin{dcases}\vec{a}-\vec{b},\vec{a}+\vec{b}\\ \vec{a}+2\vec{b},-\vec{a}\\ \textcolor{red}{2\vec{a}+\vec{b},\vec{b}}\end{dcases}}}
\end{equation*}

\begin{equation*}
     \mathcal{G} = \stackunder{p6mm}{\vec{a},\vec{b}} \overset{t[\mathit{3}]}{>}
     \mathcal{Z} = \stackunder{c2mm}{\begin{blockarray}{c}\begin{block}{\{c}\vec{a}-\vec{b},\vec{a}+\vec{b}\\ \vec{a}+2\vec{b},-\vec{a}\\ \textcolor{red}{2\vec{a}+\vec{b},\vec{b}}\\ \end{block}\begin{block}{\{r}\vec{a}-\vec{b},\vec{a}+\vec{b}\\ \vec{a}+2\vec{b},-\vec{a}\\ \textcolor{red}{2\vec{a}+\vec{b},\vec{b}}\\ \end{block}\end{blockarray}}
\end{equation*}

\end{document} 

Nesting of \stackunder{anchor}{\stackunder{b1}{\stackunder{b2}{...}}}

I have figured out another solution using stackengine by nesting \stackunder{} and using \left\{\Vectorstack{item1 item2 item3}\right. for the blocks.

\documentclass[varwidth]{standalone}
\usepackage{amsmath}
\usepackage{stackengine}

\begin{document}
\stackMath
\begin{equation*}
      \mathcal{Z} = \stackunder
      {c2mm}
      {\stackunder
         {\left\{\Vectorstack{block1 block1 block1}\right.}
         {\stackunder
           {\left\{\Vectorstack{block2 block2 block2}\right.}
           {\left\{\Vectorstack{block3 block3 block3}\right.}
         }
      }
\end{equation*}
\end{document}