Putting a tabular in a node within a matrix

The problem is that TikZ needs to make & equivalent to \pgfmatrixnextcell, which interferes with the normal functioning of & as a table cell separation marker. Fortunately, TikZ includes the ampersand replacement option that lets you choose any other macro for matrix cell separation instead of &. The following example uses \& as replacement:

\documentclass{article}
\usepackage{tikz}

\begin{document}

\begin{tikzpicture}
    \matrix[ampersand replacement=\&] {
        \node (species1) [shape=rectangle,draw] {
            \begin{tabular}{c c c}
                \multicolumn{3}{c}{{Species 1}} \\
                \colorbox{red}{G1a} & \colorbox{blue}{G2a} & \colorbox{green}{G3} \\
                \colorbox{blue}{G1b} & \colorbox{red}{G2b} & 
            \end{tabular}
        };
        \& 
        \node {b}; \\
        \node {c}; \& \node {d}; \\
};
\end{tikzpicture}

\end{document}

You can put each node inside a save box, and then it will work:

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix}

\newsavebox{\speciesone}
\sbox{\speciesone}{
\begin{tabular}{c c c}
  \multicolumn{3}{c}{{Species 1}} \tabularnewline
  \colorbox{red}{G1a} & \colorbox{blue}{G2a} &
  \colorbox{green}{G3} \tabularnewline
  \colorbox{blue}{G1b} & \colorbox{red}{G2b} & 
  \end{tabular}
  }

\begin{document}
\begin{tikzpicture}
\matrix [matrix of nodes] {
\node (species1) [shape=rectangle,draw] {\usebox{\speciesone}};\\
};
\end{tikzpicture}


\end{document}