In TikZ, is it possible to nest nodes?

You can abuse matrix nodes. Normal nodes do not let you put TikZ commands inside them (as they revert to text mode), but matrix nodes do. So you could just create a 1x1-matrix for every part. For example:

\begin{tikzpicture}
    \node[matrix] (A) {
        \draw (0,0) rectangle (1,1); 
        \node at (0.5,0.5) {A}; \\
    };
    \node[matrix,left of=A] (B) 
    {
        \draw (0.5,0.5) circle (0.5);
        \fill (0.5,0.5) circle (0.1); \\
    };
\end{tikzpicture}

I did it by including a whole \begin{tikzpicture} ... \end{tikzpicture} in the content of each node.

Maybe not the most elegant way of doing it... but it works.

Just some initial ideas:

Depending how complicated the "inner" part needs to be, one way to do this would be to make the molecules a "matrix" node (see the pgfmanual for details on how to use it). Another possibility to look in to is the "chain" library.

Another option would be to draw an invisible node at the centre of the molecule of the requisite size (draw it visibly first to get it the right size). The disadvantage of that is that you have to work out that size whereas it would be nice to get TikZ to do it by hand.