Tree in tikz, with triangles standing for sub-trees

Here's another option using variations of the isosceles triangle shape:

\documentclass[a4paper]{article}
\usepackage{tikz}
\usetikzlibrary{shapes.geometric}

\begin{document}

\tikzset{
itria/.style={
  draw,dashed,shape border uses incircle,
  isosceles triangle,shape border rotate=90,yshift=-1.45cm},
rtria/.style={
  draw,dashed,shape border uses incircle,
  isosceles triangle,isosceles triangle apex angle=90,
  shape border rotate=-45,yshift=0.2cm,xshift=0.5cm},
ritria/.style={
  draw,dashed,shape border uses incircle,
  isosceles triangle,isosceles triangle apex angle=110,
  shape border rotate=-55,yshift=0.1cm},
letria/.style={
  draw,dashed,shape border uses incircle,
  isosceles triangle,isosceles triangle apex angle=110,
  shape border rotate=235,yshift=0.1cm}
}
\begin{tikzpicture}[sibling distance=5cm, level 2/.style={sibling distance =2cm}]
\node[circle,draw,fill=black] {}
    child{ node[circle, fill = black] {}
            child{ node[circle, draw] {}
                        { node[itria] {Q1} } 
                }
            child{ node[rectangle, fill = black] {}
                    child {
                        node[rectangle, fill = black] {}
                            child{ node[rtria] {Q2} }
                            child { node[circle,draw] {} 
                                     { node[itria] {Q3} }                         
                                }
                        }
                }
        }
    child{ node[circle, draw] {}
            child{ node[ritria] {Q4} }
            child{node[letria] {Q5} }       
        };
\node[draw] at (3,-5) 
{
\begin{tabular}{cl}
\tikz\node[circle,fill] {}; & branching \\
\tikz\node[circle,draw] {}; & jump \\
\tikz\node[rectangle,fill] {}; & unary
\end{tabular}
};
\end{tikzpicture}
\end{document}

Ok, when you can find a way how to commit the node names to the new defined shapes, we are close to what you want.

\documentclass[a4paper]{article}

\usepackage{tikz}
\usetikzlibrary{shapes.geometric}

\pgfdeclareshape{left triangle}{
    \nodeparts{}
    \anchor{center}{\pgfpoint{0cm}{0cm}}
    \behindbackgroundpath{
    \path [draw,dashed] (0,0) -- (0,-2) -- (-1,-2) -- cycle;
   }
}
\pgfdeclareshape{right triangle}{
    \nodeparts{}
    \anchor{center}{\pgfpoint{0cm}{0cm}}
    \behindbackgroundpath{
    \path [draw,dashed] (0,0) -- (0,-2) -- (1,-2) -- cycle;
   }
}


\begin{document}
\begin{tikzpicture}[
sibling distance=5cm, 
level 2/.style={sibling distance =2cm},
triangle/.style={isosceles triangle,draw,shape border rotate=90, dashed, minimum height=10mm, minimum width=15mm, inner sep=0},
]
\node[circle,draw,fill=black] {}
    child{ node[circle, fill = black] {}
            child{ node[circle, draw] {}
                        child{ node[triangle] {Q1} } 
                }
            child{ node[rectangle, fill = black] {}
                    child {
                        node[rectangle, fill = black] {}
                            child{ node[left triangle] {Q2} }
                            child { node[circle,draw] {} 
                                   child{node[triangle] {Q3}}
                                }
                        }
                }
        }
    child{ node[circle, draw] {}
            child{node[left triangle] {Q4} }
            child{node[right triangle] {Q5} }       
        };
\end{tikzpicture}
\end{document}