# Matrix in TikZ and Equation

You're nesting tikzpictures, that is often a bad idea. Here's an alternative approach using the fit library.

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing, calc, positioning,fit}
\begin{document}
$$Q= \begin{tikzpicture}[ baseline, label distance=10pt % added ] \matrix [matrix of math nodes,left delimiter=(,right delimiter=),row sep=0.1cm,column sep=0.1cm] (m) { 1 & * & * & \dots & * & 0 & \dots & 0 \\ 0 & \pm 1 & * & \dots & * & 0 & \dots & 0 \\ 0 & 0 &\pm 1 & \dots & * & 0 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & \dots &\pm 1& 0 & \dots & 0 \\ 0 & 0 & 0 & \dots & 0 & * & \dots & * \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & 0 & * & \dots & *\\ }; \draw[dashed] (0.5*(m-1-5.north east)+0.5*(m-1-6.north west)) -- (0.5*(m-8-6.south east)+0.5*(m-8-5.south west)); \draw[dashed] (0.5*(m-5-1.south west)+0.5*(m-6-1.north west)) -- (0.5*(m-5-8.south east)+0.5*(m-6-8.north east)); \node[ fit=(m-1-1)(m-1-5), inner xsep=0, above delimiter=\{, label=above:k_1 ] {}; \node[ fit=(m-1-6)(m-1-8), inner xsep=0, above delimiter=\{, label=above:k-k_1 ] {}; \node[ fit=(m-1-8)(m-5-8), inner xsep=15pt,inner ysep=0, right delimiter=\}, label=right:k_1 ] {}; \node[ fit=(m-6-8)(m-8-8), inner xsep=15pt,inner ysep=0, right delimiter=\}, label=right:k-k_1 ] {}; \end{tikzpicture}$$

\end{document}

Since you are loading already the decorations.pathreplacing library and make all the preparations, I could not resist using them. ;-)

\documentclass{article}
\usepackage{tikz}
\usetikzlibrary{matrix,decorations.pathreplacing, calc, positioning}
\begin{document}
$$Q= \begin{tikzpicture}[baseline] \matrix [matrix of math nodes,left delimiter=(,right delimiter=),row sep=0.1cm,column sep=0.1cm] (m) { 1 & * & * & \dots & * & 0 & \dots & 0 \\ 0 & \pm 1 & * & \dots & * & 0 & \dots & 0 \\ 0 & 0 &\pm 1 & \dots & * & 0 & \dots & 0 \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & \dots &\pm 1& 0 & \dots & 0 \\ 0 & 0 & 0 & \dots & 0 & * & \dots & * \\ \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\ 0 & 0 & 0 & 0 & 0 & * & \dots & *\\ }; \draw[dashed] (0.5*(m-1-5.north east)+0.5*(m-1-6.north west)) -- (0.5*(m-8-6.south east)+0.5*(m-8-5.south west)); \draw[dashed] (0.5*(m-5-1.south west)+0.5*(m-6-1.north west)) -- (0.5*(m-5-8.south east)+0.5*(m-6-8.north east)); \node[above=1pt of m-1-1] (top-1) {}; \node[above=1pt of m-1-5] (top-5) {}; \node[above=-1pt of m-1-6] (top-6) {}; \node[above=-1pt of m-1-8] (top-8) {}; \node[right=10pt of m-1-8] (right-1) {}; \node[right=10pt of m-5-8] (right-5) {}; \node[right=10pt of m-6-8] (right-6) {}; \node[right=10pt of m-8-8] (right-8) {}; \draw[thick,decorate,decoration=brace] (top-1.west) -- (top-5.east) node[midway,above=1mm] {k_1}; \draw[thick,decorate,decoration=brace] (top-6.west) -- (top-8.east) node[midway,above=1mm] {k-k_1}; \draw[thick,decorate,decoration=brace] (right-1.north) -- (right-5.south) node[midway,right=1mm] {k_1}; \draw[thick,decorate,decoration=brace] (right-6.north) -- (right-8.south) node[midway,right=1mm] {k-k_1}; \end{tikzpicture}$$

\end{document}