How to typeset this in LaTeX

Example:

\documentclass{article}
\usepackage{amsmath}
\usepackage{graphicx}
\usepackage{mleftright}

\begin{document}
\[
  \Phi =
  \begin{tabular}{@{}c@{}}
    \rotatebox[origin=c]{90}{$\scriptstyle\text{states}$}
  \end{tabular}
  \mleft\{
    \vphantom{%
      \begin{bmatrix}
        \vdots \\
        \phi_{iy} \\
        \vdots
      \end{bmatrix}
    }%
  \mright.\kern-\nulldelimiterspace
  j % \,
  \overbrace{%
    \mathop{%
      \!
      \begin{bmatrix}
        \vdots \\
        \cdots \phi_{iy} \hphantom{\cdots} \\
        \vphantom{\vdots}
      \end{bmatrix}
      \!
    }\limits^{\smash{\textstyle y}}
  }^{\text{aggr. states}}
\]
\end{document}

Remarks:

• The matrix is set using environment bmatrix of package amsmath that automatically sets the brackets.
• \text of package amsmath (or amstext) sets text in text mode (automatically resized).
• \mleft and \mright of package mleftright avoid the additional surrounding space of \left and \right.
• \vphantom creates an emtpy box with width zero, but the height and depth of its argument. Thus the left brace only covers the matrix, not the stuff above.
• \right. is an invisible right delimiter. TeX inserts the space \nulldelimiterspace, thus \kern-\nulldelimiterspace removes the space.
• \begin{tabular}{@{}c@{}}...\end{tabular} is a trick to center the box around the math axis. (Internally \vcenter is used.)
• The side bearings of the big left and right brackets are quite large. Thus the horizontal brace is a little wider than necessary (thanks Barbara Beeton for noticing). This is fixed by a negative space \! before and after the matrix. A \, could be inserted after j to keep the space to the matrix constant. But I have commented it, because a smaller distance looks better IMHO.

\documentclass{article}
\usepackage{amsmath}
\usepackage{graphicx} %for the \rotatebox command
\begin{document}
\[
\setbox0=\hbox{
    $\begin{array}{c}
        y\\
        j\begin{bmatrix}
            & \vdots &  \\
            \cdots & \Phi_{jy} & \\
            & &  \\
        \end{bmatrix}
    \end{array}$
}
\Phi = \text{\rotatebox[origin=c]{90}{states}}\left\{\vphantom{\usebox0}\right.\kern-\nulldelimiterspace\overbrace{\usebox0}^\text{aggr. states}
\]
\end{document}