How to use itemize in Table environment

For simplicity, I don't think you need to itemize the list. The MWE below uses \tabitem to set a fake item, letting you create the table in the regular way (also making the code more readable):

\documentclass{article}
\usepackage{booktabs}% http://ctan.org/pkg/booktabs
\newcommand{\tabitem}{~~\llap{\textbullet}~~}
\begin{document}

\begin{table}
  \centering
  \begin{tabular}{lll}
    \toprule
    \multicolumn{3}{c}{Classification of the criticel point $(0,0)$ of $x' = Ax, |\mathbf{A}| \not= 0$.} \\[.5\normalbaselineskip]
    Types & Type of Critical Point & Stability \\
    \midrule
    1.~Real unequal eigenvalues of same sign \\
    \tabitem $\lambda_1 > \lambda_2 > 0$ & Improper node/node & Unstable \\
    \tabitem $\lambda_1 < \lambda_2 < 0$ & Improper node/node & Asym.~stable \\[.5\normalbaselineskip]
    2.~Real unequal eigenvalues of opposite sign \\
    \tabitem $\lambda_2 < 0 > \lambda_1$ & Saddle point & Unstable \\[.5\normalbaselineskip]
    3.~Equal eigenvalues \\ Subtype 1: Two Independent vectors \\
    \tabitem $\lambda_1 = \lambda_2 > 0$ & Proper node & Unstable \\
    \tabitem $\lambda_1 = \lambda_2 < 0$ & Proper node & Asym.~stable \\
    \bottomrule
  \end{tabular}
\end{table}

\end{document}

booktabs provides the supplemental tabular eye-candy.

When I had the same problem I solved it with tabularx. Here's the code for your example (I'm assuming the number in the 4th row should be 2):

\documentclass{article}
\usepackage[textwidth = 155mm]{geometry}
\usepackage{tabularx}
\begin{document}

\begin{table}
\begin{tabularx}{\textwidth}{|>{\setlength\hsize{1.4\hsize}\setlength\linewidth{\hsize}}X|>{\setlength\hsize{.9\hsize}\setlength\linewidth{\hsize}}X|>{\setlength\hsize{.7\hsize}\setlength\linewidth{\hsize}}X|}
\hline
\multicolumn{3}{|c|}{Classification of the criticel point $(0,0)$ of $x'=Ax,|\mathbf{A}|\not=0$.}\\
\hline
Types & Type of Critical Point & Stability \\
\hline
1. Real unequal eigenvalues of same sign
\begin{itemize}
\item $\lambda_1 > \lambda_2 > 0$
\item $\lambda_1 < \lambda_2 < 0$
\end{itemize} &
\vphantom{1. Real unequal eigenvalues of same sign}
\begin{itemize}
\item Improper Node/Node
\item Improper Node/Node
\end{itemize} &
\vphantom{1. Real unequal eigenvalues of same sign}
\begin{itemize}
\item Unstable
\item Asym. Stable
\end{itemize}\\
\hline
2. Real unequal eigenvalues of opposite sign
\begin{itemize}
\item $\lambda_2 < 0 >\lambda_1$
\end{itemize} &
\vphantom{2. Real unequal eigenvalues of opposite sign}
\begin{itemize}
\item Saddle Point
\end{itemize} &
\vphantom{2. Real unequal eigenvalues of opposite sign}
\begin{itemize}
\item Unstable
\end{itemize}\\
\hline
3. Equal eigenvalues \newline Subtype 1: Two Independent vectors
\begin{itemize}
\item $\lambda_1 = \lambda_2 > 0$
\item $\lambda_1 = \lambda_2 < 0$
\end{itemize} &
\vphantom{3. Equal eigenvalues} \vphantom{ Subtype 1: Two Independent vectors}
\begin{itemize}
\item Proper Node
\item Proper Node
\end{itemize} &
\vphantom{3. Equal eigenvalues} \vphantom{ Subtype 1: Two Independent vectors}
\begin{itemize}
\item Unstable
\item Asym. Stable
\end{itemize}\\
\hline
\end{tabularx}
\end{table}

\end{document}

If you don't like the item symbols in the 2nd and 3rd columns, use \item[].

Using itemization with minipage might be an option if the newline (backslash) is overlapping to another cell. For example:

     \multicolumn{2}{c}{\textbf{Classification 1}} \\ [2pt]  \hline
      \textit{Classification 1} & 
      \begin{minipage} [t] {0.4\textwidth} 
      \begin{itemize}
      \item item 1
        \subitem subitem 2
        \subitem subitem 3
     \end{itemize} 
    \end{minipage} \\ [2pt]  \hline