Rotated column titles in tabular

You can use the adjustbox package to rotate the content and limit it official width. I would also recommend to use a separate \newcolumntype for it as well as a \rot macro which inserts the required \multicolumn command. You won't avoid that one. However, such a macro must be fully expandable and start directly with \multicolumn. If you want multi line headings add the minipage=<width> option to \adjustbox.

\documentclass{article}

\usepackage{adjustbox}
\usepackage{array}

\newcolumntype{R}[2]{%
    >{\adjustbox{angle=#1,lap=\width-(#2)}\bgroup}%
    l%
    <{\egroup}%
}
\newcommand*\rot{\multicolumn{1}{R{45}{1em}}}% no optional argument here, please!

\begin{document}

\begin{tabular}{r|ccc}
&
\rot{Property 1} &
\rot{Property 2} &
\rot{Property 3}
    \\ \hline
System 1        &       &       &  X    \\ 
System 2        & X     & X     &  X    \\
System 3        & X &   &  X    \\ \hline
\end{tabular}


\renewcommand*\rot{\multicolumn{1}{R{60}{1em}}}% no optional argument here, please!

\begin{tabular}{r|ccc}
&
\rot{Property 1} &
\rot{Property 2} &
\rot{Property 3}
    \\ \hline
System 1        &       &       &  X    \\ 
System 2        & X     & X     &  X    \\
System 3        & X &   &  X    \\ \hline
\end{tabular}


\renewcommand*\rot[2]{\multicolumn{1}{R{#1}{#2}}}% no optional argument here, please!

\begin{tabular}{r|ccc}
&
\rot{90}{1em}{Property 1} &
\rot{60}{1em}{Property 2} &
\rot{45}{2em}{Property 3}
    \\ \hline
System 1        &       &       &  X    \\ 
System 2        & X     & X     &  X    \\
System 3        & X &   &  X    \\ \hline
\end{tabular}

\end{document}

Wrapping the rotated contents in a box of fixed width you can easily manipulate the intended width of each column header. All of this, including the rotation, can be wrapped in a macro so that your code reads a little better. Moreover, some of the rotation and width specifications can be given default values so you only have to worry about them in special instances.

In the minimal example below, the command \rot[<angle>][<width>]{<stuff>} rotates <stuff> at angle <angle> (defaults to 45 degrees) and fixes it to have a width of <width> (default is 1em). The first two arguments are optional, while the last is mandatory.

\documentclass{article}
\usepackage{graphicx} % http://ctan.org/pkg/graphicx
\usepackage{booktabs} % http://ctan.org/pkg/booktabs
\usepackage{xparse}   % http://ctan.org/pkg/xparse
% Rotation: \rot[<angle>][<width>]{<stuff>}
\NewDocumentCommand{\rot}{O{45} O{1em} m}{\makebox[#2][l]{\rotatebox{#1}{#3}}}%
\begin{document}

% ===== DEFAULT ROTATION PARAMETERS
\begin{tabular}{lccc}
    & \rot{Property 1}
    & \rot{Property 2}
    & \rot{Property 3} \\
    \midrule
    System 1 &   &   & X \\
    System 2 & X & X & X \\
    System 3 & X &   & X \\
    \bottomrule
\end{tabular}

\bigskip

% ===== MODIFIED ANGLE ROTATION
\begin{tabular}{lccc}
    & \rot[60]{Property 1}
    & \rot[60]{Property 2}
    & \rot[60]{Property 3} \\
    \midrule
    System 1 &   &   & X \\
    System 2 & X & X & X \\
    System 3 & X &   & X \\
    \bottomrule
\end{tabular}

\bigskip

% ===== MODIFIED ANGLE & WIDTH ROTATION
\begin{tabular}{lccc}
    & \rot[25][3em]{Property 1}
    & \rot[25][3em]{Property 2}
    & \rot[25][3em]{Property 3} \\
    \midrule
    System 1 &   &   & X \\
    System 2 & X & X & X \\
    System 3 & X &   & X \\
    \bottomrule
\end{tabular}

\end{document}

To me, this allows for less manual interaction, since some of the parameters are set by default.

I would define a new column type using a rotated minipage; in the following example, the first mandatory argument of this new column type controls the angle of rotation, and the second argument specifies the minipage's width:

\documentclass{article}
\usepackage{array}
\usepackage{booktabs}
\usepackage{rotating}

\newcolumntype{P}[2]{%
  >{\begin{turn}{#1}\begin{minipage}{#2}\small\raggedright\hspace{0pt}}l%
  <{\end{minipage}\end{turn}}%
}

\begin{document}

\begin{table}
  \centering
  \caption{Some properties}
  \label{tab:test1}
  \begin{tabular}{@{}rccc@{}}
  \toprule
  & \multicolumn{1}{P{90}{1.6cm}}{Property 1} &
    \multicolumn{1}{P{90}{1.6cm}}{Property 2} &
    \multicolumn{1}{P{90}{1.6cm}@{}}{Property 3 more text} \\
      \midrule
  System 1 &     &     & X \\
  System 2 & X & X & X \\
  System 3 & X &    & X \\
  \bottomrule
  \end{tabular}
\end{table}

\begin{table}
  \centering
  \caption{Some properties}
  \label{tab:test2}
  \begin{tabular}{@{}rccc@{}}
  \toprule
  & \multicolumn{1}{P{60}{1.6cm}}{Property 1} &
    \multicolumn{1}{P{60}{1.6cm}}{Property 2} &
    \multicolumn{1}{P{60}{1.6cm}@{}}{Property 3 more text} \\
  \midrule
  System 1 &     &     & X \\
  System 2 & X & X & X \\
  System 3 & X &    & X \\
  \bottomrule
  \end{tabular}
\end{table}

\end{document}​