Smart functions

\documentclass{article}
\newcounter{emphlevel}
\renewcommand\emph[1]{\stepcounter{emphlevel}%
  \ifnum\value{emphlevel}=1`#1'\else
    \ifnum\value{emphlevel}=2\textsc{#1}\else
    \ifnum\value{emphlevel}=3\textit{#1}\else
  \fi\fi\fi\addtocounter{emphlevel}{-1}%
}
\begin{document}
\emph{Single quotes within \emph{Small Caps within \emph{more italics} and} so on}.
\end{document}

The above ORIGINAL answer truncates levels beyond the third. This below variation repeats the cycles repeatedly through the specified emphases.

\documentclass{article}
\newcounter{emphlevel}
\renewcommand\emph[1]{\stepcounter{emphlevel}%
  \ifnum\value{emphlevel}=1\textup{`#1'}\else
    \ifnum\value{emphlevel}=2\textsc{#1}\else
    \ifnum\value{emphlevel}=3\textit{#1}\else
      \addtocounter{emphlevel}{-4}\emph{#1}\addtocounter{emphlevel}{4}%
  \fi\fi\fi\addtocounter{emphlevel}{-1}%
}
\begin{document}
\emph{Single quotes within \emph{Small Caps within \emph{more italics
  within \emph{Single quotes within \emph{Small Caps within \emph{more italics} 
  and} so on}} and} so on}.

\emph{Aa \emph{Bb \emph{Cc \emph{Dd \emph{Ee \emph{Ff \emph{Gg}}}}}}}
\end{document}

A simple application of grouping:

\documentclass{article}

\newcount\emphlevel
\DeclareRobustCommand{\emph}[1]{%
  \begingroup
  \normalfont
  \advance\emphlevel by 1
  \ifcase\emphlevel
  \or
    `\aftergroup'\normalfont
  \or
    \normalfont\expandafter\textsc
  \or
    \normalfont\expandafter\textit
  \else
    \normalfont!!!!\expandafter\textbf
  \fi
  {#1}%
  \endgroup
}

\begin{document}

Normal text.
\emph{Single quotes within \emph{Small Caps within \emph{more italics} and} so on}.
Again normal text.

\emph{Single quotes within \emph{Small Caps within \emph{more italics}}}.

\end{document}

I decided to use another command name, but the nesting can be controlled with a counter and \ifcase...\fi conditional.

The 4th level will provide an usual \@ctrerr, but this could be shifted to basically any level.

\documentclass{article}


\newcounter{smartlevel}

\makeatletter
\newcommand{\smartcmd}[1]{%
  \stepcounter{smartlevel}%
  \ifcase\c@smartlevel
  \or'#1'\addtocounter{smartlevel}{-1}%
  \or\textsc{#1}\addtocounter{smartlevel}{-1}%
  \or\textit{#1}\addtocounter{smartlevel}{-1}%
  \else
  \@ctrerr% Too deeply nested
  \fi
}
\makeatother

\begin{document}

\smartcmd{First \smartcmd{Foo \smartcmd{inner} \smartcmd{inner again}}}

\smartcmd{First \smartcmd{Foo} \smartcmd{Foo level \smartcmd{inner again}}} 

\end{document}