nesting macro identifying string length 1 using xstring ifthen packages

I think that if you turn off the expansion of the argument of \StrLen with \noexpandarg it passes your tests:

\documentclass{article}
\usepackage{ifthen,xstring}

\newcommand{\p}[1]{%
  \begingroup
    \noexpandarg
    \StrLen{#1}[\Len]%
    \ifthenelse{\Len=1}{#1}{(#1)}%
  \endgroup}

\begin{document}

\(\p{aa} = (aa)\)

\(\p{a} = a\)

\(\p{aa\p{b}}=(aab)\)

\(\p{\p{a}}=(a)\)

\end{document}

I nested the \noexpandarg in a group as egreg suggested. I opted to add it inside the definition because this expansion behavior is not always wanted, but will be the case when \p is called.

No need for packages.

\documentclass{article}
\newcommand{\p}[1]{\paux#1\endp}
\def\paux#1#2\endp{\ifx\relax#2\relax#1\else(#1#2)\fi}
\begin{document}
\(\p{aa} = (aa)\)

\(\p{a} = a\)

\(\p{aa\p{b}}=(aab)\)

\p{aa\p{bb}} = (aa(bb)) 

\(\p{\p{a}}=(a)\)

\end{document}

The mandatory expl3 version:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewExpandableDocumentCommand{\p}{m}
 {
  \rev_p:n { #1 }
 }

\cs_new:Nn \rev_p:n
 {
  \int_compare:nTF { \tl_count:n { #1 } > 1 } { (#1) } { #1 }
 }

\ExplSyntaxOff

\begin{document}

\begin{tabular}{ll}
Input       & Should be \\
\hline
\p{aa}      & (aa) \\
\p{a}       & a \\
\p{aa\p{b}} & (aab) \\
\p{aa\p{bb}} & (aa(bb)) \\
\p{\p{a}}   & (a) \\
\end{tabular}

\end{document}