Why does nesting some xstring commands causes errors?

This is a rather common problem using xstring because (most of) its commands don't work in expansion-only contexts (Inside an \edef or \expanded or \write or. . .), so each command has to execute completely before the next one starts. In short, they can't be nested. A simpler example that would show the same problem would be to define:

\newcommand\ifequal[2]{%
  \def\tempa{#1}%
  \def\tempb{#2}
  \ifx\tempa\tempb
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi}

and then use the command inside an \edef:

\edef\test{\ifequal{abc}{abc}{true}{false}}

which would throw the error:

! Undefined control sequence.
\ifequal #1#2->\def \tempa 
                           {#1}\def \tempb {#2} \ifx \tempa \tempb \expandaf...
l.13 \edef\test{\ifequal{abc}{abc}
                                  {true}{false}}
?

however if you used it outside the \edef it would work.

To allow using the results of one xstring command in another, most of the commands have a final optional argument, which is a macro to store the result of the processing, then you need to do each step separately and collect the result at each step:

\documentclass{article}
\usepackage{xstring}
\begin{document}
\section{A}
\StrChar{bbbb}{4}[\tempA]
\StrChar{aaaa}{3}[\tempB]
\StrCompare{\tempA}{\tempB}
\end{document}

Hopefully soon I'll finish my expandable version of xstring ;-)

You can't use \StrChar in the argument of \StrCompare.

Here are two implementations: one with xparse and expl3, one with xstring. Note that the former is completely expandable, the latter isn't.

\documentclass{article}
\usepackage{xparse}
\usepackage{xstring} % for comparison

\ExplSyntaxOn

\NewExpandableDocumentCommand{\comparestringitems}{mmmmmm}
 {
  % #1 = first string
  % #2 = item number
  % #3 = second string
  % #4 = item number
  % #5 = true text
  % #6 = false text
  \str_if_eq:eeTF { \str_item:nn { #1 } { #2 } } { \str_item:nn { #3 } { #4 } } { #5 } { #6 }
 }

\prg_generate_conditional_variant:Nnn \str_if_eq:nn { ee } { TF,T,F,p }

\ExplSyntaxOff

\newcommand{\comparestringitemsxstring}[6]{%
  \StrChar{#1}{#2}[\compareA]%
  \StrChar{#3}{#4}[\compareB]%
  \StrCompare{\compareA}{\compareB}[\compareC]
  \ifnum\compareC=0 #5\else#6\fi
}

\begin{document}

\comparestringitems{aaaa}{4}{bbbb}{3}{true}{false}

\comparestringitems{aaaa}{4}{bbab}{3}{true}{false}

\comparestringitemsxstring{aaaa}{4}{bbbb}{3}{true}{false}

\comparestringitemsxstring{aaaa}{4}{bbab}{3}{true}{false}

\edef\test{\comparestringitems{aaaa}{4}{bbab}{3}{true}{false}}
\texttt{\meaning\test}

\end{document}

With a simple modification, you can also pass macros to \comparestringitems:

\documentclass{article}
\usepackage{xparse}

\ExplSyntaxOn

\NewExpandableDocumentCommand{\comparestringitems}{mmmmmm}
 {
  % #1 = first string
  % #2 = item number
  % #3 = second string
  % #4 = item number
  % #5 = true text
  % #6 = false text
  \str_if_eq:eeTF { \str_item:en { #1 } { #2 } } { \str_item:en { #3 } { #4 } } { #5 } { #6 }
 }

\prg_generate_conditional_variant:Nnn \str_if_eq:nn { ee } { TF,T,F,p }
\cs_generate_variant:Nn \str_item:nn { e }

\ExplSyntaxOff

\newcommand{\strA}{aaaa}
\newcommand{\strB}{bbbb}
\newcommand{\strC}{bbab}

\begin{document}

\comparestringitems{\strA}{4}{\strB}{3}{true}{false}

\comparestringitems{\strA}{4}{\strC}{3}{true}{false}

\edef\test{\comparestringitems{\strA}{4}{\strC}{3}{true}{false}}
\texttt{\meaning\test}

\end{document}