Smaller font for subscript within subscript in math mode

You can use \DeclareMathSizes; declare also other sizes if needed.

\RequirePackage{fix-cm} % arbitrary font scaling
\documentclass{article}

\DeclareMathSizes{10}{10}{7}{4}

\begin{document}
$a_{b_{c}}$
\end{document}

Compare to the standard and decide for yourself if it's worth the trouble. I wouldn't go below four points.

For even smaller, you could scale the sub-subscript:

\documentclass{article}
\usepackage{scalerel}
\begin{document}
$a_{b_{\scaleto{c}{1pt}}}$
\end{document}

If you need consistency across a range of subscripts, you may wish to add a normalizer like a \mathstrut:

\documentclass{article}
\usepackage{scalerel}
\begin{document}
$a_{b_{\scaleto{c\mathstrut}{2pt}}}$
$a_{b_{\scaleto{g\mathstrut}{2pt}}}$
\end{document}

Here, \scaleto from the scalerel package operates on its arguments in math mode by default, unlike \scalebox of the graphicx package, which operates in text mode.

Using someone's idea in How to get an even smaller font? to scale \scriptscriptstyle content down by 30%:

\documentclass{article}

\usepackage{graphicx}

\begin{document}

$a_{b_c}$

$a_{b_{\scalebox{.7}{$\scriptscriptstyle c$}}}$

\end{document}