Reduce size of sum sub/superscript?

You can use the \smashoperator command from mathtools , and also insist for using \scriptscriptstyle instead of scriptstyle, but there might be a legibility problem:

\documentclass{article}
\usepackage{mathtools} 

\begin{document}

$\displaystyle\smashoperator{\sum_{k=1+\min(a-1,b-1)}^{\max(a-1,b-1)}}i_k$ \bigskip

$\displaystyle\smashoperator{\sum_{\scriptscriptstyle k=1+\min(a-1,b-1)}^{\scriptscriptstyle\max(a-1,b-1)}}i_k$

\end{document} 

To reduce the amount of visual overload inflicted on your readers, I would proceed along the following lines:

\documentclass{article}
\usepackage{amsmath}
\begin{document}
Put $\mu(a,b)=1+\min\{a-1,b-1\}$ and $\nu(a,b)=\max\{a-1,b-1\}$. Then
\[
\sum_{ k=\mu(a,b)}^{ \nu(a,b) } i_k
\]
is defined as \dots
\end{document}