How to draw a dotted rectangle around part of equation?

The dashbox package can be tweaked to do what you want:

\documentclass{article}

\usepackage{dashbox}
\newcommand\emptydottedbox{%
    \setlength{\dashlength}{1pt}%     % length of dash + gap
    \setlength{\dashdash}{0.5pt}%     % length of dash
    \setlength{\fboxrule}{0.5pt}%     % width of line
    \setlength{\fboxsep}{0pt}%        % separation between box and inner content
    \dbox{\ensuremath{\phantom{x}}}%
    }

\begin{document}

\[
\int_{\emptydottedbox}^{\emptydottedbox} \emptydottedbox
\]

\end{document}

With tikz you can draw these boxes:

\documentclass{article}
\usepackage{tikz}
\newcommand{\smallbox}{\begin{tikzpicture}
\draw[thick,dotted] (0,0) rectangle
(0.3, 0.3);
\end{tikzpicture}
}
\newcommand{\medbox}{\begin{tikzpicture}
\draw[thick,dotted] (0,0) rectangle
(0.5, 0.5);
\end{tikzpicture}
}
\begin{document}

\[\int_{\smallbox}^{\smallbox} {\medbox} \]
\end{document}

Maybe you need to fiddle a bit with the dimensions and/or the spacing.