How to avoid final vertical glue for boxes with very tiny font sizes

There is no final glue. The problem is that centered minipages (and \parbox) uses internally \vcenter, and it centers along the math axis -- which in turn depends on the current font size. So it matters if you change the font inside or before the minipage:

\documentclass{article}

\usepackage{lmodern}

\newlength\myfontdim

\newcommand{\mytest}[1]{%
  \myfontdim=#1%
  \noindent\fboxsep0pt%
  $\frac{a}{a}$ x
  \fbox{%
   $\vcenter {\hbox to 1cm {\fontsize{\myfontdim}{\dimexpr 1.05\myfontdim}\selectfont x\hfil}}$}
  \fbox{%
   \fontsize{\myfontdim}{\dimexpr 1.05\myfontdim}\selectfont
   $\vcenter {\hbox to 1cm {x\hfil}}$}
}%


\begin{document}

\mytest{10pt}

\mytest{5pt}

\mytest{1pt}

\mytest{0.5pt}

\end{document}

Add [b] to the minipage.

\documentclass{article}

\usepackage{lmodern}

\newlength\myfontdim

\newcommand{\mytest}[1]{%
  \myfontdim=#1%
  \noindent\fboxsep0pt%
  \fbox{\begin{minipage}[b]{1cm}
    \fontsize{\myfontdim}{\dimexpr 1.05\myfontdim}\selectfont
    \parskip 0.15\myfontdim plus 0pt minus 0pt\par
    A\par
    in\par
    Test
  \end{minipage}}
}%


\begin{document}

\mytest{10pt}

\mytest{5pt}

\mytest{1pt}

\mytest{0.5pt}

\end{document}

Alternately, depending on how you want the line spacing external to your \fbox, put the \fontsize before the minipage (within a group):

\documentclass{article}

\usepackage{lmodern}

\newlength\myfontdim

\newcommand{\mytest}[1]{%
  \myfontdim=#1%
  \noindent\fboxsep0pt%
  \bgroup%
  \fontsize{\myfontdim}{\dimexpr 1.05\myfontdim}\selectfont
  \fbox{\begin{minipage}{1cm}
    \parskip 0.15\myfontdim plus 0pt minus 0pt\par
    A\par
    in\par
    Test
  \end{minipage}}%
  \egroup%
}%


\begin{document}

\mytest{10pt}

\mytest{5pt}

\mytest{1pt}

\mytest{0.5pt}

\end{document}

---

Without the grouping, the line-spacing shrinks, but of course, the smaller font size persists outside of your macro.

\documentclass{article}

\usepackage{lmodern}

\newlength\myfontdim

\newcommand{\mytest}[1]{%
  \myfontdim=#1%
  \noindent\fboxsep0pt%
  \fontsize{\myfontdim}{\dimexpr 1.05\myfontdim}\selectfont
  \fbox{\begin{minipage}{1cm}
    \parskip 0.15\myfontdim plus 0pt minus 0pt\par
    A\par
    in\par
    Test
  \end{minipage}}
}%


\begin{document}

\mytest{10pt}

\mytest{5pt}

\mytest{1pt}

\mytest{0.5pt}

\end{document}