How can I prevent a column break before the first sub-entry in the index?

The solution is to move \nobreak after \@idxitem:

% arara: lualatex
% arara: makeindex
% arara: lualatex

\documentclass{article}
\usepackage{makeidx}

\makeatletter
% we don't want a page break before a subitem
\renewcommand\subitem{\@idxitem\nobreak\hspace*{20\p@}}
\makeatother

\makeindex
\begin{document}
\index{gap!radial}\index{gap!axial}
\index{gapa!radial}\index{gapa!axial}
\index{gapb!radial}\index{gapb!axial}
\index{gapc!radial}\index{gapc!axial}
\index{gapd!radial}\index{gapd!axial}
\index{gap1!radial}\index{gap1!axial}
\index{gapa1!radial}\index{gapa1!axial}
\index{gapb1!radial}\index{gapb1!axial}
\index{gapc1!radial}\index{gapc1!axial}
\index{gapd1!radial}\index{gapd1!axial}
\index{gap2!radial}\index{gap2!axial}
\index{gapa2!radial}\index{gapa2!axial}
\index{gapb2!radial}\index{gapb2!axial}
\index{gapc2!radial}\index{gapc2!axial}
\index{gapd2!radial}\index{gapd2!axial}
\printindex
\end{document}

The standard definition of \subitem is without \nobreak; the command \@idxitem just does a \par and sets \hangindent, so if we add the penalty (\nobreak is \penalty 10000) after it, TeX will not break a page at this point.

---

If you just want that the first subitem remains attached to the main entry, then change the code between \makeatletter and \makeatother into

\makeatletter
% we don't want a page break before the first subitem
\newif\iffirst@subitem
\def\@idxitem{%
  \par\hangindent40\p@ % original
  \first@subitemtrue   % added
}
\def\subitem{%
  \par\hangindent40\p@
  \iffirst@subitem
    \nobreak
    \first@subitemfalse
  \fi
  \hspace*{20\p@}}
\makeatother

This will add the penalty only between a main item and the first subitem.

I'm working with lengthy index for a textbook. I found it useful implement egreg's suggestion using LaTeX's renewcommand, which I could drop into the preamble. This code also suppresses breaks before first subsubitem and keeps \see entries on the same page. (The \see fix is somewhat experimental, as I'm not sure it will be appropriate for longer entries, but it does eliminate widows...)

    \newif\iffirst@subitem
    \newif\iffirst@subsubitem\first@subsubitemfalse
    \renewcommand\@idxitem{\par\hangindent 40\p@ \first@subitemtrue}
    \renewcommand\subitem{\par\hangindent 
           40\p@\iffirst@subitem\nobreak\first@subitemfalse\fi 
           \hspace*{20\p@}\first@subsubitemtrue}
    \renewcommand\subsubitem{\par\hangindent 
           40\p@\iffirst@subsubitem\nobreak\first@subsubitemfalse\fi 
           \hspace*{30\p@}}
    \renewcommand\see[2]{\samepage\emph{\seename} #1}