Optional arguments in plain TeX

Look ahead for the [:

\def\stile{\futurelet\next\dostile}
\def\dostile{%
  \mathrel{|\mkern-4mu{-}}
  \ifx[\next %]
    \expandafter\dostileopt
  \fi
}
\def\dostileopt[#1]{_{\rm #1}}

$\stile \alpha$

$\stile[K] \alpha$

\bye

You could use the plain compatible eplain macros:

\input eplain

\catcode`\@=11

\def\stile{\@getoptionalarg\xstile}

\def\xstile{{|\kern-.225em-}%
    \ifx\@optionalarg\empty\else_{\rm\@optionalarg}\fi}

\catcode`\@=12

$$\stile \alpha$$

$$\stile[K] \alpha$$

\bye

With a primitive implementation of \@ifnextchar (taking the one from latex.ltx would be better, I guess):

\catcode`\@=11
\long\def\@firstoftwo#1#2{#1}%
\long\def\@secondoftwo#1#2{#2}%
\long\def\@ifnextchar#1#2#3#4{%
  \ifx#1#4%
    \expandafter\@firstoftwo
  \else
    \expandafter\@secondoftwo
  \fi
  {#2}{#3}#4}%
\def\stile{\@ifnextchar[{\stile@i}{\stile@ii}}%
\def\stile@i[#1]{{|\kern-.225em-}_{\rm #1}}%
\def\stile@ii{{|\kern-.225em-}}%
\catcode`\@=12

$$\stile \alpha$$

$$\stile[K] \alpha$$

\bye

A more stable version of \@ifnextchar (similar to the one of latex.ltx but not the same):

\long\def\@ifnextchar#1#2#3{%
  \let\@tmpa=#1%
  \def\@tmpb{#2}%
  \def\@tmpc{#3}%
  \futurelet\@future\@ifnextchar@i}%
\def\@ifnextchar@i{%
  \ifx\@tmpa\@future%
    \expandafter\@tmpb
  \else
    \expandafter\@tmpc
  \fi}%