\star vs. \ast in formulas. Which one to use?

At the character level, there are several star-like symbols, but the most common are:

U+002A ASTERISK “*” is the well-known Ascii character with multiple and vague semantics. In mathematics, it is often used as a unary postfix operator or modifier (e.g., N* denotes the set of natural numbers without zero), and its appearance in most fonts is superscript-like, at least somewhat above the baseline of text. In LaTeX math mode, you cannot write it directly, as it gets displayed as asterisk operator. Use mbox{*} instead or, if you prefer a larger size for the symbol, ^{*}. I would expect that this is what you need for “r-star”.

U+2217 ASTERISK OPERATOR “∗”, classified as a mathematical symbol. It is generally placed vertically near the middle of the x-height, like “+”, “×” etc. are. It is the standardized symbol for convolution operator, and it has many specialized uses. It can be identified with the symbol that the normal Ascii asterisk “*” character produces in LaTeX math mode. It can be written as \ast if desired.

U+22C6 STAR OPERATOR “⋆”. This has no generally standardized usage, but it is classified as a mathematical symbol. It, too, is generally placed vertically near the middle of the x-height. It can be identified with \star in LaTeX.

I think it's just personal preference. In the standard setup * and \ast are the same.

fontmath.ltx has

\DeclareMathSymbol{*}{\mathbin}{symbols}{"03} % \ast
\DeclareMathSymbol{\ast}{\mathbin}{symbols}{"03}

So both pick up character 3 from the symbol font with infix binary operator spacing.