Dismissed from PhD program and reapplying; how to answer question about dismissal?

The fact that you were not allowed to continue toward the Ph.D. is not "dismissal" in the sense that that question means. (I can certainly see how the terminology could be confusing though.) The question is there to identify students who have been involved in academic (or other) misconduct. Academic failure does not fall into the misconduct category, so you should answer "No" to the question. Doing so will not imperil your application in any way.

Obtain a copy of your transcript. It should say there whether you were dismissed or expelled. To be on the safe side, you should ask your former advisor.

At my program, a few students who we do not have confidence in might be asked to leave the program prior to advancing to candidacy. If they've passed their quals, they can leave with a MA if the department approves it.

This is a mutually agreed form of separation in lieu of the proceedings for formal expulsion so it does not count as a dismissal.

This is simple. Your answer to their question should be NO. The question they are asking is designed to identify people who have a history of misconduct, and whose past institutions have taken action against them. Your situation has nothing to do with that. You were not invited to continue on to the PhD, because of the lack of research fit. This is not the same thing as being expelled, dismissed, suspended, or any other situation involving disciplinary action.