How to stop the execution of DialogResult based on a condition?

As Hans Passant mentions in an comment, just set the DialogResult to None!
Like this:

private void btnOk_Click(object sender, EventArgs e)
{
    if(ValidationFailed())
    {
        this.DialogResult = DialogResult.None;
        return;
    }
    //...
}

Personally I wouldn't use DialogResults on buttons in this scenario. I only tend to set the DialogResult when there's only distinct options that do not require any additional logic (i.e. making a custom MessageBox).

What I would do is to just send the DialogResult yourself on success:

private void btnOk_Click(object sender, EventArgs e)
{
    if (allIsOK())
    {
        this.DialogResult = DialogResult.OK;
    }
}