Executing a command within `if` statement and on success perform further steps

The echo always succeeds. Do without it and the subshell:

#!/bin/bash
echo "enter the directory name"
read ab
check(){
    if mkdir "$ab" 2>/dev/null; then
      echo "directory created "
      ls -ld "$ab"
      exit
    else
      echo "try again "
      echo "enter new value for directory: "
      read ab
      check
    fi
}
check

(( echo `mkdir $ab` 2>/dev/null ))
bash: ((: echo  2>/dev/null : syntax error in expression (error token is "2>/dev/null ")

Double parentheses surround arithmetic expressions. That command that doesn't do anything close to what you intended. mkdir $ab doesn't write anything to standard output, so `mkdir $ab` is an empty string, so the content of the arithmetic expression is echo 2>/dev/null. The > sign is the “greater than” operator here, and echo is a variable name. The arithmetic expression is syntactically incorrect (you can't have a variable followed by a literal integer). Since the command is syntactically incorrect, it returns an error status, and the else branch is taken.

To test the return status of a command, just write that command.

if mkdir -- "$ab" 2>/dev/null

Additional tips:

• It's a bad idea to hide the errors from a command. The user needs to know if the error is permission denied, I/O error, etc. To create a directory if it isn't there and succeed if the directory already exists, use mkdir -p -- "$ab".
• Always use double quotes around variable substitutions.
• If the value of ab begins with a dash, it'll be interpreted as an option. Put -- first to mark the end of options.