Entering math mode in AUCTeX using \( and \)

I use yasnippet. awesome package for inserting snippets. there is latex snippet bundle you can use. Infact there is already a written snippet to insert exactly what you are asking for.

You can setup a key binding in your .emacs file. This definition is based on the one for TeX-insert-braces (C-c{) from tex.el.

(add-hook 'LaTeX-mode-hook
  '(lambda ()
    (define-key TeX-mode-map "\C-cm" 'TeX-insert-inline-math)
    (defun TeX-insert-inline-math (arg)
      "Like TeX-insert-brackes but for \(...\)" (interactive "P")
      (if (TeX-active-mark)
        (progn
          (if (< (point) (mark)) (exchange-point-and-mark))
          (insert "\\)")
          (save-excursion (goto-char (mark)) (insert "\\(")))
          (insert "\\(")
          (save-excursion
            (if arg (forward-sexp (prefix-numeric-value arg)))
            (insert "\\)"))))))

As described in its documentation, AUCTeX provides the function TeX-insert-dollar which inserts the two strings of TeX-electric-math and adjusts point position to between those strings when "$" is typed.

The following elisp can be added to a hook on LaTeX-mode to insert "\(" and "\)" then place the point after the opening paren when an unmatched "$" (i.e., the "$" is not closing an existing math environment) is typed:

(add-hook 'LaTeX-mode-hook
          (lambda () (set (make-variable-buffer-local 'TeX-electric-math)
                  (cons "\\(" "\\)"))))