TikZ matrix delimiters are too big

The delimiters can be shifted through the styles every (left|right) delimiter:

\documentclass[a5paper]{article}% (a5paper for smaller image width)
\usepackage{amsmath,tikz}
\usetikzlibrary{matrix}
\begin{document}

AMSMath:

\[ \begin{pmatrix}
 1 & 2 & 3 \\
 4 & 5 & 6
\end{pmatrix} \]

TikZ:

\[
  \begin{tikzpicture}[
    every left delimiter/.style={xshift=.75em},
    every right delimiter/.style={xshift=-.75em},
  ]
    \matrix[
      matrix of math nodes,
      left delimiter=(,
      right delimiter=),
    ] (m) {
      1 & 2 & 3 \\
      4 & 5 & 6 \\
    };
  \end{tikzpicture}
\]

\end{document}

A TikZ matrix is a node which contains other nodes, so the distance between nodes and brackets is twice the inner sep one due to internal nodes and the second is the inner sep between nodes and matrix border.

You can fix a matrix-node inner sep to 0pt and later on fix another inner sep for nodes (see How to fix a certain distance between a TikZ matrix border and its nodes?).

\documentclass[a5paper]{article}% (a5paper for smaller image width)
\usepackage{amsmath,tikz}
\usetikzlibrary{matrix}
\begin{document}

AMSMath:

\[ \begin{pmatrix}
 1 & 2 & 3 \\
 4 & 5 & 6
\end{pmatrix} \]

TikZ: default inner sep

\[
  \begin{tikzpicture}
    \matrix[draw, %<- just used to show differences
      matrix of math nodes,
      left delimiter=(,
      right delimiter=),
    ] (m) {
      1 & 2 & 3 \\
      4 & 5 & 6 \\
    };
  \end{tikzpicture}
\]

TikZ: matrix inner sep = 0pt

\[
  \begin{tikzpicture}
    \matrix[draw, %<- just used to show differences
      inner sep=0pt, %<- set inner sep for all nodes 
      nodes={inner sep=.3333em}, %<- set another inner sep for inner nodes
      matrix of math nodes,
      left delimiter=(,
      right delimiter=),
    ] (m) {
      1 & 2 & 3 \\
      4 & 5 & 6 \\
    };
  \end{tikzpicture}
\]
\end{document}

perhaps you would like to reduce the inner sep a bit more, as below it gives

\documentclass{article}
\usepackage{amsmath,tikz}
\usetikzlibrary{matrix}    
\begin{document}

AMSMath:

\[ \begin{pmatrix}
 1 & 2 & 3 \\
 4 & 5 & 6
\end{pmatrix} \]

TikZ:

\[ \begin{tikzpicture}
\matrix[matrix of math nodes,left delimiter=(,right delimiter=),
outer sep = 0pt,inner sep=2pt] (m) {
 1 & 2 & 3 \\
 4 & 5 & 6 \\ }; \end{tikzpicture} \]

TikZ without delimiters + AMSMath:

\[ \begin{pmatrix} \begin{tikzpicture}
\matrix[matrix of math nodes] (m) {
 1 & 2 & 3 \\
 4 & 5 & 6 \\ }; \end{tikzpicture} \end{pmatrix} \]

\end{document}