Spacing within brackets of a matrix

Instead of trying to modify the inner workings of the vmatrix* environment, it may be more straightforward to start over with a basic array environment and simply fine-tune some of its settings.

I gather that your main concern is with the spacing between the vertical bars and the contents of the arrays. The following example shows how the spacing may be adjusted. The adjustments amounts used in the example -- 2mu on the left and 5mu on the right -- can, of course, be changed to suit your needs and tastes. (The spacing used internally by vmatrix is 0mu on both sides.)

\documentclass{article}
\usepackage{mathtools,array}
\begin{document}

\begin{tabular}{l>{$}l<{$}}
\texttt{vmatrix*[r]} &
\begin{vmatrix*}[r] 
    \vec{i} &   \vec{j} &   \vec{k} \\
    -3       &  -6      &  9  \\  
    2        &  -7     &   -4
\end{vmatrix*} \\[5ex]
\texttt{array} & 
\left\lvert\begin{array}{@{\mkern3mu} rrr @{\mkern5mu}}
    \vec{i} &   \vec{j} &   \vec{k} \\
    -3       &  -6      &  9  \\  
    2        &  -7     &   -4
\end{array}\right\rvert\\
\end{tabular}

\end{document}

I also encountered the same problem recently. I am not quite satisfied with the array environment because you need to specify the number of columns every time you make a matrix. It can be very tedious to replace every single instance of matrices in a document with the array environment.

Therefore, instead of using array, I dig a little bit into the source code of mathtools and found the definition of vmatrix*

\newenvironment{vmatrix*}[1][c]
  {\left\lvert\MT_matrix_begin:N #1}
  {\MT_matrix_end:\right\rvert}

This is quite easy to override. We just need to redefine the environment in the preamble and add some kerning before and after vertical line

\MHInternalSyntaxOn
\renewenvironment{vmatrix*}[1][c]
  {\left\lvert\mkern5mu\MT_matrix_begin:N #1}
  {\MT_matrix_end:\mkern5mu\right\rvert}
\MHInternalSyntaxOff

---

My original intention was to increase the spacing of bmatrix and vmatrix under newpxmath. For anyone who just wants to use bmatrix and vmatrix environment from amsmath, you can do something similar

\makeatletter
\renewenvironment{bmatrix}
{\left[\mkern3.5mu\env@matrix}
{\endmatrix\mkern3.5mu\right]}

\renewenvironment{vmatrix}
{\left\lvert\mkern5mu\env@matrix}
{\endmatrix\mkern5mu\right\rvert}
\makeatother

Here is a minimal working example

\documentclass{article}
\usepackage{mathtools}
\usepackage{newpxmath}

\MHInternalSyntaxOn
\renewenvironment{vmatrix*}[1][c]
  {\left\lvert\mkern5mu\MT_matrix_begin:N #1}
  {\MT_matrix_end:\mkern5mu\right\rvert}
\MHInternalSyntaxOff

\makeatletter
\renewenvironment{bmatrix}
  {\left[\mkern3.5mu\env@matrix}
  {\endmatrix\mkern3.5mu\right]}

\renewenvironment{vmatrix}
  {\left\lvert\mkern5mu\env@matrix}
  {\endmatrix\mkern5mu\right\rvert}
\makeatother

\begin{document}
\begin{align*}
  \begin{vmatrix*}[r]
    \vec{i} & \vec{j} & \vec{k} \\
    -3      & -6      & 9       \\
    2       & -7      & -4
  \end{vmatrix*} \\
  \begin{vmatrix}
    \vec{i} & \vec{j} & \vec{k} \\
    -3      & -6      & 9       \\
    2       & -7      & -4
  \end{vmatrix} \\
  \begin{bmatrix}
    \vec{i} & \vec{j} & \vec{k} \\
    -3      & -6      & 9       \\
    2       & -7      & -4
  \end{bmatrix}
\end{align*}
\end{document}

and a preview