Horizontal curly braces inside matrix

Here there is my idea: to have the curly braces inside the matrix of your determinant I have used always underbrace command.

\documentclass[a4paper,12pt]{article}
\usepackage{amsmath,amssymb}
\begin{document}
\begin{equation}
R(f,g)=
\begin{vmatrix}
\underbrace{\begin{matrix}
a_0 &   &   &   \\
a_1 & a_0 &  &  \\
a_2 & a_1 & a_0 & \\
 & \ldots &  &  \\
 &   &   & a_m 
\end{matrix}}_{n \text{ columns}} & \underbrace{\begin{matrix}
b_0 &   &   &   \\
b_1 & b_0 &  &  \\
b_2 & b_1 & b_0 & \\
 & \ldots &  &  \\
 &   &   & b_m 
\end{matrix}}_{m \text{ columns}}\\
\end{vmatrix}
\end{equation}
\end{document}

This MWE have the curly brackets more inside:

\documentclass[a4paper,12pt]{article}
\usepackage{amsmath,amssymb}
\begin{document}
\begin{equation}
R(f,g)=\begin{array}{c}
\begin{vmatrix}
\underbrace{\begin{matrix}
a_0 &   &   &   \\
a_1 & a_0 &  &  \\
a_2 & a_1 & a_0 & \\
 & \ldots &  &  \\
 &   &   & a_m 
\end{matrix}}_{n \text{ columns}} & \underbrace{\begin{matrix}
b_0 &   &   &   \\
b_1 & b_0 &  &  \\
b_2 & b_1 & b_0 & \\
 & \ldots &  &  \\
 &   &   & b_m 
\end{matrix}}_{m \text{ columns}}\\
 & 
\end{vmatrix}
\end{array}
\end{equation}
\end{document}

You can use nicematrix. Some work is needed to pull the braces nearer the matrix, but not so much: the idea is to use a phantom matrix to set the width for the underbrace; then it is raised up.

\documentclass{article}
\usepackage{amsmath}
\usepackage{nicematrix}

\usepackage{lipsum} % mock text to show spacing around the display

\begin{document}

\lipsum[1]
\begin{equation}
R(f,g)=
\begin{vNiceMatrix}[last-row]
  a_0 & & & & b_0 &  &  &  \\
  a_1 & a_0 & & & b_1 & b_0 &  &  \\
  a_2 & a_1 & a_0 & & b_2 & b_1 & b_0 &  \\
  \multicolumn{3}{c}{\cdots} & &  \multicolumn{3}{c}{\cdots}  &  \\
  & & & a_m &  &  &  & b_n \\
  \multicolumn{4}{c}{%
    \raisebox{1ex}{$
      \underbrace{%
        \hphantom{\begin{matrix} a_0 & a_1 & a_2 & a_m\end{matrix}}
      }_{\text{$n$ columns}}
    $}%
  }
  &
  \multicolumn{4}{c}{%
    \raisebox{1ex}{$
      \underbrace{%
        \hphantom{\begin{matrix} b_0 & b_1 & b_2 & b_n\end{matrix}}
      }_{\text{$m$ columns}}
    $}%
  }
\end{vNiceMatrix}
\end{equation}
\lipsum[2]

\end{document}

Note the \multicolumn instructions in the body to avoid misalignments.

I would place the two underbraced explanatory expressions outside the vertical bars. This may be done by with the help of 2 "\smashed" matrix environments and two judiciously chosen typographic struts -- one inside the \left\vert ... \right\vert group, containing a matrix with 5 columns, and one outside that group, containing a matrix with 7 columns (and a \smash[t] directive).

\documentclass{article}
\usepackage{amsmath}
\begin{document}
\begin{equation}
R(f,g)=
\left\vert 
  \smash{\underbrace{\begin{matrix}
    a_0 &     &     &    \\
    a_1 & a_0 &     &   \\
    a_2 & a_1 & a_0 &   \\
        & \cdots   \\
     & & & a_m  
     \end{matrix}}_{\text{$n$ columns}}}\;
  \smash{\underbrace{\begin{matrix}
    b_0 &     &     &    \\
    b_1 & b_0 &     &   \\
    b_2 & b_1 & b_0 &   \\
        & \cdots   \\
     & & & b_n  
     \end{matrix}}_{\text{$m$ columns}}}
  %% insert a first typographic strut:
  \vphantom{\begin{matrix} 1\\2\\3\\4\\5 \end{matrix}}
\right\vert
%% insert a second typographic strut:
\smash[t]{\vphantom{\begin{matrix} 1\\2\\3\\4\\5\\6\\7 \end{matrix}}}
\end{equation}
\end{document}