Knowing how much \resizebox scales

\resizebox is just \scalebox with a scale factor being the number that you ask for. So you can just persuade it to expose that number rather than just use it in a local group and discard it.

\documentclass{article}
\usepackage[paperwidth=12cm]{geometry}
\usepackage{graphicx}
\makeatletter
\let\zz\Gscale@box
\long\def\Gscale@box#1{%
\xdef\thelastscalefactor{#1}%
\zz{#1}}

\makeatother
\begin{document}
\def\figurea{\includegraphics[width=5cm]{example-image-a}}
\def\figureb{\includegraphics[width=2.5cm]{example-image-b}}
This is \texttt{fig-a} with its natural size:

\figurea
\bigskip

This is \texttt{fig-b} with its natural size:

\figureb
\bigskip

This is \texttt{fig-a} scaled to fit the page width:

\noindent%you need this:-)
\resizebox{\textwidth}{!}{\figurea}
\let\savethis\thelastscalefactor
\bigskip

How do I rescale \texttt{fig-b} to keep its relative size compared to \texttt{fig-a} without knowing its actual size?
\bigskip

\noindent
\scalebox{\savethis}{\figureb}

\end{document}

You can use

\resizebox{\fpeval{(2.5cm) / (5cm)}\textwidth}{!}{\figureB}

where \fpeval is provided by xfp.

Essentially you're resizing the image to a scaled version of \textwidth based on the ratio of the two figure's widths. This is known in this case, so it seems superfluous to evaluate (2.5cm) / (5cm) rather than just using .5. But in general this might not be known.

The following example is slightly different, using 2.5pc for the smaller image:

\documentclass{article}

\usepackage[paperwidth=12cm]{geometry}
\usepackage{graphicx,xfp}

\setlength{\parindent}{0pt}% Just for this example

\begin{document}

\def\figureA{\includegraphics[width=5cm]{example-image-a}}
\def\figureB{\includegraphics[width=2.5pc]{example-image-b}}

This is \texttt{figureA} with its natural size:

\figureA
\bigskip

This is \texttt{figureB} with its natural size:

\figureB
\bigskip

This is \texttt{figureA} scaled to fit the page width:

\resizebox{\textwidth}{!}{\figureA}
\bigskip

How do I rescale \texttt{figureB} to keep its relative size compared to \texttt{figureA} without knowing its actual size?
\bigskip

\resizebox{\fpeval{(2.5pc) / (5cm)}\textwidth}{!}{\figureB}

\end{document}

You could also use

\resizebox{\fpeval{\textwidth * (2.5pc) / (5cm)}pt}{!}{\figureB}

since the result from \fpeval on dimensions is expressed in points.

---

If the figure dimensions are unknown, you can store the figures in boxes from which one can readily extract the width and/or height. Here is the above example written with that in mind:

\documentclass{article}

\usepackage[paperwidth=12cm]{geometry}
\usepackage{graphicx,xfp}

\setlength{\parindent}{0pt}% Just for this example

\begin{document}

\newsavebox{\figureAbox}% Store figure a in a box
\savebox{\figureAbox}{\includegraphics[width=5cm]{example-image-a}}
\newsavebox{\figureBbox}% Store figure b in a box
\savebox{\figureBbox}{\includegraphics[width=2.5pc]{example-image-b}}

This is \texttt{figureA} with its natural size:

\usebox\figureAbox% figure a
\bigskip

This is \texttt{figureB} with its natural size:

\usebox\figureBbox% figure b
\bigskip

This is \texttt{figureA} scaled to fit the page width:

\resizebox{\textwidth}{!}{\usebox\figureAbox}% Scaled figure a
\bigskip

How do I rescale \texttt{figureB} to keep its relative size compared to \texttt{figureA} without knowing its actual size?
\bigskip

\resizebox{\fpeval{\textwidth * (\wd\figureBbox) / (\wd\figureAbox)}pt}{!}{\usebox\figureBbox}% Scaled figure b

\end{document}

Here I define \Resizebox, a variant of \resizebox that takes an optional argument; if absent it does the same as \resizebox, if present, it should be a (definable) control sequence where the scale factor is stored, for subsequent use.

\documentclass{article}
\usepackage[paperheight=40cm,paperwidth=21cm]{geometry}
\usepackage{graphicx}
\usepackage{xfp}

\newcommand\figureA{\includegraphics[width=5cm]{example-image-a}}
\newcommand\figureB{\includegraphics[width=2.5cm]{example-image-b}}

\makeatletter
\newcommand{\Resizebox}[4][]{%
  \if\relax\detokenize{#1}\relax
    \resizebox{#2}{#3}{#4}%
  \else
    \begingroup
    \sbox0{#4}%
    \sbox2{\resizebox{#2}{#3}{#4}}%
    \@ifdefinable{#1}{\xdef#1{\fpeval{\wd2/\wd0}}}%
    \usebox{2}%
    \endgroup
  \fi
}
\makeatother

\begin{document}
\raggedright
This is \texttt{figure-a} with its natural size:

\figureA

\bigskip

This is \texttt{figure-b} with its natural size:

\figureB

\bigskip

This is \texttt{figure-a} scaled to fit the page width:

\Resizebox[\scaleforfigureA]{\textwidth}{!}{\figureA}

\bigskip

How do I rescale \texttt{figure-b} to keep its relative 
size compared to \texttt{figure-a} without knowing its actual size?

\bigskip

\texttt{Insert a nice piece of code here :)}

\scalebox{\scaleforfigureA}{\figureB}

\end{document}