Code crashes when inserting a macro into a `\label` and `protecting` the macro doesn't fix

The macros of xstring are not, generally, expandable, which is needed here. Thus, I change over to listofitems, which produces expandable results.

\documentclass{amsart}
\usepackage{soul}
\usepackage{listofitems}
\begin{document}
\setsepchar{.}
\readlist\fileroot{file.png}
\edef\tmp{fig:\fileroot[1]}
\tmp
\begin{figure}
\caption{nothing}
\label{\tmp}
\end{figure}

I refer to figure \ref{fig:file}
\end{document}

And here's a version that doesn't even need listofitems:

\documentclass{amsart}
\usepackage{soul}
\newcommand\fileroot[1]{\filerootaux#1.\relax}
\def\filerootaux#1.#2\relax{#1}
\begin{document}
\edef\tmp{fig:\fileroot{file.png}}
\tmp
\begin{figure}
\caption{nothing}
\label{\tmp}
\end{figure}

I refer to figure \ref{fig:file}
\end{document}

I would just use the ability of xstring to store the result of the extraction in a macro.

\documentclass{amsart}
\usepackage{soul}
\usepackage{xstring}
\begin{document}
\StrBefore{file.png}{.}[\tmp]
\edef\tmp{fig:\tmp}
\tmp
\begin{figure}
\caption{nothing}
\label{\tmp}
\end{figure}
\end{document}

The argument to \label should be something that eventually becomes a string of characters, but \StrBefore eventually becomes instructions to print some characters, which is a very different thing.

It's simpler, unless your file names can have multiple periods in their names.

\documentclass{article}

\makeatletter
\newcommand{\stripext}[1]{\strip@ext#1.\@nil}
\def\strip@ext#1.#2\@nil{#1}
\makeatother

\begin{document}

\stripext{file.png}
\label{\stripext{file.png}}

\stripext{filenoext}
\label{\stripext{filenoext}}

\end{document}

The trick is to use a macro with delimited arguments. If we do \stripext{file.png}, we get

\strip@ext file.png.\@nil

and the macro \strip@ext uses as #1 whatever comes along until the first .; in this case it is file.

In the case of \stripext{filenoext}, we get

\strip@ext filenoext.\@nil

and things are good as well. In the first case #2 is png., in the second case it is empty; but the second argument is simply thrown away.

The .aux file from the code above will contain

\relax
\newlabel{file}{{}{1}}
\newlabel{filenoext}{{}{1}}

so you see it's as expected.