Macro definition not working when using LuaLaTeX and polyglossia (with `\setmainlanguage{german}`)

You can use \AtBeginDocument to delay \setmainlanguage (in opposite to babel polyglossia does language depending catcode changes immediate at least for german, so it is somehow recommended to do language selections as late as possible to avoid problems loading other packages or additional preamble definitions):

% !TeX program = lualatex

\documentclass[a4paper,12pt,twoside=semi,openright]{scrbook}

\usepackage{polyglossia}
\AtBeginDocument{%
  \setmainlanguage{german}%
  \setotherlanguage{english}%
}
\usepackage{fontspec}

\usepackage{subfig}

\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}

\renewcommand{\vec}[1]{\bm{#1}}

\begin{document}
    \begin{align}
        \vec{x} = 1
    \end{align}
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-engine: luatex
%%% TeX-master: t
%%% End:

---

BTW: The problem seems to be related to a combination of lualatex, polyglossia with main language german, caption and bm. A MWE would be:

% !TeX program = lualatex
\listfiles
\documentclass{article}

\usepackage{polyglossia}
\setmainlanguage{german}
\usepackage{caption}
\usepackage{bm}

\begin{document}
$\bm{x} = 1$
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-engine: luatex
%%% TeX-master: t
%%% End:

If you use xelatex instead of lualatex or delay \setmainlanguage{german}, e.g., using \AtBeginDocument, or load bm before polyglossia the problem does not longer exist.

So you can also use:

% !TeX program = lualatex
\listfiles
\documentclass{article}

\usepackage{bm}
\usepackage{polyglossia}
\setmainlanguage{german}
\usepackage{caption}
\usepackage[german=quotes]{csquotes}
\begin{document}
$\bm{x} = 1$
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-engine: luatex
%%% TeX-master: t
%%% End:

You should not use \AtBeginDocument to load a package, because unused global options are reported before execution of the \AtBeginDocument hook. And you cannot use \AtBeginDocument to load package csquotes. But you can use \AtEndPreamble of etoolbox, because the corresponding hook is executed at the very beginning of \begin{document}:

% !TeX program = lualatex
\listfiles
\documentclass{article}

\usepackage{etoolbox}
\usepackage{bm}
\usepackage{polyglossia}
\AtEndPreamble{\setmainlanguage{german}}
\usepackage{caption}
\AtEndPreamble{\usepackage[german=quotes]{csquotes}}
\begin{document}
$\bm{x} = 1$
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-engine: luatex
%%% TeX-master: t
%%% End:

Last but not least, you can use babel instead of polyglossia to have German as main language and English as second language with lualatex too:

% !TeX program = lualatex
\listfiles
\documentclass{article}

\usepackage[main=ngerman,english]{babel}
\usepackage{bm}
\usepackage{caption}
\usepackage[german=quotes]{csquotes}
\begin{document}
$\bm{x} = 1$
\end{document}

%%% Local Variables:
%%% mode: latex
%%% TeX-engine: luatex
%%% TeX-master: t
%%% End:

Under LuaLaTeX, the packages amsmath, amssymb, and bm should always be loaded before both fontspec and polyglossia. (It also used to be standard to advise to load fontspec after any packages that modify math-related macros -- such as, of course, amsmath, amssymb, and bm. However, at some point in the recent past, an update to fontspec has occurred which no longer makes this recommendation a necessity.)

If, for some reason, you simply must load polyglossia first, then at least defer the instructions \setmainlanguage{...} and \setotherlanguage{...} to a point later on in the preamble, i.e., after amsmath, amssymb, bm and fontspec have been loaded.

Do note that I deliberately started the answer with "Under LuaLaTeX, ...": I'm afraid I have no experience with package loading orderings under XeLaTeX.

The following, modified version of your MWE compiles fine:

% !TeX program=lualatex
\documentclass[a4paper,12pt,twoside=semi,openright]{scrbook}

\usepackage{amsmath,amssymb,bm}
\usepackage{fontspec}

\usepackage{polyglossia} % load it *after* amsmath, amssymb, bm, and fontspec
\setmainlanguage{german}
\setotherlanguage{english}

\usepackage{subfig} % it doesn't seem to matter when this package is loaded

\renewcommand{\vec}[1]{\bm{#1}}

\begin{document}
\[ \vec{x} = 1 \]
\end{document}

The problem is that \setmainlanguage{german} sets " to a shorthand character and loading bm after that declaration completely breaks the definitions made by the package when LuaLaTeX is used.

The bizarre string you get is essentially random. Without loading subfig you get nothing (but no x either).

Loading amsmath and bm before or after polyglossia is irrelevant, as long as the language declarations are made after package loading.

\documentclass{article}

\usepackage{polyglossia}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}

\usepackage{subfig}

\setmainlanguage{german}

\renewcommand{\vec}[1]{\bm{#1}}

\begin{document}

$\vec{x}$

\end{document}

Try moving \usepackage{bm} after \setmainlanguage{german} and the issue will appear again.