Please use \mathaccent for accents in math mode

You have to use \acute{e} instead of é in math mode, but I think it depends on your inputenc.

I don't have that error and your code works fine (even if is a too long line in math mode with no breaks).

You should not set natural language words in math mode (although you only get an error for the accented letters)

The markup should be more like:

\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath}
\usepackage[hungarian]{babel}% I think? :-)

\begin{document}

$S = (s_0,S)$,
ahol 
$s_0 = x_1 := \langle\rangle,\dots,x_n := \langle\rangle$
 és 
\[S = \{x_0, x_1 := \operatorname{lorem}(x_0),
\operatorname{hiext}(x_1,f_0(x_0.\mathrm{lov})),\]
ha 
$x_0 \not\equiv \langle\rangle$

\end{document}