Unable to compile no matter the package

You need to use xelatex rather than xetex. Likewise, lualatex instead of luatex.

When you run xetex, it loads with the plain TeX format that doesn't include the LaTeX definitions so you get the error that you encountered.

As supplement to nice @Don Hosek answer (+1).

With use of the pdfLaTeX engine, you need to declare font encoding, i.e. T1 which support all "probne znakove":

Edit:

• Option [utf8] in the \documentclass is not defined, consequently using it you receive warning: Unused global option(s): [utf8]. Correctly you should write:

\documentclass{article}
\usepackage[utf8]{inputenc}

• LaTeX version after April 2018 LATEX News has default utf8 encoding, so you can dropout \usepackage[utf8]{inputenc}
• The default LaTeX font encoding is OT1, the encoding of the original Computer Modern fonts, which not define letter đ (\dj), consequently in your case you receive error Command \dj unavailable in encoding OT1 using LaTeX or pdfLaTeX compiler. Using them, this error is removed with use of T1 input encoding:

\documentclass{article}
\usepackage[T1]{fontenc}  % <---
\title{Mačak u vreći}
\author{Pero Perić}

\begin{document}
\maketitle

\section{Probni znakovi}
šđžčć ŠĐŽČĆ

\end{document}

• using XeLaTeX and luaLaTeX you not need T1 encoding.