Why doesn't \dim_eval:n return appropriate result and behaves weirdly?

\dim_eval:n uses ε-TeX's \dimexpr to perform the calculation, and \dimexpr doesn't do floating point arithmetics; it can only divide dimensions by integers. What you get when you do

\dim_eval:n{300pt/5mm}

is actually

\dim_eval:n{300pt/5}mm

(thus the “silly units”), and with

\dim_eval:n{300pt/14.22636pt}

you are actually doing

\dim_eval:n{300pt/14}.22636pt

which are exactly the output you see.

Use \fp_to_dim:n (or \fp_to_decimal:n if you don't want the units). With versions of expl3 older than May 23, 2019 you need to put the denominator around parentheses to stick the unit to the number. In newer versions multiplication by juxtaposition (e.g., 5mm or 2pi) has higher precedence than explicit multiplication and division so that the unit is converted before the division:

\documentclass[varwidth]{standalone}
\usepackage{expl3}
\begin{document}
  \ExplSyntaxOn
    \fp_to_dim:n{300pt/5mm}\\
    % \fp_to_dim:n{300pt/(5mm)}\\
    \fp_to_dim:n{5mm}\\
    % \fp_to_dim:n{300pt/(14.22636pt)}
    \fp_to_dim:n{300pt/14.22636pt}
  \ExplSyntaxOff
\end{document}

It is more natural to express a ratio of two lengths as a number than as a length:

\documentclass[varwidth]{standalone}
\usepackage{expl3}
\begin{document}
  \ExplSyntaxOn

    \fp_eval:n{(300pt)/(5mm)}

   \ExplSyntaxOff
\end{document}