How can I type formula cosine of two vectors nice?

The square brackets seem to be needlessly tall. Specifically, I don't think it's necessary to make the square brackets sufficiently tall to have them enclose the arrows. Nobody should be confused by the arrows "sticking out" above the brackets. Hence, using \big instead of \bigg for the size of the brackets should be fine.

Where I also see room for improvement, typographically speaking, is in the uneven heights of the arrows that are produced by \vv. Since the uneven heights are caused by the presence of the "primes" in the first argument of the \cross macro, one way to address this issue is to automatically add a "vertical phantom" (composed of #1...) to the second argument of the \cross macro.

\documentclass[12pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{mathtools}
\DeclarePairedDelimiter{\abs}{\lvert}{\rvert}
\usepackage{fourier,esvect}
\usepackage[margin=2cm]{geometry}
\newcommand{\cross}[2]{\bigl[ \vv{#1},\vv{#2\vphantom{#1}} \bigr]}
\newcommand\z{\vphantom{{}'}} % insert a vertical phantom as tall as a superscript prime
\begin{document}
\[
\cos \varphi =\dfrac{\cross{CA'}{CB} \cdot \cross{CA'}{CD}}
{\abs*{\cross{CA'}{CB}} \cdot  \abs*{\cross{CA'}{CD}} }\,. 
\]
\end{document}

Another option is to use bold letters for vectors. I have also changed the backets to parenthesis, hoping that won't change the meaning in your subject. physics package is used for making vectors bold with \vb* macro. If you want upright letters for vectors, use \vb without star. Since \cross is defined by defined by physcis, I have changed it to \Cross.

\documentclass[12pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{fourier}
\usepackage{physics}
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\newcommand{\Cross}[2]{(\vb*{#1},\vb*{#2})}
\begin{document}
\[\cos \varphi =\dfrac{\Cross{CA'}{CB} \cdot \Cross{CA'}{CD}}{\vert \Cross{CA'}{CB} \vert \cdot \vert \Cross{CA'}{CD} \vert} \]
\end{document}

I have also removed \left and \right from \vert.

One possible solution (since you haven't told us how you would like it to be, it's a guess):

\documentclass{article}

\usepackage{mathtools}
\usepackage{fourier}

\DeclarePairedDelimiter{\abs}{\lvert}{\rvert}
\newcommand*\cross[2]{\left[\overrightarrow{#1},\overrightarrow{#2}\right]}

\begin{document}

\begin{equation}
  \cos\varphi
  = \frac{\cross{CA'}{CB} \cdot \cross{CA'}{CD}}{\abs*{\cross{CA'}{CB}} \cdot \abs*{\cross{CA'}{CD}}}.
\end{equation}

\end{document}

In this way the brackets will automatically scale relative to the material.

Note that I have used \overrightarrow instead of \vv to avoid loading the esvect package. (I don't think it's a 'bad' package but I just prefer to load as few packages as possible.)

Update

In case you always have vectors like in the example, you can make the code simpler:

\documentclass{article}

\usepackage{mathtools}
\usepackage{fourier}

\DeclarePairedDelimiter{\abs}{\lvert}{\rvert}
\newcommand*\cross[2]{\left[\overrightarrow{#1},\overrightarrow{#2}\right]}
\newcommand*\crossProduct[3]{
  \frac{\cross{#1}{#2} \cdot \cross{#1}{#3}}% nominator
       {\abs*{\cross{#1}{#2}} \cdot \abs*{\cross{#1}{#3}}}% denominator
}

\begin{document}

\begin{equation}
  \cos\varphi
  = \crossProduct{CA'}{CB}{CD}
\end{equation}

\end{document}