TikZ: Curve through specified points

Like this (note that I replaced your last coordinate by -- cycle)?

\draw [rounded corners=5pt]
  (A5-1-3.north west) -- (A5-1-4.north east) --
  (B5-1-4.south east) -- (B5-1-4.south west) -- cycle;

or more like this?

\draw
  [decoration={random steps,segment length=6pt,amplitude=1.5pt},
   decorate, rounded corners=2pt]
  (A5-1-3.north west) -- (A5-1-4.north east) --
  (B5-1-4.south east) -- (B5-1-4.south west) -- cycle

I’d prefer the first one because it looks better matching to the style of printed text. The latter seems like some one attacked your paper ;-)

PS: I don’t know why the zig zag curve doesn’t close itself maybe one can fix this, please?

Replace the last line with:

\draw plot [smooth cycle,tension=0.3] coordinates {(A5-1-3.north west)  (A5-1-4.north east)  (B5-1-4.south east) (B5-1-4.south west)};

The option smooth cycle results in the points to be closed using a smooth curve (so you don't include the initial point twice as was the case in your original code).

The tension can be adjusted to suit. Also refer to Easy curves in TikZ.

\documentclass{standalone}
\usepackage{comment}

\usepackage{tikz,pgf}

\usetikzlibrary{fit,scopes,calc,matrix,positioning,decorations.pathmorphing}

\begin{document}

\begin{tikzpicture}
    \begin{scope}
            \matrix (A5) [matrix of nodes, ampersand replacement = \&] {
                {7}  \& {2} \& {7} \& {7} \& {2} \& {8}\\
            };
    \end{scope}
    \begin{scope}
        \matrix (B5) at ([yshift=-2.5cm]A5-1-1.south) [anchor=B5-1-1.north, matrix of nodes, ampersand replacement = \&] {
            {12} \& {7} \& {2} \& {7} \& {2} \& {8}\\
        };
    \end{scope}
\draw plot [smooth cycle,tension=0.3] coordinates {(A5-1-3.north west)  (A5-1-4.north east)  (B5-1-4.south east) (B5-1-4.south west)};
\end{tikzpicture}
\end{document}