\midrule splits vertical lines in table

I would never do what I did in the first table, but it is your choice, not mine. I just offer a much nicer alternative. However, please:

• Do not use \it and \bf: Does it matter if I use \textit or \it, \bfseries or \bf, etc
• Do not manually modify the size of the table nor the style of the caption, it's not semantic. What if you later decide that the captions should not be bold, but you've already add \bfseries to 50 captions? Are you going to remove them by hand? Well, you'll have to. Or you use packages that modify the styles globally. (Both things can be found on this website, however, simply look into the documentation of packages caption and float.)
• Do not specify [h] float placement alone, you can expect bad things happening.
• centering is not an environment, it is a command: \centering.
• Your \scalebox has no effect whatsoever since the thing you scale is the macro \small, which doesn't output anything. Use either \scalebox or \small (or rather none of them).

---

\documentclass{article}

\usepackage{array}
\newcolumntype{C}{>{\rule[-1.5ex]{0pt}{4.5ex}}c}

\makeatletter
\def\Hline{% a thicker \hline
  \noalign{\ifnum0=`}\fi\hrule \@height 1pt \futurelet
   \reserved@a\@xhline}
\makeatother

\usepackage{booktabs}
\newcommand\Midrule{\midrule[\heavyrulewidth]}

\begin{document}

\begin{table}
\centering
\begin{tabular}{| c | l @{\qquad} | c c C |}
\Hline
  \bfseries HD Level  &&&&\\
\hline
  1 & \itshape Rep-tile:  & $\{1,2\}$ & $\{1,3\}$ & $\{2,3\}$ \\
\cline{2-5}
    & \itshape Encoding: & 1       & 2       & 3 \\
\Hline
  2 & \itshape Rep-tile: & $\{2\}$ & $\{3\}$ & \\
\cline{2-5}
    & \itshape Encoding: & 1       & 2       & \\
\Hline
\end{tabular}
\caption{Bla bla}
\label{tab:SMuTexEncoding}
\end{table}


\begin{table}
\centering
\begin{tabular}{ c >{\itshape}l c c c }\toprule
\bfseries HD Level\\\Midrule
1 & Rep-tile: & $\{1,2\}$ & $\{1,3\}$ & $\{2,3\}$ \\\cmidrule{2-5}
  & Encoding: & 1         & 2         & 3 \\\Midrule
2 & Rep-tile: & $\{2\}$   & $\{3\}$   & \\\cmidrule{2-5}
  & Encoding: & 1         & 2         & \\\bottomrule
\end{tabular}
\caption{Bla bla}
\label{tab:SMuTexEncoding}
\end{table}

\end{document}

You can have thick \hlines with the makecell package and its \Xhline{width} and \Xcline{col1-col2}{width} commands. You can replace the vertical spacing added by booktabs around horizontal rules (that's responsible for non joining vertical rules) with the cellspace package; it lets you define minimal vertical spacingabove and below cells of a given column, prefixing its specifier with the letter S.

I also took the opportunity to define a set command, based on an example in the documentation of the mathtools package, that produces better horizontal spacing and variable-sized braces

\documentclass{article}
\usepackage{makecell}
\usepackage{multirow}
\usepackage{hhline}
\usepackage{cellspace}
\setlength\cellspacetoplimit{5pt}
\setlength\cellspacebottomlimit{5pt}
\usepackage{mathtools}
\DeclarePairedDelimiterX\set[1]\{\}{\nonscript\,#1\nonscript\,}

\begin{document}

\begin{table}[!htb]
\centering
\small
\begin{tabular}{| Sc | >{\itshape}l | Sc c c |}
\hline
{\bfseries HD Level} & & & & \\
\Xhline{2pt}
\multirow{2}{*}{1} & Rep-tile: & $\set{1,2}$ & $\set{1,3}$ & $\set{2,3}$ \\
\cline{2-5}
        & Encoding:\ \ \ \ & 1 & 2 & 3 \\
\Xhline{2pt}
\multirow{2}{*}{2} & Rep-tile: & $\set{2}$ & $\set{3}$ & \\
\cline{2-5}
        & Encoding: & 1 & 2 & \\
\hline
\end{tabular}
\caption{\small\bf Bla bla}
\label{tab:SMuTexEncoding}
\end{table}

\end{document}

If you want to have coloured rules, it's more complelex: the \arrayrulecolor command (from the colortbl package) doesn't work. A workaround consistsin using the hhline package, setting \arrayrulewidth to 0pt and colouring the interrule space of a double rule. Demo:

\begin{table}[!htb]
\centering
\small\arrayrulecolor{red}
\begin{tabular}{| Sc | >{\itshape}l | Sc c c |}
\hline
{\bfseries HD Level} & & & & \\
\Xhline{2pt}
\multirow{2}{*}{1} & Rep-tile: & $\set{1,2}$ & $\set{1,3}$ & $\set{2,3}$ \\
\cline{2-5}
        & Encoding:\ \ \ \ & 1 & 2 & 3 \\
\Xhline{2pt}
\multirow{2}{*}{2} & Rep-tile: & $\set{2}$ & $\set{3}$ & \\
\cline{2-5}
        & Encoding: & 1 & 2 & \\
\hhline{>{\arrayrulewidth = 0pt\doublerulesep = 2pt \doublerulesepcolor{red}}=====}
\end{tabular}
\caption{\small\bf Bla bla}
\label{tab:SMuTexEncoding}
\end{table}