Vertical dashed line of a custom height

One possibility, is to tackle the problem in the other sense: instead of shortening the dashed line, you can enlarge the delimiters:

\documentclass[a4paper]{article}
\usepackage{arydshln}
\usepackage{mathtools}

\begin{document}

\begin{equation*}
\delimitershortfall=0pt
\setlength{\dashlinegap}{2pt}
\left[\begin{array}{cccc:c}
a_{11} & a_{12} & \cdots & a_{1n} & b_1 \\
a_{21} & a_{22} & \cdots & a_{2n} & b_2 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
a_{n1} & a_{n2} & \cdots & a_{nn} & b_n
\end{array}
\right]
\end{equation*}

\end{document}

And, of course, there's the (overkill?) solution using \tikzmark; this gives you more control for fine tuning:

\documentclass[a4paper]{article}
\usepackage{tikz}
\usepackage{mathtools}

\newcommand\tikzmark[1]{%
  \tikz[remember picture,overlay]\coordinate (#1);}

\begin{document}

\begin{equation*}
\left[\begin{array}{ccccc}
a_{11} & a_{12} & \cdots & a_{1n}\hfill\tikzmark{a} & b_1 \\
a_{21} & a_{22} & \cdots & a_{2n} & b_2 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
a_{n1} & a_{n2} & \cdots & a_{nn}\hfill\tikzmark{b} & b_n
\end{array}
\right]
\end{equation*}

\tikz[remember picture,overlay]
  \draw[dashed,dash pattern={on 4pt off 2pt}] ([xshift=0.5\tabcolsep,yshift=7pt]a.north) -- ([xshift=0.5\tabcolsep,yshift=-2pt]b.south);

\end{document}

It seems to me that you are typesetting matrices here. Note that amsmath provides matrix environments (matrix, pmatrix, bmatrix, etc…), so that you do not need to type your own delimiters around an array. (You would want bmatrix.)

To get augmented matrices, use the code from the question: How does this macro for augmented matrices work?

---

Edit: Included the code for convenience.

\usepackage{arydshln}
\makeatletter
  \renewcommand*\env@matrix[1][*\c@MaxMatrixCols c]{%
    \hskip -\arraycolsep
    \let\@ifnextchar\new@ifnextchar
  \array{#1}}
\makeatother

Put the code in the preamble.

---

You can then use the same syntax as for arrays, but would end up with the code:

\begin{equation*}
\begin{bmatrix}[cccc:c]
a_{11} & a_{12} & \cdots & a_{1n} & b_1 \\
a_{21} & a_{22} & \cdots & a_{2n} & b_2 \\
\vdots & \vdots & \ddots & \vdots & \vdots \\
a_{n1} & a_{n2} & \cdots & a_{nn} & b_n
\end{bmatrix}
\end{equation*}

---

Edit 2: Another interesting link: http://texblog.net/latex-archive/maths/amsmath-matrix/