How to create vertical and horizontal dotted lines in a matrix?

You can adjust the dash and gap of the \hdashline by modifying the lengths \dashlinedash and \dashlinegap:

\documentclass{article}
\usepackage{tabularx}
\usepackage{arydshln,leftidx,mathtools}

\setlength{\dashlinedash}{.4pt}
\setlength{\dashlinegap}{.8pt}

\begin{document}
\[
  \leftidx{_{m-r}^{\phantom{m-r}\llap{$\scriptstyle r$}}}{\left( \begin{array}{c:c}
    \smash{\overset{r}{B}} & \mathclap{\smash{\overset{n-r}{C}}} \\
    \hdashline
    D & E
  \end{array} \right)}{}
\]

\noindent
\begin{tabularx}{\linewidth}{X:X}
  \hdashline
  ABC & DEF \\
  \hdashline
\end{tabularx}
\end{document}

The default for both of the lengths is 4pt.

Since revtex4 seems to be incompatible with the arydshln package, here's an option using TikZ:

\documentclass[aps,prl,twocolumn,nofootinbib,superscriptaddress,floatfix]{revtex4}
\usepackage{tikz}
\usetikzlibrary{matrix}

\begin{document}

\[
\Biggl(\mkern-5mu
\begin{tikzpicture}[baseline=-.65ex]
\matrix[
  matrix of math nodes,
  column sep=1ex,
] (m)
{
B & C \\
D & E \\
};
\draw[dotted]
  ([xshift=0.5ex]m-1-1.north east) -- ([xshift=0.5ex]m-2-1.south east);
\draw[dotted]
  (m-1-1.south west) -- (m-1-2.south east);
\node[above,text depth=1pt] at (m-1-1.north) {$\scriptstyle r$};  
\node[above,text depth=1pt] at (m-1-2.north) {$\scriptstyle n-r$};
\node[left,overlay] at ([xshift=-1.2ex]m-1-1.west) {$\scriptstyle r$};
\node[left,overlay] at ([xshift=-1.2ex]m-2-1.west) {$\scriptstyle n-r$};
\end{tikzpicture}\mkern-5mu
\Biggr)
\]

\end{document}

This uses a stacking approach. This answer is helping me to think of ways to automate dotted/dashed lines as part of stacks, which in this MWE, had to be done very manually.

\documentclass{article}
\usepackage[usestackEOL]{stackengine}
\usepackage{graphicx}
\stackMath
% DASHED LINE OF SPECIFIED LENGTH
% From morsburg at http://tex.stackexchange.com/questions/12537/
% how-can-i-make-a-horizontal-dashed-line/12553#12553
\def\dashfill{\cleaders\hbox to .5em{\rule{.4ex}{.4pt}}\hfill}
\newcommand\dashline[1]{\hbox to #1{\dashfill\hfil}}
\newlength\tmplen
\begin{document}
\[
\strutshortanchors{F}\def\stackalignment{r}\setstackgap{S}{5pt}
\stackanchor[15pt]{{}^r}{{}_{m-r}}
\def\stackalignment{c}
\stackon[1pt]{
  \left(\raisebox{2pt}{%
  \savestack{\num}{$~B~\,$}\tmplen=\wd\numcontent\relax%
  \stackunder{\stackon{\dashline{\tmplen}}{\num}}{D\;}%
  \rotatebox{90}{\kern-1.1\baselineskip\dashline{2.5\baselineskip}}%
  \savestack{\num}{$\,~C~$}\tmplen=\wd\numcontent\relax%
  \stackunder{\stackon{\dashline{\tmplen}}{\num}}{\,E}}%
  \right)
}{%
  ~~{}_r~\,~_{n-r}%
}
\]
\end{document}