tikz intersection of named path

With a real intersection you can write (I changed the coordinates of D)

\begin{tikzpicture}
\tikzset{dot/.style={circle,inner sep=1pt,fill,label={\tiny #1},name=#1}}
\node[dot=A] (A) at (0,0) {};
\node[dot=B] (B) at (1,0) {};
\node[dot=C] (C) at (0,1) {};
\node[dot=D] (D) at (1,-0.5) {};

\draw [name path=P1] (A) -- (B);
\draw [name path=P2] (C) -- (D);

\path [name intersections={of=P1 and P2,by=E}];
\node[dot=E]  at (E) {};
\draw[dotted] (B)--(E)--(D);
\end{tikzpicture}

else

\begin{tikzpicture}
\tikzset{
    dot/.style={circle,inner sep=1pt,fill,label={\tiny #1},name=#1}}
\node[dot=A] (A) at (0,0) {};
\node[dot=B] (B) at (1,0) {};
\node[dot=C] (C) at (0,1) {};
\node[dot=D] (D) at (1,0.5) {};

\draw [name path=A--B] (A) -- (B);
\draw [name path=C--D] (C) -- (D);

\coordinate (E) at (intersection of A--B and C--D);
\node[dot=E]  at (E) {};
\draw[dotted] (B)--(E)--(D);
\end{tikzpicture}

There must be a real intersection between path P1 and path P2. You could use something like

\path[name path=P1,overlay] (A) -- (B)--([turn]0:5cm);
\path[name path=P2,overlay] (C) -- (D)--([turn]0:5cm);

to enlarge the paths without enlarging the bounding box of the picture.

Then you can use

\path [name intersections={of=P1 and P2,by={CS}}];

to define a coordinate CS at the intersection of the enlarged paths P1 and P2.

Code:

\documentclass[border=1mm]{standalone}
\usepackage{tikz}
\usetikzlibrary{intersections}

\begin{document}
\begin{tikzpicture}
\tikzset{
    dot/.style={circle,inner sep=1pt,fill,label={\tiny #1},name=#1},
}

\node[dot=A] (A) at (0,0) {};
\node[dot=B] (B) at (1,0) {};
\node[dot=C] (C) at (0,1) {};
\node[dot=D] (D) at (1,0.5) {};

\path[name path=P1,overlay] (A) -- (B)--([turn]0:5cm);
\path[name path=P2,overlay] (C) -- (D)--([turn]0:5cm);

\path [name intersections={of=P1 and P2,by={CS}}];

\draw (A) -- (B);
\draw (C) -- (D);
\node[dot=E] (E) at (CS) {};
\draw[dotted] (B)--(E)--(D);
\end{tikzpicture}
\end{document}

Notation intersection of A--B and C--D is old and doesn't appear in TikZ 3.0 documentation.

New intersections library is documented in section "13.3.2 Intersections of Arbitrary Paths" where is explained how to use named paths.

In your particular case the problem is that paths P1 and P2 doesn't intersect because they are too short. You can make them longer with calc library help and syntax explained in "13.15.3 The Syntax of Partway Modifiers" which defines points on line between two coordinates but not necessarily in between.

\documentclass[border=1mm]{standalone}
\usepackage{tikz}
\usetikzlibrary{intersections,calc}

\begin{document}
\begin{tikzpicture}
\tikzset{
    dot/.style={circle,inner sep=1pt,fill,label={\tiny #1},name=#1},
}

\node[dot=A] (A) at (0,0) {};
\node[dot=B] (B) at (1,0) {};
\node[dot=C] (C) at (0,1) {};
\node[dot=D] (D) at (1,0.5) {};

\path[name path=P1] (A) -- ($(A)!3!(B)$);
\path[name path=P2] (C) -- ($(C)!3!(D)$);

\path [name intersections={of=P1 and P2, by={E}}];
\draw (A) -- (B);
\draw (C) -- (D);
\node[dot=E] at (E) {};
\draw[dotted] (B)--(E)--(D);
\end{tikzpicture}
\end{document}