! Package amsmath Error: \tag not allowed here

You're still using inline math. (La)TeX distinguishes between math that is supposed be written on a line of text, delimited by $ ... $ or \( ... \), and displayed math, which is placed on its own paragraph.

For a single unnumbered, displayed equation you can use \[ ... \], for a numberered equation there is \begin{equation} ... \end{equation}. For sets of equations, or multiline equations, amsmath provides several environment, including align and gather, as well as the starred forms align* and gather* that are unnumbered.

Displayed equations are by default centered, to make them left-aligned add fleqn as an option to amsmath or the document class, e.g. \usepackage[fleqn]{amsmath}.

For more information about amsmath, read the manual. For math typesetting in general, you could take a look at Herbert Voss' Mathmode.

A demonstration with your example:

\documentclass{article}
\usepackage{amsmath}

\begin{document}
Now solve for $E[X]$:
\begin{gather}
E[X] = 1 + E[X] - p \cdot E[X] \\
0 = 1 - p \cdot E[X]\tag{"xyz"} \\
p \cdot E[X] = 1\tag{"xyy"} \\
E[X] = \frac{1}{p}\tag{"xzy"}
\end{gather}

Now solve for $E[X]$:
\begin{align}
E[X] &= 1 + E[X] - p \cdot E[X] \\
0 &= 1 - p \cdot E[X]\tag{"xyz"} \\
p \cdot E[X] &= 1\tag{"xyy"} \\
E[X] &= \frac{1}{p}\tag{"xzy"}
\end{align}
\end{document}

These are still inline equations. You need to use

\[ formula \]

of

\begin{equation}
formula
\end{equation}

to make your formulas display style. Read about it here.