Character Table From Presentation
Short answer:
Multiplication by one-dimensionals will not be sufficient. Looking at the central quotient will not be sufficient.
Finding one dimensional representations of a finitely presented group is easy: every such representation is a representation of the abelianization. In you case: $$\begin{align} G/[G,G] &\cong \langle x, y : x^8 = y^2 = 1, x = yxy^{-1} = x^3 \rangle \\ &= \langle x, y : x^2 = y^2 = 1, xy = yx \rangle \\ &\cong C_2 \times C_2\end{align}$$
Hence it has 4 one-dimensional representations which take x to ±1 and y to ±1.
Since we happen to know G has order 16, this leaves 12 to be written as a sum of squares of divisors of 16, so there are three two-dimensional representations.
One optimistic way to proceed is to think $H=\langle x \rangle$ is abelian, normal, and the conjugation action (and transversal) are written right in the presentation. This means (1) I know all of the representations of H and (2) I can easily induce a representation of H to G. Luckily all irreducible representations of H are one-dimensional, and inducing to G will double their dimension.
In particular, take the representation of H which sends x to ζ, a primitive eighth root of unity. The induced representation sends x to $\left(\begin{smallmatrix} \zeta & 0 \\ 0 & \zeta^3 \end{smallmatrix}\right)$ and y to $\left(\begin{smallmatrix} 0 & 1 \\ 1 & 0 \end{smallmatrix}\right)$, which is clearly irreducible (the only eigenspaces of x are not eigenspaces of y).
One can multiply this representation by $x \mapsto -1$, $y \mapsto \pm1$ to get a second irreducible representation of degree 2.
Neither of these two representations are representations of $G/\langle x^4\rangle$, because they are faithful.
However, the third irreducible twoo-dimensional representation could be found from the quotient group. I would just induce it from $x \mapsto \zeta^2 = i$ to get $x \mapsto \left(\begin{smallmatrix} i & 0 \\ 0 & -i \end{smallmatrix}\right)$ and $y\mapsto \left(\begin{smallmatrix}0&1\\1&0\end{smallmatrix}\right)$. This irreducible representation is unchanged (up to isomorphism) by multiplication by any of the one-dimensional representations.
You asked about generality: the group you are given is incredibly special. It is called a quasi-dihedral group. Having a prime index cyclic normal subgroup is very, very special, and I suspect you are expected to recognize this property and use it.
If your presentation exhibits a chief series with abelian factors, you can use similar techniques to find the character table and representations. In general though, a presentation provides almost no algorithmic information. I recommend Pahlings–Lux's textbook if you are interested in methodically finding character tables in realistic and difficult situations.
If you're given a presentation, you can compute the abelianization of $G$. Every $1$-dimensional representation factors through the abelianization, and conversely every irreducible representation of the abelianization gives an irreducible representation of $G$, so you get precisely all of the $1$-dimensional characters this way. Here the abelianization is
$$G/[G, G] \cong \langle x, y | x^8 = y^2 = 1, x = x^3 \rangle \cong C_2 \times C_2$$
so the four $1$-dimensional characters take value $\pm 1$ on $x$ and $\pm 1$ on $y$. Note that this requires no guesswork, and in particular doesn't require you to fiddle around with normal subgroups; it's much cleaner to fiddle around with quotients.
The remaining three characters are, as you say, necessarily $2$-dimensional. (In general when doing this step, keep in mind that dimensions divide $|G|$.) Probably the best general procedure for finding irreducible characters of dimension greater than $1$ is induction. It's a good idea to induce from subgroups of small index, and also probably a good idea to induce from subgroups whose irreducible characters have low dimension (e.g. abelian subgroups). As others have noted, here you have an abelian subgroup of index $2$...!
Other general techniques to keep in mind:
- Tensoring is good, but so is taking the symmetric or exterior square.
- A Galois conjugate of a character is still a character (exercise; or see this math.SE question).
- The number of self-inverse conjugacy classes is equal to the number of self-dual irreducible characters (exercise).
- If $g \in G$ has order $d$ and is conjugate to $g^k \in G$, then $\chi(g)$ always lies in the subfield of $\mathbb{Q}(\zeta_d)$ ($\zeta_d$ a primitive $d^{th}$ root of unity) fixed by $\zeta_d \mapsto \zeta_d^k$ (exercise). In particular, if $g$ is conjugate to $g^k$ for all $(d, k) = 1$, then $\chi(d)$ is always (rational, hence) an integer.
You could also proceed like this, once you have (correctly) established that $G$ has four irreducible characters of degree 1 (linear characters). Note that $x^{2}$ is in $G^{\prime}$, the derived group of $G$, since $x^{-1}y^{-1}xy = x^{2}.$ Hence $x^{2}$ is in the kernel of each linear character of $G$, so $x^{4}$ is certainly in all these kernels. But there must be some irreducible character $\chi$ of $G$ whose kernel does not contain $x^{4}.$ Let $\sigma$ be a representation affording $\chi$, so you have established already that $\sigma$ must be two-dimensional. Now $\sigma(x)$ must have order $8$, since $\sigma(x^{4}) \neq I_{2 \times 2}$, by the way we chose $\chi$.
What can the eigenvalues of $\sigma(x)$ be? At least one, say $\omega$, is a primitive
$8$-th root of unity. But then $\omega^{3}$ is also an eigenvalue of $\sigma(x)$, since
if $\sigma(x).v = \omega v$, then $\sigma(y^{-1}xy).\sigma(y^{-1}).v = \omega^{3}\sigma(y^{-1})v$.
This tells us that $\chi(x) = \omega + \omega^{3}$ (and, more generally, tells us that
$\chi(x^{r}) = \omega^{r} + \omega^{3r}$ for $ 0 \leq r \leq 7).$ You could finish by noting that $\sum_{j=0}^{7} |\chi(x^j)|^{2} = 16,$ so that $\chi$ must vanish outside $\langle x \rangle$, but there are several other ways to use the orthogonality relations
to deduce this. Note that this really gives us two different irreducible characters, since we could have used ${\bar \omega}$ instead of $\omega$. However, this procedure
is rather specific to this particular group.
Here are a couple of well-known useful general facts. If $G$ is a finite non-Abelian $p$-group, then the number of distinct linear characters of $G$ is divisible by $p^{2}.$ This is because $p^{2}$ divides $|G|$, and divides $\chi(1)^{2}$ for each non-linear irreducible character of $G$. Also, if $G$ is a finite $p$-group, and $\chi$ is an irreducible character of $G$, then $\chi(1)^2$ divides $[G:Z(G)].$ (Since both of these are powers of $p$, it suffices to prove that $\chi(1)^2 \leq [G:Z(G)].$ But $$|G| = \sum_{g \in G}|\chi(g)^2| \geq \sum_{z \in Z(G)} |\chi(z)|^2 = |Z(G)|\chi(1)^2).$$