Representation theory of $\mathbb{R}$?

In the case of an infinite topological group, you probably want to consider only continuous characters. In the compact case, the Peter-Weyl theorem holds, and the situation is similar to the finite case. If you consider arbitrary representations of $\mathbb{R}$, then you can construct some uninteresting and pathological ones from a Hamel basis. In the abelian case, the characters are more or less determined by Pontryagin duality, assuming local compactness. For the particular case of $\mathbb{R}$, it follows that the continuous characters on $\mathbb{R}$ are all of the form $\chi(t) = e^{2\pi i \xi t}$ for some real $\xi$, with the correspondence induced by the Fourier transform.

For fun, let's drop the assumption of continuity. $\mathbb{R}$ is a $\mathbb{Q}$-vector space, and so (assuming choice) it has a basis, and hence is an infinite direct sum of copies of $\mathbb{Q}$. The representation theory of $\mathbb{Q}$ is surprisingly complicated: explicitly, the $1$-dimensional representations are given by picking nonzero complex numbers $z_n \in \mathbb{C}^{\times}$ such that

$$z_{nm}^m = z_n$$

for all $i, j$, and then taking

$$\mathbb{Q} \ni \frac{p}{q} \mapsto z_q^p \in \mathbb{C}^{\times}.$$

The obvious examples are given by taking $z_n = \exp \frac{2 \pi i \xi}{n}$ for some $\xi \in \mathbb{C}$ but other examples are possible.

Moreover, the representation theory of $\mathbb{Q}$ isn't semisimple; for example,

$$\mathbb{Q} \ni r \mapsto \left[ \begin{array}{cc} 1 & r \\ 0 & 1 \end{array} \right]$$

is a $2$-dimensional representation which is indecomposable but not irreducible. I don't know off the top of my head how to classify these. And then $\mathbb{R}$ is an infinite direct sum of copies of $\mathbb{Q}$ on top of that...

For topological groups like $\mathbb{R}$, the useful objects to study are the continuous representations (or the continuous characters in the case of an abelian group).

It turns out that all continuous characters of $\mathbb{R}$ have the form $\chi(x)=e^{2\pi i\xi x}$ for some $\xi\in\mathbb{R}$. This is generally proved in books that cover Fourier analysis. See for instance Chapter 8 of Folland's Real Analysis.