Categories which are not compactly generated

As David says, D(R) is compactly generated. This means Brown representability for COHOMOLOGY is automatically true, but that does NOT mean Brown representability for HOMOLOGY is true, and in fact it is not always true. That is what the Christensen-Keller-Neeman example shows.

Let K(R) be the category of chain complexes over R and chain homotopy classes of chain maps. Then K(Z) is not compactly generated (and K(R) is not either for many R). This is proved in a paper of mine with Christensen.

Quillen model structures for relative homological algebra

One example is the following - suppose that $M$ is a non-compact connected manifold of dimension $\geq 1$. Then the unbounded derived category of chain complexes of sheaves of abelian groups on $M$ has no compact objects other than zero. It is, however, well generated. This can be found in Neeman's paper "On the Derived Category of Sheaves on a Manifold".

Another example is suppose that $R$ is the ring $k\oplus V$ where $k$ is a field, $V$ is an infinite dimensional vector space and the ring structure is the unique one making $V$ a square zero ideal. Then the category $K(Proj\; R)$, the homotopy category of complexes of projective $R$-modules is not compactly generated. Again though it is well generated (even $\aleph_1$-compactly generated). In general this category need not be compactly generated if $R^{op}$ is not coherent.

One could also produce more examples along these lines I imagine by considering for instance the homotopy category of flat modules over a suitable ring.

These last two examples are very natural objects to study - I can provide more details on why if anyone is interested.

These are the most naturally occurring examples I know of off the top of my head.

I'll try and think of a modular representation example if I can although when I think about these things it is normally the stable category for modular reps of a finite group and these are always compactly generated.

Not from modular representation theory, but the following seems like a geometrically natural example:

Neeman has a nice note where he shows that the only compact object, in the derived category of sheaves of abelian groups on a non-compact, connected, positive-dimensional manifold, is the zero object.