Example where angular momentum and angular velocity are not parallel

In the basic discussion of angular momentum where something is rotating around a fixed symmetrical axis

$\vec{L}=\vec{r}\times\vec{p}$

reduces to

$\vec{L}=I*\vec{\omega}$

Like in this animation where each vector is colored appropriately:

However angular velocity and angular momentum can have different directions in two cases: If the axis of rotation is not symmetric or the axis of rotation is moving.

Here's an example:

You can see that $\vec{L}=\vec{r}\times\vec{p}$ is not the same direction as $\vec{\omega}$ nor would the simplification $\vec{L}=I*\vec{\omega}$ be correct.

The position vector $\vec{r}$ is the vector between the reference point and the mass (note these problems are ignoring the mass of the rod), only in simple rotational cases like the first case is it perpendicular to $\vec{\omega}$. In a system of masses for example these vectors to the masses about a reference point can be complex. It is much easier to take the reference point as the center of mass. In each case $\vec{r}$ is the positional vector between your reference point and the mass and their composite angular momentums will superimpose (add) together.

Let's limit the discussion to rigid objects rotating with a fixed $\vec{\omega}$, i.e. fixed axis and constant angular velocity. Here are a couple of observations:

1. $\vec{L}$ rotates with the rigid body. We could glue a little arrow representing $\vec{L}$ to the body, so that the arrow rotates around $\vec{w}$ at the same rate as the body.

2. If $\vec{L}$ is parallel to $\vec{w}$, then it is unaffected by the rotational motion. Otherwise, $\vec{L}$ is changing throughout the rotational motion.

3. If $\vec{L}$ is changing, it means an external torque is acting through the axis.

Given these observations, we can state the following:

$\vec{L}$ is not parallel to $\vec{\omega}$ if and only if a torque is needed to keep an object rotating around a fixed axis.

To get a feel for this, try holding a spinning bicycle wheel in your hands. If the wheel is slightly imbalanced, you will feel an oscillating torque from the axis - this is coming from the changes in $\vec{L}$ as it spins around $\vec{\omega}$. If the wheel is perfectly balanced, then you will feel nothing - $\vec{L}$ is parallel to $\vec{\omega}$ and is fixed throughout the rotation.

Here is a diagram contributed by @user6972 showing exactly how $\vec{L} = \vec{r} \times \vec{p}$ can have a different direction to $\vec{\omega}$:

The answer above gives examples but if you want to know 'HOW' it happens, consider the following equation

$$ L_i = \sum_j I_{ij} \omega_j $$

You can diagonalize the symmetric moment of inertia tensor (by rotating your basis. In that basis you get

$$ L_{i'} = \sum_{j'} \lambda_{j'} \delta_{i'j'} \omega_{j'} = \lambda_{i'} \omega_{i'} $$

and thus you see that in this basis whenever you have two or more components of $\omega$ non-zero along directions with unequal $\lambda$ you cannot have the angular momentum parallel to the angular velocity.