How does a heatsink on top of a CPU, which is hot, cool down your CPU?

"I expect a heatsink to always be cold"

The purpose of the heat sink is to transfer heat. The rate of heat transfer depends in a complicated way on (a) the temperature difference between heat sink and air, (b) the exposed surface area, and (c) the air speed.

The heat sink is at its most effective if it is at the same temperature as the CPU. This is because this gives the maximum temperature difference between heat sink and air. This is why heat sinks are often made of copper: copper conducts heat well.

Let us suppose that the CPU is producing heat $Q$ (Watts) and that the heat sink transfer heat at a rate linear with the temperature difference:

$$ Q = K_\text{eff} (T_\text{CPU} - T_\text{air})$$

where $K_\text{eff}$ is the effective heat transfer coefficient. Solving for the CPU temperature:

$$ T_\text{CPU} = T_\text{air} + Q / K_\text{eff}$$

So, the CPU temperature is lowest when $K_\text{eff}$ is highest and $K_\text{eff}$ is highest when the heat sink, due it to having high thermal conductivity, is at nearly the same temperature as the CPU.

In general, a heatsink should feel hot, if it's doing its job right. If it feels hot, that means it's transferring energy to your hand, which means it's transferring energy away from the CPU.

This also holds for other cooling things. The cooling coils on a fridge or an old A/C should feel hot for the same reason: the heat you're taking away has to be dumped somewhere. If the cooling coils themselves felt cold, your fridge would be working in reverse; it'd be heating up your food.

The goal of the heat sink is to improve heat flux from CPU to surrounding air. Heat flow is a function of the area of contact and the temperature difference. By adding a component that increases the area, you increase one factor; but it's worth remembering that the heat sink will have to have a temperature between that of the CPU and the air: if it's at the same temperature as the CPU, no heat flows from the CPU to the heat sink; it it's at the same temperature as the air, no heat flows to the air.

In practice, it's easy to make the thermal resistance between the heat sink and the CPU packaging quite low (note - the chip is typically quite a bit hotter than the package, and it's only the latter that touches the heat sink). So with the conductivity between CPU and heat sink being better than conductivity from heat sink to air, the heat sink will tend to a temperature close to that of the CPU.