How can friction do no work in case of pure rolling?

In a scenario of pure rolling of a rigid wheel on a flat plane, you don't need any friction. Once the wheel is rolling, it will continue to do so, even if the friction coefficient becomes zero. If this were not the case, you'd be violating conservation of angular momentum. There is no force, no torque and therefore no work done.

The more interesting case is that where the object is rolling under an external force, say down an inclined plane. See the diagram below

Now, you can analyze it in two ways. One is similar to Farcher's answer where the point of contact moves perpendicular to the frictional force and hence, there is no work done. But you were interested in it from the point of view of torques (where we consider the whole wheel, not just the point of contact) so let's do that.

Friction does two things to the wheel as a whole. It does negative work when you look at the linear motion of the wheel. Indeed,

$$W_1 = -f.dS,$$

where $f$ is the force of friction and the wheel has moved a linear distance $dS$. Next, the friction provides a torque about the center of the wheel and the wheel has angular displacement. Hence, it does positive rotational work i.e.

$$W_2 = \tau. d\alpha,$$

where $\tau$ is the torque and $d\alpha$ is the angular displacement of the wheel. But note that $\tau = fR$ and $R d\alpha = dS$. Hence $W_2 = f.dS$ and you get

$$W_{tot} = W_1 + W_2 = 0$$

The pure rolling here is under a constant external force on the object so friction does act.

As an example consider an object rolling, without slipping, down an inclined plane.
There will be a frictional force $F$ acting, at the point of contact between the object and the inclined plane, upwards and parallel to the slope as an object is rolling down an inclined plane as well as the weight of the object $mg$ and the normal reaction $N$ both of which act through the centre of mass of the rolling object.

If you consider the motion of a point on the rolling object its path is a cycloid.
The image below is taken from the Wikipedia article on the cycloid and I need you to imagine that the black line at the bottom is inclined to the horizontal and thus representing the inclined plane.

At the point of contact the cycloid is perpendicular to the inclined plane ie along the line $XC$ where $X$ is the point of contact and $C$ the centre of mass of the rolling object.

This means that the direction of motion of the point of contact is perpendicular to the frictional force and so the frictional force does no work on the object.

It terms of the torque about the centre of mass $C$ because the motion of the point of contact is towards the centre of mass of the object there is no rotation of the torque about the centre of mass and so no work is done by the frictional torque.

Yep you're right. Consider the case of a cylinder rolling down a hill, starting from rest. Friction does provide a torque and definitely does work on the cylinder. However, in the case of pure rolling motion, i.e no slipping, the work done by friction = gain in rotational KE. So we can say that the change in gpe = change in total KE.

It therefore appears based on that conservation statement that the frictional force does no work.

Edit: Note delta KE means the change in linear KE, i.e $mv^2/2$