Formula result is data type (Object), incompatible with expected data type (Text)

I'm not sure why the problem is occurring, but following approach might solve your issue permanently.

First, no need to put a condition like Opportunity.Product2.Id is not null, because, Opportunity line item doesn't exist without a product. So, it should be No criteria—just execute the actions!

Secondly, for update action, use formula of return type Text and refer the product family.