Dealing with uncertainty when reviewing a paper

Yes, it is fair to raise concerns when you don't understand a paper. It is the authors' job to explain in enough detail what they did. Especially if the method used is uncommon, then it needs to be justified. That justification needs to be done in a way that it is understood by the intended audience, not just the specialist within that audience.

Tell the authors in as much detail as possible why you think their method does not feel right. Maybe the method is problematic, and they will appreciate that feedback. Maybe they did not explain their method well enough, and they can use that feedback to pinpoint what was not clear.

If possible, ask someone you know who is an expert on the technique for advice. Being able to give a more informed response, or drop any objection if there is no basis, would be preferable. Of course you would need to get input while maintaining confidentiality, but normally I wouldn't expect this to be a problem. If you can describe it in the hypothetical, without reference to the specifics of the research, it should be fine.

There is no harm in admitting limits to your knowledge, or giving criticism that you are not 100% confident in. However, I don't think saying "I have a vague sense this doesn't seem right" is likely to be very useful. If you can't at least communicate a concrete reason for your concern, I don't think an objection is very fair to the authors; nor is it likely to be productive. So if you aren't able to further clarify what you find wrong, I would not raise this as a criticism of what they did. You don't really have enough to object to that.

However, it's fair game to raise the issue that they failed to explain/justify what they did (I 100% agree with Maarten Buis's answer on this point).

This happens more frequently than you think. Editors are sometimes under a lot of pressure to secure reviewers and sometimes they settle for non-experts.

In such cases it is more than fair to state your concerns of the paper, while writing to the editor (private comment) explaining the situation by stating that you are not expert, but you tried to understand the paper but you feel that something is wrong for the reasons outlined in the review. This, will force the authors answer one way or another

It is not possible to undermine like that your review. Editors want honest opinions. In any case, such techniques should be clear to a wider context, not only on the few experts on their fields, so if you feel something is unclear, please make service to the authors and the editors and state it!