How to suggest reviewers for a journal?

Being editor of a journal where authors can provide preferred and non-preferred reviewers, I can provide some "inside" thoughts on the subject based on what has happened in "my" journal. Note that it is possible to suggest names for review but also provide names which are not preferred. The latter can be because of a scientific disagreement, personal issues or whatever. Such suggestions appear but not often and we usually follow the suggestions (not that we have to!).

When it comes to the preferred or suggested reviewers, I have been tempted to use such reviewers on occasion when it has been hard to identify reviewers directly. Sometimes because the topic is local and where it would make sense to have local input. In these cases, I cannot remember a single reasonable review that has come out of such reviewers. This can be for several reasons but most often the review is a close colleague who might have an incentive to help the author. In some cases the preferred names have been very senior scientists who, I am afraid, has lost touch with the subject and provide poor and in some cases almost non-existent reviews. Out of all immediate "Accept" review recommendations I get, the vast majority have come from these reviewers. So, I not longer trust these names and avoid them at all costs unless I personally know or know of the reviewer and his or her good reputation.

In addition to what I just describe, I also must state that it is often the weakest manuscripts that have listed several suggestions. This can be identified by the disparate review results, sometimes one accept (by the suggested reviewer) and one reject.

Now, in principle, there is nothing wrong with suggesting reviewers, I have done so myself when being requested. I have then as a principle gone for established and well renowned names in the community. The problem lies in suggesting names for a purpose other than to get a fair and objective review.

It is clear that the system can and is abused and since I became Editor-in-Chief, I have come to rely less and less on these suggestions and now mostly look upon them with suspicion and make selections from my own understanding of the field and investigations into the subject literature. The best suggestion, I can provide is to not avoid mentioning names but pick names that in your opinion can provide good constructive critique on your work (and not just favorable). A note on why you have selected names as preferred or non-preferred would greatly help as well since it puts your choice in a perspective.

I was asked to do that several times by an editor after being told (s)he couldn't find referees for my submission (to the point that I now spontaneously tell the editor upon submission that I can suggest referees if need be), but I don't know of any journal (or conference) for which this is required.

Anyway, you don't need to know experts personally: you are suggesting referees, not forcing them on the editor (or your work on them), and whether or not you actually know them should be irrelevant (it's even better if you don't). Read your bibliography, see which authors come up most often, or whose work form the most important basis for your submission, or who would be the most interested in reading it based on their own work, and I'm sure you'll have plenty of names to suggest.

Some journals explicitly ask about suggestions for reviewers with a submission, some will consider any suggestions that you make in the cover letter, and others (probably) will just ignore any such suggestion.

There are in fact scientific studies about the comparison between reviewers suggested by the authors, and those selected by the editor, for example this article in BMC Medicine. The overall conclusion seems to be that reviewers suggested by authors provide reviews of equal quality than those selected by the editor. While they are more likely to suggest acceptance in the initial review, at later stages these suggestions seem to equalize.

As an author, you should have a high interest in getting over that initial review, and if you do it well, suggesting reviewers is a very good opportunity for that. I'd always suggest to make use of such an opportunity, since you probably can judge best which potential reviewers will look favorably at your paper. And that's of course what you want.

If you know an expert personally, that's usually a good option. It has to be handled with care though. When you're too close to a suggested reviewer, the editor will give significantly less weight to the recommendation of that reviewer if he knows about personal ties. But if you go to conferences and talk to people about your research, you could suggest them as reviewers afterwards if they have similar interests. Or look at your reference list, as suggested in the answer by Anthony Labarre.