If research design, data analysis, and results only are good, where is the line between suggesting major revision and rejection?
I would recommend emailing the journal editor to ask for some of these clarifications. There might be certain unwritten policies endemic to that particular journal.
In general, you can ask for multiple rounds of revisions (major or minor) until you are satisfied. Peer review is supposed to be an iterative conversation. However, as I mentioned earlier, there could be certain journal policies regarding the number of revisions which you might get addressed if you talk to the editor.
As a reviewer, if you feel that the article should be rejected, you are free to make that claim. However, personally, I always give at least one chance to a paper (however horrible it is) to redress my comments.
Journal instructions vary so exactly what can be done may also vary. But, as a reviewer you re "only" suggesting a verdict based on your review. An editor will make the final decision based on your and one or more reviews.
One question you can ask is whether the revisions you suggest can easily be accommodated within some reasonable time frame. As an example, I edit a journal where we allow six weeks for "major revisions" (three for minor). Unfortunately, this time frame may not be obvious to anyone but the editor. But, you probably get a sense if the paper contains enough good science to be worth waiting for or if a new submission at some later time would be better.
Usually, you provide a report that is passed on to the authors in some way but you also provide a confidential note to the editor. I suggest that you detail your qualms about the paper and try to explain your thinking about the potential for this paper and whether it may benefit from either rejection and possibly later resubmission or by taking it down the "major revision" route. You can then provide you sense of how much time might be required for the changes so that the editor obtains a sure footing for the decisions. It is always good to receive a little of "pros and cons" from the reviewer.
In the end the editor will make the choice and it will be based on more views than just yours. Providing the best substantiated view you can will most likely be welcomed by the editor
It fails to clearly state the research questions. No all the hypotheses are well supported by the literature review. It omits several key studies in the literature review. It often makes strong claims without a proper reference. It often fails to quantify stuff, falling to the tedious "a lot", "many", "few", "circa", etc. It is quite mysterious on certain aspects of the research design-e.g., a control group is suddenly mentioned without really clarifying what constituted the control group, and the paper reports that the experiment is not even a controlled experiment.
I strongly recommend including this paragraph verbatim in your referee report.
This description is not obviously consistent with your claim that the paper "appears to report a well-conducted empirical study, with proper data analysis". Is it actually possible, even in principle, to have a well-conducted study without a crisply stated research question or a clearly-defined control group, or a proper data analysis that uses "a lot" and "circa"?
These are very basic flaws, independent of the authors' first language. If they were spread throughout the paper, they would be sufficient to recommend rejection. If these problems appear only in the introduction (and later sections clarify the research questions, experimental design, and so on), then they may not warrant rejection, but they certainly require major revisions.
The discrepancy between the initial sections and the body of the paper suggests that a student wrote the former, and the PI wrote the latter. At least, I hope that's what happened. It's appropriate to be gentle (but firm) with the student, but a bit less forgiving of the PI for not advising the student better.
In case of suggesting a major revision, what if the authors still fail to provide a clearly written and organized manuscript in a second round? Could I still request for major revisions?
Yes.
If the authors' revision does not sufficiently address the concerns in your initial report — regardless of whether your concerns are about presentation or content — your response to the editor should be "The authors' revision does not sufficiently address the concerns in my initial report; therefore, I cannot recommend acceptance."