What makes a research paper low-quality?

Kind of a broad question, but I'll play along.

It is easier to say what makes a paper high quality: I learn something new and interesting by reading it. A twinge of jealousy -- as in, "Gosh, I wish I had thought of / done this" -- is a good sign there. The absence of all this makes the quality of the paper not too high.

There are some papers that don't teach me anything new and interesting that I nevertheless feel are of medium (publishable) quality: sometimes I don't really understand the value of a result but can't claim (or can't justify a claim) that the results will not be of value to someone else or someone in the future. Sometimes the results are known to me in some form but don't appear in the literature. Sometimes the results are novel when viewed from the knowledge base and comfort zone of one subfield but less so from another (or perhaps from the vantage point of a true expert -- in a field where true expertise is rare), and these can be worth publishing.

Probably what remains is low quality. I am inclined to object a bit to the framing of the question in terms of "obvious signs": as a referee, I aspire to read a paper carefully enough to be confident in my evaluation of its quality. Obvious signs may inform my preliminary impression, but I will read further to confirm. But here are some things that would make me evaluate a paper as of low quality: as I go down the list, the quality decreases:

  • The results of the paper already exist in the literature. Usually this means the author was not aware of them.
  • The results of the paper already exist in the literature, in a better form than in the author's paper. Thus publishing the paper could actually be a step backwards.
  • Same as the above, but the current paper omits a key technique or perspective that is part of the contemporary understanding of the topic. For instance, there are some expository papers, and these do not need to contain new results or even the sharpest possible form of the old results it exposes, but the author needs to have at least the level of mastery of the topics being exposed as an advanced student in the area, or the exposition could have a negative effect.
  • The results are faulty, either in ways that are not easily corrigible, or are easily corrigible but when corrected yield absolutely nothing new.
  • The results look fishy, but the exposition is so obscure that it is a chore to tell right from wrong from basically-right-but-highly-garbled from totally absent.
  • The author cannot or will not make a clear distinction between what is attained in the paper and what is desired to be attained. For me and many others in my field (mathematics), this is certainly the worst. I saw an especially good example of this recently: a paper posted this month on the arxiv begins (immediately following an abstract which is at turns obscure and vacuous) by stating "Registration contains colored markers:" A red dot stands for "a fact which is not proven at present or an assumption". A yellow dot stands for "the statement which requires additional attention". A green dot stands for "statement which is proved earlier or clearly undestandable [sic]". And then immediately following that, still on page 1, there is a "THEOREM", which is a statement of the Riemann Hypothesis (for those who don't know, this is arguably the single most important unsolved problem in all of mathematics; inarguably solving it will get you $1 million)...marked with a red dot. Then we have "PROOF", followed by 28 more pages of colored-dotted mathematics, which I spent a minute flipping through out of sheer morbid curiosity. But if I were a referee then, rather exceptionally, this first page would have been all the sign I needed to be confident of the low quality of the paper.

I tend to look for three things:

  1. Clarity : can I easily understand the paper? Are the language / logic / symbols coherent? Complex ideas can be hard to grok, but are worthwhile.
  2. Correctness : are there any mistakes in the presentation, mathematics or assumptions?
  3. Contribution : does this paper present something new?

If any of these three is lacking, then I usually mark the paper down (reject, or accept with major modifications).

Of these, the last (contribution) is the most important.

A low quality paper will miss on two of the three.

If a paper misses on 1 and 2, then I generally can't see whether 3 is true or not.


Mismatch between advertising and content

The unifying theme with low quality papers is a large difference between what they claim they've achieved and what they show they've achieved. Generally, in such a paper the abstract will look exactly like an abstract of a strong paper, but the advertised promise is not fulfilled by the paper content.

A quick and obvious way to look for that is the following - read the abstract to identify the most significant claims; then read the conclusions to see if the claims are still there (or the vague description in abstract implies something much more valuable than actually concluded), then skim the content chapters to see what evidence they provide that the conclusions are valid to their full extent. Good papers will contain exactly what it says in the description, poor papers will have done something small, but puffed it up to seem valuable.