Does the number of references influence reviewers decision?

There won't be any hard minimum or maximum number of references that a reviewer will be looking for. In my particular field (engineering/robotics) and in my experience, 20 - 40 references seem to be common, though twice or half that will not necessarily raise any eyebrows. The most important factors are that the author is covering previous work sufficiently (covering keystone papers in their area of research, as well as recent similar work) while well establishing their current contribution, and that any included citations are relevant. Too many or too few papers beyond a reviewer's general expectation in their field will probably prompt a closer examination that these are upheld, but won't necessarily lead them to immediately conclude the work is incorrect. The correctness of your paper is an almost completely separate issue that should be recognizable by a reviewer who works in your field based on your paper content.

As a reviewer, rejections I've given due to lack references were because of specific missing references that, if considered, effectively invalidated the author's claim of novelty or reduced their stated contribution significantly.

"Number" is not an issue, it's "currency", for sure, and possibly (in my world, math) recognition of prior art/history. Certainly there's no algorithm that rejects an ms based on number of references. Rather, when I am asked to referee/review/offer-an-opinion-on an essay, if it expresses (explicitly or implicitly) ignorance of standard background, and/or ignorance of current events, then I have serious doubts, from the outset, about whether-or-not the author has done anything useful.