Should I cite a really bad research paper published in my small research area?
Yes you should cite it because:
- You know about that paper
- You read it in the context of preparing your work. Finding gaps and flaws in previous works is a major part of designing and refining your own research
- By your own judgement it’s highly relevant.
Explain your rebuttal of their claims or method using proper scientific arguments. If I were you I would refrain from using words like “hand waving” or “cranks” in a scientific context.
Whether or not you are the first to discover something isn’t really something you can control. It’s possible that they actually did find something of value but were bad at communicating it. From what you say there seem to be still time for you to be the first to get it right but you won’t achieve that by ignoring previous work.
The paper must be acknowledged as proper research is based on published literature. You don’t really need to provide some passionate negative critique of the paper like you’ve done here. You can merely point out main reasons why it could not be included in your analysis directly. However I am sure many of the points you raised here should be fairly straightforward to be clarified by the first author (e.g. some relevant passage in seemingly alien language). It would be better to state you’ve tried to clarify key points you couldn’t understand on your own. You never know: perhaps not everyone agrees in that the previous paper is that unclear.
If you can find a counterexample to one of the paper's proofs without too much work, then this would be critique enough, I think. You are lucky to be working in mathematics, where the falseness of a claim can be clearly shown.
I would not spend too much time on providing a critique. Ignoring the paper is what most people would do in this situation, but is unethical. Some journals' Guidelines for Authors explicitly require citing all relevant literature.