How to know whether my submission to a mathematics journal is significant enough?
"Good enough for journal X" is not a mathematical claim. Operationally speaking, it is precisely a matter of collective opinion. Math journals group themselves into rough, approximate equivalence classes (which is also entirely a matter of collective opinion), and you should take this negative referee result as one data point that your paper is not good enough for a journal of that class. If your prior belief was that the paper was roughly equal or better in quality to other papers published in journals of the same class, you should try at least once more at a different journal of about the same level.
A good referee report reveals something about why the referee thinks the quality of the paper is not sufficient, either in absolute or relative terms. This may give you something to improve: either actual mathematical improvements or improvements in the exposition to make referees believe your work is more valuable.
In summary: it doesn't necessarily mean much on its own. Revise if applicable and resubmit. Good luck.
Significance may be a very subjective judgement. Of course, for old, famous problems such as Fermat, any proof would count as significant, but the question becomes much more dodgy when one enters a side alley. As heuristic, ask yourself: is the result solving an old, unsolved problem? Is it connecting two unconnected or loosely parts of the field? Is it a beautiful, unexpected result? Is it generalising existing concepts?
In a comment to @PeteLClark's answer, you state "I want to avoid this report type". We all do, of course. But remember that good journals reject around 2/3 of all submitted papers. You will not be able to avoid having at least some of your papers rejected if you continue to publish, simply based on statistics and the fact that the reviewing process has a random element to it. It's just part of the process. Go with the flow -- take the reviewer comments serious (they are typically written by people with a good amount of experience), address them in your manuscript, find a different journal one rung down the ladder, and resubmit.