How to interpret this rejection email from Journal of American Math Society? Anything to read between the lines?

There isn't anything between the lines. It's simply a rejection. The editor and reviewers feel that your paper is clearly not strong enough for this journal, and they're telling you so.

Note that JAMS is an extremely selective journal, generally considered one of the top four out of all the many hundreds of journals in mathematics. It is very very hard to get a paper published there. Most of the papers they publish will be major breakthroughs in an area, or solve a famous open problem. The "10%" is actually misleadingly high, because the vast majority of papers are never submitted there in the first place because the authors know they have no chance.

Such journals often do a quick first round of reviews, where the reviewers are only asked if they think the results are potentially important enough to warrant publication in JAMS. Only if they say yes does the paper go on to a full technical review. In this case, it sounds like they said no. You didn't receive a detailed report because they didn't need to do a detailed review of the paper to make a decision. Remember that their job is ultimately to make the decision; it's not their job to give you feedback or help you improve your paper. Sometimes that happens along the way, but when the decision is clear, they won't spend time on a paper that, in their view, can't be made publishable in this journal.

The reviewers can, as in this case, make comments about things they happened to notice as they were doing their quick review. That's why you got the note about the reference. You'll have to read the reference to determine exactly what was meant here; it may be that the reviewer thinks that paper may already contain your results, or at least that they are similar enough that you need to carefully explain the difference. But it sounds like the reason for rejection was the overall significance of your paper, and this was just something they noted in passing.

How to move forward: submit to a different journal. If you want feedback on your paper first, you'll have to get it from someone else, e.g. a mentor, PhD advisor, collaborator or fellow researcher. Such a person could also help you get some ideas of journals for which your paper might be better suited - where it would have a better chance of acceptance or at least a full review.

Top journals, and JAMS is at the very top (most mathematicians probably will not have ever seriously considered submitting there), typically have 2 first passes before a proper peer-review (referee): (1) editorial consideration and (2) "quick" opinions from experts. Which is to say, there are 2 common ways for a paper to be rejected before it is sent for a full peer-review: (i) the editor decides from the title/abstract/info that it is not appropriate, or (ii) the editor solicits experts for an opinion of whether it is worth the effort of a full peer-review.

Based on your letter, and since you didn't receive a referee report, I would say the most likely scenario is (ii), and this is not surprising because unless you have done something really spectacular (I don't know that you haven't, but playing the odds...), you shouldn't bother submitting to JAMS, Annals, etc. (It can also happen that a referee who is asked to do a normal review, just sends back a few lines in the case of recommended rejection, in which case a full report would not be passed on to you either.)

So my suggestion listen to the expert's advice and compare what you've done with what's in that paper. Then, if possible, ask some more senior people for advice for where is an appropriate place to submit your work.

Here's a positive from this experience that nobody's mentioned:

You got your answer in 2 months. This is actually quite fast for an answer on a mathematics paper. This is much better than waiting 12 months or more, then getting that same response.

You should address the cited reference (which may be duplicating some of your ideas), and resubmit to another journal.