Should I abandon an open math problem, if it hasn't been actively studied since the mid-1980s?
Based on your brief description, I'd say a more likely explanation for the lack of work on this problem is that people are stuck. Not that there's no interest in it.
Still, I wouldn't recommend this problem for a masters' thesis (or a PhD thesis). At least at the start, you should work on something manageable. If you hit a manageable problem or two out of the park, then you can start trying things professional researchers have attempted and failed.
To my mind, there are two critical questions to ask here:
- Do you and/or your advisor have any ideas for making progress and/or any new results on the problem?
- In general, what does your advisor advise?
Regarding (2): In mathematics, the single most important role a thesis advisor can play is helping a student choose a good problem. Problems are good because of a combination of interest on the part of the student and the advisor, community interest, potential or actual applications, and perceived difficulty level (i.e., tractable but not trivial). Asking a bunch of random internet academics whether to attack this problem seems a bit weird to me: what does your advisor think?
Interregnum: I would like to respectfully disagree with @Wolfang Bangerth's answer. A problem which was studied in the past and on which many papers obtaining partial results were written is a problem of interest to the mathematical community. In my circles at least, solving longstanding open problems is at least as good as solving problems that were posed last year, because the older problems have a higher level of demonstrable difficulty. If the papers in question had been written, say, 80 years ago, then one might have some concern that no living mathematician cares about it (still, you can make us care by doing something sufficiently nice), but problems from 30 years ago that are still being mentioned in contemporary papers are likely to be viewed as having a strong pedigree.
Regarding (1): if you have some traction on this old, unsolved problem, it sounds like a great thing to work on...at least for a while, to see what happens. Conversely, if you have no ideas....tell me again why you and your advisor started studying this problem? Or rather: ask your advisor again.
Am I making the correct interpretation of this professor's remark?
All you've told us about the professor's remark is "He confirms that his work on the problem is very old, so I'm guessing he hasn't done anything newer with the problem since his last papers." And what you said about your interpretation of his remark is "I'm taking his words as an indication that there is not much interest in this problem". While this may very well be a correct interpretation, there are certainly other possible interpretations. For example, I worked on problems 15 years ago that I no longer have any interest in. If a student came to ask me about them I would probably shrug and not show much enthusiasm, but those problems are still very interesting to many other people.
In other words, your description of the professor's remark (and possibly also the remark itself) is too vague to be able for anyone here to be able to meaningfully say whether the problem is still of interest to anyone or not. You and advisor might want to get a second opinion from another person who is knowledgeable on the subject.
Should I still stick with the problem, even though it might be true that it is not currently being actively studied?
I'm currently writing a paper on a problem from the 1960's that has been the subject of only very few papers since then, the last of them being from the early 1990's. I don't know for sure how the world will react to my paper, but I think I've made very nice progress on the subject and have hopes that my new results will excite new interest in the problem, which is intrinsically very appealing. I am also a tenured professor and can easily afford to risk the scenario where this doesn't happen. Nonetheless, I am of the opinion that pure math research shouldn't be about following fashions or fads (which math is very much susceptible to, much like other areas of academia) but should be driven by an innate desire to understand a structure one is interested in and finds beauty in. See also Pete L. Clark's comment on Wolfgang Bangerth's answer for examples where working on an unpopular or archaic subject paid off bigtime.
With that said, a lot of people prefer working on popular topics and think that working on such topics is a safer route to success in math, especially for someone who is just starting out. I don't have a strong opinion that that's false -- it's simply not my style -- and I completely respect someone who makes their decisions based on such a belief. So keep in mind that working on a subject no one else is working on is a somewhat lonely pursuit with a very uncertain payoff. But if that where your heart tells you to go, you should know that it is certainly possible to find success working on unpopular subjects.