How much time does a great mathematician take to solve an extreme problem?

I recommend reading John Littlewood's Miscellany (more recent editions ed Bela Bollobas, another great mathematician but who is still alive). Littlewood was one of the great mathematicians of the last century. He noted that mathematicians tend to be good on different timescales. Few are good on all timescales.

To become a "great" mathematician, you have to solve big problems, which means you have to be good on timescales of a decade or so. That may leave you doing badly in olympiads, let alone coffee table conversations where the timescales are much, much shorter.

It is tragic that Bela himself feels he has not lived up to his early promise as one of the great math contest competitors. Of course, it does not help that his work has been pioneering new fields of maths, which tends to generate acclaim long after you are dead.

There are of course examples the other way like Grisha Perelman who excelled at contests and solved the Poincare conjecture. Precise correlations are difficult, because it is hard to be objective about achievements (why for example does Andrew Wiles tend to get** the credit for proving* the Shimura conjecture rather than share it with Richard Taylor?).

But the general advice if you want to do maths research is not to spend too much time on problems that are easy (mere "exercises"). You should be regularly struggling with a single problem for an hour or more, even in your teens.

Of course, there is a balance, it is easy to get discouraged. That is why a good mentor is really helpful and why the relationship between graduate student and supervisor is traditionally close.

Unfortunately the current system has far too many people supposedly doing research, when the only things of value they really do are teach, distil, and show scholarship (ie acquire a good knowledge of what is already out there to share with others). The difficulty is that the system is set up to reward research rather than those other equally important university tasks, whereas the sad truth is that only the few make a significant contribution to discovering important new maths.

The other big difficulty is the ludicrous publication pressure. 50 years ago, UK universities were happy if you published something good every 5-10 years and put up with you even if you published nothing for 20 years, but now the pressure is to publish several times a year. The result is journals full of trivia.

You also asked how you can improve at the olympiad type examples. The only way to improve is practise, practise, practise. It is interesting to compare the IMO 1959 problems with the more recent ones. Any current competitor would regard the early years' IMO problems as completely trivial. Their difficulty has increased enormously. When I took part in 1968, it was a relatively amateurish affair. The Soviet bloc took it reasonably seriously, but were much more interested in the All-Soviet Union competition, whilst the UK team did no training at all!

Again, I suspect things have gone too far. I suspect you have no chance of getting full marks in the IMO today without dedicating years to it to the exclusion of almost everything else.

@AbhijitAJ also gives good advice in recommending George Polya's popular book. You might also want to look at his less popular work. If you are interested in analysis his two volume Problems and Theorems in Analysis (with Gabor Szego) is a classic which got many mathematicians started on their research careers. His more elementary works which expand on "How to solve it" are also good (Mathematics & Plausible Reasoning, and Mathematical Discovery). I got more out of them than "How to solve it".

Note also @Thomas comment below. Biographies can also be interesting. I particularly enjoyed Masha Gessen's Perfect Rigour about Grisha Perelman. The descriptions of the St Petersburg math club that he attended after school in his teens are fascinating. Mentors can be even better, but can be two-edged. The main reason I left maths in 1973 was because two year's contact with Peter Swinnerton-Dyer made me feel I would find it hard to compete with people like that. :)

** warning 109 page pdf $\text{ }$ * warning 22 page pdf

An important thing is to first find a resolution strategy. Your intuition should tell you how the computation will proceed.

In this case, I noticed that the goal is to eliminate the variables $M$ and $N$, and you can do that by completing the square in the first two equations. So a possible strategy is to explicit $M$ and $N$ and plug them in the third equation and we will find a relation. $$M^2 + 2mM\cos\theta+m^2\cos^2\theta=(M+m\cos\theta)^2=1-m^2+m^2\cos^2\theta=1-m^2\sin^2\theta.$$ Similarly $$(N+n\cos\theta)^2=1-n^2\sin^2\theta.$$

Then I breathed a little (instead of rushing to the obvious solution of expliciting $M$ and $N$ completely) and noticed that the LHS of the third relation was very close to the product of the LHS, so (without taking the square roots prematurely)

$$(1-n^2\sin^2\theta)(1-m^2\sin^2\theta)=(M+m\cos\theta)^2(N+n\cos\theta)^2=(MN+Mn\cos\theta+Nm\cos^2\theta+mn\cos^2\theta)^2=(0-mn\sin^2\theta)^2$$

and from there the claim.

It's all about practicing, observing and spotting familiar patterns.

PS: I am no great mathematician.