I almost quit self-studying mathematics, but should I continue?
Do the things you like. If something you're doing isn't fun, stop, and do something else.
Mathematics is hard enough that if you are doing it it should be because you enjoy it. There is no point in doing it to become a mathematician if you don't enjoy it; you'd be in the position of someone who is trying to become a rabbi, but isn't Jewish, or maybe someone who is wants to become a professional violinist but doesn't like playing the violin. It's not just silly, it's insane.
If you find some topic uninteresting, skip it. Perhaps someday it will become interesting to you, perhaps because you need it to solve some problem you want to solve, and you can learn it then. Or perhaps it never grabs you; so what? You aren't going to run out of things to learn. (Or if you do run out of subjects that seem interesting, perhaps you should take that as a sign that you don't really want to be studying mathematics.)
If you find you're burned out and you don't want to do any more problems, stop and do other things for a while. I would not be surprised if you found that that after a few weeks off you discovered you were no longer burned out, and you wanted to start again. At sixteen, a few weeks or months seems like a long time. But when you look back on it, it seems a lot less important. When I was a teenager I loved programming the computer and I did it all the time. But I burned out on computer programming when I was sixteen and stopped doing it for a year and a half. Then one day it was interesting again and I went back to programming. My 18-month vacation did not prevent me from having a productive career as a programmer, or from becoming a well-known expert on computer programming. Do I look back on my life and wish I had that extra 18 months, do I wonder if I wouldn't have been more successful with 39 years of practice instead of only 37½? Of course I don't. Instead I look back and it seems that taking a year-and-a-half vacation from programming was most likely the right thing to do at the time.
But suppose you did burn out and didn't want to solve math problems ever again? What then? All I can think is "So what?" The world would have to do without your mathematical contributions. That's okay; you can find something else to contribute instead.
It sounds like the mathematics you've been exposed to so far has been very problem/solution driven. If that's not floating your boat at the moment, why not try some math that is theorem/proof driven? Here are a few recommendations.
I've read a fair chunk of books 1 and 2, and they're great! They might be a bit advanced for a sixteen year old with no background in proof, so perhaps a better starting point would be books 3 and 4. I haven't read them, but the amazon reviews speak for themselves.
Hope that helps!
Remark
I've also heard that introductory number theory books are great for people in your situation; that is, bright high school students who are looking for a bit more of a "kick" and a bit more insight. Perhaps someone else will be able to recommend something in this domain.
I self studied at your age. But I progressed through high school at a regular rate. Looking back, the self-study was very helpful and useful. Maybe I didn't necessarily learn things in the right order, but the insights I developed with my self-study were invaluable.
This is what I would do. I would focus less on where you want to go, and more on how to enjoy getting there. In time, assuming you major in Math at college, you will be told lots of things in the right order, and the foundations will become strong. But in the meantime, find books that you enjoy reading from the library.
Also, one thing I did (and still do at the tender age of 50) is to sometimes speed-read a book. Rather than doing each and every exercise, I try to read it and get the flavor. Maybe there are things I don't fully understand, but later - sometimes years later - I look at it again, and then it makes so much more sense.
I did have an extra advantage. My mother had taken Math at college. And she did provide me with some of the books I read in my earlier years. But later on, I started to self-select. And it sounds like you are in the same place I was at your age.
I really think you should continue to self-study. I think you should focus on enjoying it. And also, try those impossible problems (like the Putnam problems). Don't feel bad if you cannot get most of them to work out. Rather do it because it is fun.