The Inevitable Wall of the Autodidact
It sounds like you're having trouble with the "auto" aspect of autodidact.
Although I received a minor in mathematics, I consider myself largely self-taught with my interest kindling after graduation. I have had a similar experience, having worked through small pieces of a large number of advanced books.
Very often, my difficulties amount to something trivial that could be resolved in a matter of minutes if I had access to a mentor.
Notational issues are sometimes tricky. Worse yet are issues where you have some misconception. While I do not advice trying every exercise, I would spend a great deal of attention on any exercise that looks contradictory.
A personal example I had was one problem which demotivated me for over a year. It had to do with me misunderstanding the distinction between smooth and irreducible curves. (The two are equivalent in the case of quadratic plane curves, but the situation I had was with quadratic surfaces).
It can be difficult to figure out who and where to ask. I have had plenty of my questions ignored on this site. I have found Freenode IRC's ##math channel to be quite useful at times. But there are still relatively few experts there. Very occasionally, I have emailed authors of my texts when I feel I have found a typo. (I am about 2 for 4 on whether it really was a typo). I don't really have a good answer to this problem.
As a small note about this site, the StackExchange websites do not properly address the issues learners have with a subject. Most questions students need to ask are not questions at all, but rather, are discussions. And the SE policy is that "We're not a discussion board". And while that attitude makes for a clean Q&A archive, it is unhelpful to the student.
However, I do have a few strategies for coping with it. First, have great patience when learning any subject. The advantage of an autodidact versus a conventional student is you are not expected to learn the subject. You are not accountable for any grades or progress towards your thesis. Use that to your advantage, and take your time. If a subject is worth mastering now, it will be worth mastering in a year.
Secondly, draw on as many resources as you can. Textbooks vary wildly in quality. And even "excellent" textbooks can be frustrating if you don't have the proper background or perspective yet. Sometimes just seeing an idea from two authors' words can resolve your confusion. It emphasizes the important aspects and deemphasizes the secondary issues. Authors will often motivate an idea with a clever example. It's good to know when the example is only a piece of trivia versus a central tool to reconcile the subject against.
Third, don't try to go into a new subject with high expectations of rigor. Perhaps this was an issue particular to the way I approach things, but I always wanted airtight proofs. It cost a lot of time and frustration, and is contrary to the way mathematics is actually done in many cases. I am not saying proofs are not important. But sometimes it's more important to know that something is true than why. And sometimes it's ok to have only a sketch of the proof (or even just a collection of ideas used in the proof) in your head. Your goal is not to memorize loads of mathematics. It is to understand it.
Similarly, your perspective and preconceived notions can sometimes work against you. I remember this has happened to me twice. First, when learning abstract algebra after exposure to category theory (Very backwards, yes. Thank the Haskell community). I wanted everything to fit into a category, and I wanted all my arguments to be in terms of universal properties. Somewhat related, I had learned about constructive logic, and so I was hyperfocused on where I used the principle of excluded middle and I was overly concerned about how precisely my induction was stated. It turns out, handwaving is an important tool you need, and being overly precise became an issue for me.
A second personal example was when learning complex analysis after having learned about smooth manifolds and basic algebraic geometry. In particular, complex integration was difficult to reconcile. The idea that you should mix real curves with complex curves caused me issues. And I couldn't see at the time why real line integrals used essentially a dot product while complex contour integrals used complex multiplication. Lastly, because I had learned the basics of topology, I spent a disproportionate amount of time trying to see how to remove the "piecewise smooth" condition that seemed to be ubiquitous. After all, in my mind, what would smoothness on the boundary have to do with analyticity?
Lastly, I'd say play to your strengths. Are you stuck on some theoretically-heavy part in your linear algebra book? Skip it for now. Focus on the chapters that do make sense. Find another book which speaks your language. It's more important to be making progress somewhere than to understand what's right in front of you. Come back to what's got you stuck a few months later, and if it's still a sticking point, then become a 'nuisance' to Math.SE until your question is resolved in terms that you understand.
Self-study is a really good idea but it can be discouraging quite quickly as you said. If you're in class, then study with a more "formal" and "dry" book is not bad because you listen oftenly the teacher giving explanations and do exercises with friends. Of course, when you're alone it's not really motivating to feel stuck every pages and ask lot of questions about the book you just found.
My advice would be : choose the right book ! They are plenty of really good PDF online. For self-study, I especially enjoy PDF with little exercises left to the reader : normally they are easy and it's a quick and nice way for learning, in addition of reading proofs, theorems and doing problems. (Of course, it depends of the author. I already saw a book where the first exercise was prove fully the Banach-Tarski paradox, with really few hints, not that easy ...)
Books are rarely clear at the beginning and becoming really obscure at the end. If you are able to understand the first chapters, normally you should be able to understand fair part of the book (with regular work !) Usually, the subject is becoming more difficult, but this is normal to spend more time the more you did advance in your book. That simply mean you are learning something new. You'll never learn anything if everything is clear from the beginning for you ! Learning maths (for me at least) is mainly fail, try, fail, try again (alone if you are motivated enough) and finally understand something new. I don't have motivation everyday but discovering something and feeling you did understood something well is one of the best motivation you can have (for me at least).
Another advice : once you did choose a book, and try a few pages for seeing if you feel comfortable with it, you should keep it until the end. When I try to self-study I was permanently studying with different sources, and thinking "Well, I don't understand so another author probably explain it better !!" and finally I was never making efforts for really understanding the subject.
Hope it helps a bit !