How to deal with an abusive advisor?
My advice: Go to the Chair of your mathematics department, and tell him exactly what you told us. You need to be straightforward and honest.
If not, contact the Dean of the college/or the dean of faculty affairs, and tell them what you told us. No need to tell us your problems where we can't solve them, when you can (and must) approach those who can handle these affairs.
It may seem a bit embarrassing, but I can assure you, with my years of experience, if you tell whom I mentioned (or another authority), at least some solution will transpire.
I cannot stop typing this long response.
I'm sorry to hear that you are in this situation and I have lots of sympathy for you. Your story sounds similar to but not as horrible as mine in the previous university and the department I was in (NOT the one I list in my profile now, that is the one I'm currently in, the one I enjoy so far). I'll first give you a one line advice, then answer your question in detail. In the end, I attached my story just for your reference (can't resist telling it!).
One-Line Advice
Most likely, the best option is to peacefully walk away and find a new advisor.
Detailed Answer
Based on your description, I can hardly imagine things will be better later on if you don't take action now. You've worked reasonably hard, but have slow progress for the reasons you don't have control. There are many misunderstandings in the communication with your advisor, and most importantly you're scared of meeting with your advisor. It's impossible to produce any good research work under a scenario like this. Time to look for someone else that fits you better.
You do not specify which stage of graduate study you are in. This may affect your approach now. If you are in the early stage (before passing preliminary/qualify), you should simply walk away, pass the exams, claim your interest change and convince someone else to be your advisor. It's not unusual at all for students to change advisor at this stage. If you're in the late stage (all but dissertation), you might consider enduring the pain for another year to get your degree, the potential cost (like dropping out) of making any unsuccessful move is too huge at this stage. If you're in the middle stage(pass qualify not yet advance to candidacy), the situation is awkward. You probably should talk to some department/university authority, and find out what's the best way to continue your study. However, you need to be very careful about who you talk to and how you talk.
When you describe your scenario to any third party (other professors, department chair, ombudsman, etc.), focus on the key issues you're facing and give them enough information so they can offer reasonable solutions. Avoid ranting and complaining, even if what you're saying is absolutely correct and can provide evidences. You run into danger of convincing other people you're a lazy student with many excuses. For example, you should avoid suffering the things I suffered (see my story+rant).
Last but not least, when it comes to advisor, you should have doubts on words of senior students and most recent graduates, they are the people who rely on advisor's letter to find jobs, therefore are unlikely to tell you any bad words, even if they don't like the advisor themselves.
End of Detailed Answer
My story+rant (only for reference)
Almost the same structure as your story, three years ago I was a new graduate student (in Ph.D. program) working in computational chemistry. It was OK in the beginning. About a few months later, I found out the software and source code I was supposed to use to do the calculation of my main project was fundamentally flawed. Some physical quantities were calculated on a non-trivial wrong way and it was clear to me that any data outputted would be meaningless unless the problems were fixed. I temporarily stopped working on the project to resolve the issue with my collaborators (one postdoc., one software engineer). It was a much slower process than I expect, because the person who wrote the code, although an expert software engineer, knows relatively little about the research I did and couldn't understand why the code was wrong for a long time.
My former advisor then behaved the way like your current advisor. In our meeting, he ignored the issue I faced and only blamed me on slow progress. Very often I found the suggestion he made was wrong and the solution I figured out myself was right. Whenever I pointed out the errors in other papers, providing more than sufficient evidences, he thought I must make up excuses for failing to reproduce the result in the paper. When I wanted to pull the discussion back to the research and seek for advises, all I got is "This is not undergraduate, you're supposed to figure it out."
After many meetings like that eventually he kicked me out of the lab(where he had already kicked out more than half of his students for many different weird reasons), originally with the promised of funding me finishing masters. He gave my research project to another student. Later on he blocked my access to all the data I had, and requested me to come back to lab if the student continue on my project had any problem, otherwise he was going to cut the funding. Even worse, I worked as a TA of my former advisor the academic quarter he kicked me out of his lab, he simply found every chance I made an insignificant mistake in TA job to blamed me, and sent abusive emails.
(Here is what I did, and don't want you to repeat) Finally, I cannot endure many abusive emails like that, decided to response one of them rudely (no personal insult and threat though), accused him lying and attached with evidences. The email was cc to all his group members and some department faculties and staffs. He was scared, he turned into the graduate community center (something equivalent to counseling center), described me as a potential criminal, and asked them to physically isolated me from the department. It took a while for me to convince them that I was the victim, not the other way around. After a few weeks of long conversations via the third party, we finally achieved some compromise- He agreed to do what he promised originally (fund me 2 more months to finish M.S.).
On the other hand, despite the evidence, it was impossible for me to convince other faculties in that department to be my new advisor. Most of them believed my former advisor's words and thought I was a failing student with excuses. They set up an "advancement exam" for me, at the date one faculty who potentially supported me had a doctor appointment, to find an official reason to kicked me out of their Ph.D. program.
End of my story+rant
You've taken an important first step here.
A PhD student is in a dependent position with regard to the advisor; there are many parallels between the abuse that has occurred her, and the more commonly known type, family abuse.
In abusive relationships, there is often something very insidious: gradually, the abuse increases, but at each tiny incremental step, the victim rationalizes staying in the relationship, because there is so much at stake (e.g. keeping the family together, financial pressures, etc.; holding onto funding for continuing studies, intellectual satisfaction, successful thesis, letters of recommendation for future jobs, etc.). Often, the longer the abuse goes on, and the worse it gets, the harder it is to escape from it!
That is why I say that you have taken an important first step.
There are many possible next steps out of this abusive relationship. Perhaps one of the simplist is to make an appointment to see a doctor. Nowadays, many doctors ask a key question at each visit: Do you feel safe at home and at work? Even if the doctor doesn't ask you this, you can simply inform him or her that you don't feel safe at work.
Often, when one makes an appointment, the receptionist asks what the problem is. It will probably be easier for you to make something up, like an earache, than to spit out your problem over the phone to the receptionist.
The advice from @fmlin is excellent. I would add a couple more things:
Find a domestic violence advocacy center in your campus or your town. They are trained in assisting victims of emotional abuse.
Take a trusted person with you to each and every meeting you have on campus related to the problems with your advisor.
There are many possible eventual solutions to the current situation. I will describe one. My crystal ball doesn't tell me whether this will come to pass for you, but I will describe it because I imagine that right now, the paths out from your current situation probably feel extremely limited.
Student (you) enters mental health treatment, paid for by the university (which, after all, is responsible for the professor's treatment of his students).... Student takes a month or more off from studies and responsibilities, but university continues financial and moral support during this medical leave of absence.... Professor enters treatment, paid for by his health insurance.... Professor is required to send all emails to student through an intermediary.... The university arranges for another professor, in a related field, and with a talent for being emotionally supportive, to be present for ALL face to face or phone conversations between the abusive professor and the student.