I got accepted into a math PhD program but I don't feel adequate enough to attend
First off, congratulations on being accepted to the PhD. That means the faculty have carefully evaluated your application materials and decided that you are a promising young mathematician. They are experts and they think you have what it takes to finish a PhD.
No one gets admitted just because of their gender and/or race. It would be a waste of the department's resources to admit unqualified students and it would be a disservice to those students -- setting them up to fail.
It sounds like you have Impostor Syndrome. The truth is that almost everyone, from time to time, feels like they don't belong. I have felt that way at many points.
No, you should not "just quit". You have a great opportunity in front of you. It sounds like you have other things that you want to consider doing. Perhaps you will decide that those other opportunities are better for you. But, if you do turn down the PhD, you should not think of it as quitting. You should think of it as taking a positive step in another direction.
That said, academia is hard. And I would not advise anyone to do a PhD unless they really enjoy research.
You should think seriously about what you want to do, as it sounds like you are uncertain about where your best option lies. Talk to people you know personally, as they can best advise you.
One thing to note is that a PhD should give you some flexibility to study other things or, better yet, to combine different research topics. My PhD program allowed — required, in fact — me to take graduate courses in totally different fields.
Whether this is "impostor syndrome" or not, you should realize that you can always quit later. Just because you start a Ph.D. program, doesn't mean you have to labor away for 5 or 7 years at something you suck at. Try it for a year and see how things go. If you still love math and still get A's, carry on. If you can't stand it, then drop out. At least then you'll know for sure. You won't spend the rest of your life wondering if you should have given the Ph.D. a shot.
If you don't drop out. Swell. If you do, the year is not wasted. You still learn some valuable math that you can put to some use somewhere. Some graduate credit might transfer to the philosophy department. If you like teaching, then you will have taken a few more courses which will enhance that skill.
Win-win.
I think you should give the math PhD a try. I am almost exactly like you in terms of strengths and weaknesses, yet doing a stats PhD was the best few years of my life, and now I am smarter and better-rounded than if I'd done a PhD in a personally "easier" subject.
Math has always been my hardest subject in school; all the way through elementary school, high school, and college, I disliked math classes and excelled at the other subjects. I'm quite error-prone when handling messy formulas, and I have a hard time manipulating abstract mathematical objects in working memory. But, like you, I was a good student and managed good grades in math, even though it was definitely my weakest subject.
Then somehow I fell in love with statistics and, rather to my surprise, found myself at a very selective statistics PhD program. I had the time of my life and graduated early. Here is what I'd recommend:
Know your own intellectual strengths and weaknesses. Imposter syndrome is a real thing, but don't let others tell you to automatically attribute your perceived weaknesses to imposter syndrome. Rather, calibrate your strategies and expectations to your own strengths and weaknesses. For example, like you, I easily forget useful math facts, which is a real pain on timed, closed-notes exams. As mentioned above, I'm also pretty slow at actually doing math accurately. Therefore, for high-stakes exams like the quals, my study strategy focused on training my weaknesses aggressively through assiduously memorizing math facts. During the exams themselves, as you'd do at, for example, an athletic event, my strategy switched to playing to my strengths in order to earn as many points per unit time as possible. For me, that meant using my strong understanding of the concepts to earn lots of points on that front, and quickly bailing on any integral that seemed like a pain to simplify (because the probability that I'd mess it up is pretty high, and it's just not a good points:time ratio). Again, this kind of attitude is not imposter syndrome: you are actively working to improve your weaknesses, while finding ways around them on tests.
Math research is tremendously different from "school math". With no false modesty, I have many classmates who are much better than me at math. However, many students who love and excel at school math neither love nor excel at research math. Among the key differences are:
The role of high-level creativity and conceptual understanding is much higher in research math than school math. Major research advances don't always come from breathtaking mathematical shrewdness. Sometimes they arise from out-of-the-box thinking or the creativity to recognize and repurpose a useful analog in a disparate subdiscipline. Like me, you may find yourself quite adept at the latter.
Research math requires many other skills besides muscling through integrals. Given that you excel at humanities classes, I suspect that, like me, you're an excellent writer. You will spend a lot of time writing when doing a PhD and subsequently as a researcher. If you write, say, 30% better and faster than the average math PhD student, you will find yourself flying through your dissertation and paper submissions. I wrote each of my 3 dissertation papers in 1 week. This more than made up for the time lost along the way to asking Wikipedia for the 844th time how to do a Taylor series expansion.
Research math is also the ultimate "open-book exam". There are no time limits, and you can Google Scholar to your heart's content. This also means that, as discussed above, you can play to your strengths. For example, whereas others with perhaps more mathematical agility develop new research directions from the bottom up by playing with formulas and combining things in clever ways to see what happens, this does not work for me: I simply get stuck in an algebraic morass. Instead, I work from intuition and conceptual understanding first, then reach for simulation tools, and only then put pencil to paper.
Good luck with your decision.