What underlies formal logic (or math, generally)?
Some comments mentioned category theory. The book Conceptual Mathematics by Lawvere and Shanuel is meant as an introduction to category theory for people with only a little background in mathematics. In this book the authors suggest doing all the exercises. If you don't know much about category theory this is imperative. Hope this helps.
You are raising some very interesting questions. Unfortunately, as I'm not an expert myself, I can only point you in a few interesting directions, and give "in-a-nutshell" type characterizations, rather than expound on the logical and philosophical foundations of math in full detail. But hopefully you will find something useful here!
In the early 20th century, many of history's greatest mathematicians and philosophers asked these very same questions, about the connections between logic and mathematics. Attempts were made to build up a system that could represent all of mathematics, from the most basic of logical primitives. One of the most famous of these was Hilbert's program, expounded by German mathematician David Hilbert. Hilbert set forth several properties he wanted this logical system to have. Perhaps two of the most important of these were:
- Consistency: the logical system should not be able to prove any statements that were not true.
- Completeness: the logical system should be able to prove all true statements.
Despite much initial enthusiasm, it turned out however that this program was doomed from the start. In 1931, Kurt Gödel published the now-famous Gödel's incompleteness theorems, which basically sent Hilbert's program crashing to the ground. In a nutshell, here's what Gödel's theorems proved:
- Any logical system that is both consistent, and powerful enough to represent relatively basic mathematical constructs such as basic arithmetic, is necessarily incomplete. There will always be some statements that are logically true but that also cannot be proven from within the system.
- The consistency of such a system cannot be proven from within the system itself.
The details of Gödel's proofs are a bit technical, but basically, he constructed a mathematical analog of Russell's paradox - i.e. "this sentence is false", except with "not provable" instead of "false".
Today, Hilbert's program may no longer live in the form in which it was originally envisioned, but that doesn't mean there aren't still logical underpinnings to mathematics. A very widely used logical foundation of modern mathematics, for example, is the Zermelo-Frankel set theory, combined with the Axiom of Choice. This theory gets around some of the aforementioned paradoxical issues by ruling them out axiomatically - for example, it does not allow sets which are contained within themselves.
Even with these theories, however, there is still no universal and comprehensive "Theory of Everything" that arises from the void and manages to explain everything. Even the most robust and well-developed mathematical thought still ultimately rests on underlying primitive notions - base ideas and concepts which are "defined" by an appeal to experience, or "common sense", and upon which a myriad of derived concepts are constructed. While it is desirable for these to be as "primitive" as possible, ultimately, modern mathematicians and philosophers are all acutely aware that these primitive notions are, in the end, essentially arbitrary from a philosophical standpoint.
For example, ZFC has the primitive notion of a "set" - a notion which is "defined" not in the usual mathematical sense, which would be by rigorous definition in terms of primitive notions and/or other definitions, but rather in an intuitive sense, by appeal to general knowledge of what a "group of things" is like. In the end, any attempt to define a primitive notion rigorously and not just intuitively must have one of two results: either (1) it will be circular, sometimes in a way that's not always easy to spot, or (2) it will end up creating new primitive notions underneath the old ones. For example, instead of positing the primitive notion of set theory as a "set", one could conceive instead of its single salient property - "contains" - and have this be the primitive notion instead, where a set is then just a "container" / "contain-er" / "something that contains". On the other hand, though, why not just stick with "set" and call "contains" "the thing that sets do"? Is one more "primitive" than the other? Hard to say.
- (Is it possible that this "common sense" on which our primitive notions are based is perhaps less common than we might imagine? I'd say it's quite possible. For example, who's to say whether our basic intuitions about things like sets would seem intuitive at all to, say, a civilization of alien extraterrestrials? Maybe billions of years of evolution would lead them to a radically different and entirely inconceivable set of "intuitions" about what groups of objects are, resulting in a radically different and entirely inconceivable mathematics? On the other hand, maybe there really is something much more universal to these intuitions than we ever realized, something that humanity has simply yet to put its collective finger on.)
In any event, primitive notions can also vary dependent on the academic context. A notion can be primitive in one context - for example when a particular research program is content to leave its exposition to others - and defined based on more basal primitives in another context.
In addition to primitive notions, these foundational theories often rest on axioms. I'd say axioms have a similar flavor to primitive notions in that both are a priori, but axioms differ in that they have a logical structure. While the semantic validity of an axiom may be arbitrary, and in fact whole fields of mathematics have arisen out of simply asking "what happens if we change this axiom?", its structure is not arbitrary. Axioms are basically logical statements, taken as true by assumption, about the properties and inner workings of primitive or derived objects.
For example, the aforementioned Zermelo-Frankel set theory is made up of a set of 8 axioms. Mathematicians later discovered that these axioms weren't quite adequate to constitute a logical basis for everything they wanted to construct, so nowadays you often see "ZFC" - Zermelo-Frankel set theory (8 axioms) plus the Axiom of Choice (1 additional axiom).
Regarding the question of your own particular path of study - mathematics is such a wide and deep field, that I don't know that it would even be possible to pinpoint a specific area and target only that from the get-go. There is just so much out there when it comes to math!
I think what you should do is this: first, start with the basics. No matter what you end up deciding to study, there will be certain foundational areas of math which you will need. If you haven't already, you should take whatever you need in order to be ready to study calculus, and then you should take calculus. It may be different where you live, but where I am, university-level calculus is usually divided into 3 semesters.
I would also recommend courses such as linear algebra, differential equations, real analysis, measure theory, or abstract algebra. I saw category theory mentioned - personally, I'm of the mind that pedagogically, category theory comes best around the middle sometime, not so early that it's incomprehensible but not so late that it's too late to take advantage of its representational utility.
By the time you get to these courses, your ideas about precisely where your interests lie may start to crystallize a bit more. Foster relationships with professors (if you're shy, it can be intimidating, but they love to see students who are curious and passionate!), TA's, grad students in your areas of study, undergrads who share your interests, etc. Take as many classes as you can, and consider auditing high-level courses to expand your knowledge. Take full advantage of any administrative, counseling, etc. resources that your school has to help you discover your interests. These things together will help you figure out which direction to head in!