David Joyce, Professor of Mathematics at Clark University
Yes just to add to what David Joyce said - it's actually now known that it is theoretically impossible to have a single basic theory that thoroughly grounds all of mathematics. Though we do have theories that are consdiered "good enough" to be reasonably acceptable.
In certain areas - for instance in Euclidean geometry you can create a complete and consistent theory, with few axioms, that is provably correct. This is for simple ruler and compass type geometry - not the likes of trigonometry. Those constructions you see with lines and circles in geometry books - they can be given a completely rigorous foundation.
First you have to improve on Euclid's axioms (he left a few out, for instance he didn't think to add an axiom that a line that enters a triangle has to exit it through one of its other sides). But after you do that, it is "done and dusted" - no more needs to be done to complete the theory.
THE C19 WILD EXAMPLES THAT LEAD TO THE MODERN QUESTIONING OF FOUNDATIONS OF MATHS
It used to be thought, following this example of Euclidean geometry, that all mathematics could be given this treatment. But back in the C19 mathematicians came across some really wild functions. E.g. a function that is everywhere continuous - so you can draw it theoretically with a continuous motion of a point - but nowhere differentiable, so that there isn't even a single point on the line where it has a "direction" or gradient.
Another example - they found a construction for a one dimensional jagged line that "fills space" so that every point in space is somewhere on that line.
Many other things like that. For instance, while doing a proper mathematical analysis of heat flow, they found it necessary to bring in many abstract ideas of higher orders of infinity which you'd think were unnecessary for such a subject grounded in physics - and what's more - when trying to ground those ideas in logic, they ended up with paradoxes at first.
HILBERT'S PROGRAM TO PUT ALL MATHS ON A FIRM FOUNDATION
Hilbert put forward a program to put the whole of mathematics back onto a secure foundation. And for a while progress seemed to be going well.
But then in the 1930s, Godel proved, remarkably, that Hilbert's program was impossible.
WHAT GODEL SHOWED
He showed that - though Euclidean geometry is okay - as soon as you add in the idea of a number, with addition and multiplication (nothing very complicated) - that you can no longer come up with any single finite list of axioms to use as a basis for your proof methods.
He showed that no matter what list of axioms you give, that you can find out new things not covered by those axioms.
For instance take any axiomatization of addition, multiplication and adding 1. Then you can find a statement that can't be deduced from any of those axioms, which you can also see must be true, for the numbers as ordinarily understood.
He showed also that you can add the negation of that statement to your theory and it's negation is also consistent. The result is an "omega inconsistent" theory that says simultaneously that some statement about the numbers has an exception, says in the abstract that there exists an exception, and also says, for each individual number that you name, that the statement is true.
So - he showed that any attempt at axiomatizing addition, multiplication, and adding 1, ends up with a theory that is incomplete (there are some things it can't prove or disprove) and you can also add new statements to it like this.
It's generally thought that the axioms for numbers, the Peano axioms, are consistent. There's good reason indeed for supposing they are. But you can't prove for sure that they are. And certainly also, they don't let you deduce all possible truths about numbers.
So - with numbers so basic and essential to maths, then it is pretty clear that Hilbert's program can't be accomplished, even for the maths we most need for physics, practical maths like the calculus methods used for those heat flow problems.
It can only be done for a few very selective areas such as "ruler and compass constructions" in geometry.
INTERPRETATION OF NEARLY ALL MATHS IN A FOUNDATIONAL THEORY
The usual way around this is to work within the axioms of "Zermelo Frankel set theory" or some other foundation theory in logic.
Nobody ever actually proves long complex results directly from these axioms as that would be immensely tedious. But the idea is that theoretically you could do that if challenged.
It turns out that nearly all maths can be interpreted, in one way or another in this set theory.
This is surprising as it only talks about the empty set (set with no members), and the set whose only member is the empty set, and various other complex sets you can build up from the empty set in this way. So where are the points, where are the lines, where are the numbers even, where are the sines and cosines?
Well it turns out you can define all of those in terms of complicated constructions using the empty set - and relationships between those constructions, and properties of them.
So - this gives us the nearest we can have to a unified foundation for all of maths, as Hilbert seeked for.
There are several of those set theories. Most use ZF, but there are others you can use. But they are "inter-interpretable" that they can, most of them, be modeled in each other. Just as you can interpret numbers and lines etc as complicated arrangements of structures of the empty set, so you can also reinterpret these various types of sets as each other.
So -you can just choose your favourite set theory, usually ZF, and interpret everything in that. And for ordinary maths such as trigonometry, then nobody will notice any difference no matter what your grounding set theory is - as you never actually prove results using it. Just rely on the logicians who have proved the theorems needed to show that your maths can be interpreted in this way.
PRACTICAL MATHS - REDUCE TO CONCEPTS YOU ARE HAPPY AS A MATHEMATICIAN TO TREAT AS PRIMITIVES
As for practical maths - well - I think better to just understand it as best you can. And that's done by reducing to prior concepts that you are happy to take as "primitives".
So for instance, unless working in foundations of maths, it is reasonable to take the numbers themselves, and the Peano axioms, addition, and multiplication, all as primitives that you just accept as true, self evident. And build up from there.
CAN DEFINITELY DO THAT WITH EUCLIDEAN GEOMETRY - IT IS NOT IMPROVED BY INTERPRETING IT IN SET THEORY
You can definitely do that for Euclidean geometry, just ruler and compass constructions, as it stands alone as a provably complete, consistent theory to use, irrespective of any of these foundational arguments.
You could interpret Euclidean geometry in set theory, but is no need at all and indeed that makes it less secure as we know it is consistent and complete already.
While nobody has proved for sure that ZF is consistent, and nobody can give a totally impossible to question proof of this because of Godel's result, though we believe it is.
ALL THESE FOUNDATION THEORIES ARE JUST A CONVENIENCE FOR LOGICIANS
And - I'd say also, similarly, that philosophically, these foundations theories are just a "convenience".
The numbers are not in any sense "really" the intricate Von Neumann ordinal structures built up by nesting methods from the empty set. That is just a way to help reason about the foundations of maths and reduce the number of assumptions you need to make, so fewer axioms to inspect and test, so making things easier for mathematicians working on logical foundations.