How did early mathematicians make it without Set theory?
Some hints...
You can start reading Cauchy's elucidation of function (1823) :
On nomme quantité variable celle que l'on considère comme devant recevoir successivement plusieurs valeurs différentes les unes des autres. On appelle au contraire quantité constante toute quantité qui reçoit une valeur fixe et déterminée.
[We name variable a quantity that receives successively many different values. We name constant a quantity that receives a fixed and determined value.]
Lorsque des quantités variables sont tellement liées entre elles, que, la valeur de l'une d'elles étant donnée, on puisse en conclure les valeurs de toutes les autres, on conçoit d'ordinaire ces diverses quantités exprimées au moyen de l'une d'entre elles, qui prend alors le nom de variable indépendante; et les autres quantités, exprimées au moyen de la variable indépendante, sont ce qu'on appelle des fonctions de cette variable .
[When some variable quantities are linked together in a way that, having fixed the value of one of them, all others quantities can be determined, on conceive these different quantities as expressed by way of one of them, named independent variable. The remaining quantities, expessed by way of the independent variable, are named functions of that variable.]
Thus, in a nutshell, the concept of "function" was a primitive one, like today for set. A function is a correspondence (a relation) between two "variable quantities".
It is worth noting that Cauchy's definition of "variable quantity" was already present into de L'Hôpital's textbook : Analyse des infiniment petits pour l'intelligence des lignes courbes (1st ed, 1696), the first calculus' textbook. See :
- Robert Bradley & Salvatore Petrilli & CEdward Sandifer (editors), L’Hôpital's Analyse des infiniments petits (2015).
An early occurrence of "function" is in Leibniz, in De linea ex lineis numero infinitis ordinatim ductis inter se concurrentibus formata, easque omnes tangente, ac de novo in ea re Analysis infinitorum usu (1692), but a "reasonable" definition of function is available only with Johann Bernoulli, Remarques sur ce qu'on a donne jusqu'ici de solutions des problemes sur les isopdrimitres (1718) and Leonhard Euler, Introductio in analysin infinitorum (1748).
Regarding group we may see e.g. Arthur Cayley : he uses the name "set" in his definition of group (1854) :
A set of symbols : $1,α,β,\ldots$ all of them different, and such that the product of any two of them (no matter in what order), or the product of any one of them into itself, belongs to the set, is said to be a group.
Set here is not a mathematical object : no specific properties of sets are assumed.
I recently read a scholarly book relevant to this question.
Jeremy Gray, Plato's Ghost: The Modernist Transformation of Mathematics
The transformation in question goes from 1880 to 1910 (roughly speaking). Gray discusses how mathematics was done before, the turmoil surrounding the change, and how it was done after.
Recommended. But not for the faint of heart.
We can get a hint to an answer by considering some of today's undergraduate textbooks which do not assume a knowledge of set theory, and do not include a formal definition of a function. There are still a lot of such books.
An entire calculus course can be taught without dwelling on set theory. In fact, I know that if I have to teach calculus to a big class, the worst thing I can do is to spend half an hour on set theoretical issues; I can see the glaze settle over their eyes (except for the small minority of students who instead get a glint of excitement in their eyes, and who I am constantly on the lookout for).
A student can ingest what they need to know about derivatives and their applications, integrals and their applications, differential equations, et cetera, and that student can go on to apply calculus in many different branches of inquiry, without me ever spending that half hour on set theoretical issues. And that's a good thing.