"Gaps" or "holes" in rational number system
It depends on what you consider a “gap” in the rational numbers. As long as this is not a formally defined concept, we’re just talking about our everyday, geometrically informed conceptions of gaps.
The mere fact that a certain equation doesn’t have a rational solution doesn’t seem like a basis for identifying a “gap”. The equation $x^2=-1$ also has no solution in the rational numbers, and this fact also gives rise to an extension of the number system (to the complex numbers, in this case), but it doesn’t fit with our everyday notion of a gap to call this deficiency a “gap”. This corresponds to the fact that when we fill the need to solve the equation $x^2=2$ by introducing irrational numbers, we depict them on the same axis as the rational numbers, between rational numbers, whereas when we fill the need to solve the equation $x^2=-1$ by introducing imaginary numbers, we depict them along a different axis.
So the mere fact that some equation can’t be solved does not indicate a gap in the number system, if by “gap” we mean anything like what we mean by it in everyday language (where a “gap” would certainly be depicted along the same axis as the things between which it lies). By contrast, the fact that you can split the rational numbers into two sets, with all numbers in one set greater than all numbers in the other but without a number that marks the boundary, does seem to suggest that there “should” be a number at the boundary, so that, in a sense not too removed from our everyday use of the word, there is a gap at the boundary.
There is a difference between a thing not existing in some set, and the existence of "gap" corresponding to that thing. For example, there is no rational number $p$ such that $p > q$ for all rational numbers $q$. Does this mean that there is a "gap" in the rationals corresponding to some "largest" rational number? I think that most people would argue that, no, there is no "gap" there.
Or, perhaps more interestingly, there is no rational number $p$ such that $p^2 = -1$. In order to solve the equation $p^2 + 1 = 0$, it is necessary to introduce the imaginary unit $i$ and the complex number system (or, perhaps, the Gaussian rationals; we don't need a continuum, really). Is the lack of existence of a rational $p$ such that $p^2 = -1$ a "gap"? Again, I think that most people would argue that it is not.
Similarly, it is not a priori obvious that the non-existence of a (positive) rational number $p$ such that $p^2 = 2$ represents any kind of gap in the rational number system. By showing that no such $p$ exists, all Rudin has done is show that no such $p$ exists. This seems tautological (because it is), but the situation is analogous to the non-existence of a largest rational number or the imaginary unit.
What Rudin then does is demonstrate that there is a "rational number like object", $s$, which can meaningfully be said to have the following properties:
$s^2 = 2$,
there is a set of positive rational numbers $A$ such that $a \in A$ implies that $a < s$, and
there is a set of positive rational numbers $B$ such that $b \in B$ implies that $b > s$.
Thus, in a very meaningful sense, this object $s$ fits into the rational number system in a natural way. It "plugs a hole" in the rationals. Contrast this with the imaginary unit $i$, which doesn't fit into the rational number system in any natural way—it lives in a place that is orthogonal to the rationals.
The best option here is to read Dedekind's original Continuity and irrational numbers or its exposition in Hardy's A Course of Pure Mathematics.
Expansion of numbers systems can be seen driven by algebraic needs as one moves along the path $\mathbb {N}\to\mathbb{Z} \to\mathbb {Q} $. But the next step to $\mathbb {R} $ is totally non-algebraic and not based on finding solutions to polynomial equations. Rather the need is to enhance the order relations. When one tries to analyze the structure of set $\mathbb {Q} $ in terms of order relations $<, >$ a different kind of inadequacy presents us. The idea first popularized by Dedekind is not difficult to grasp and it's a wonder why the issue is not dealt with in high school curriculum.
Dedekind makes use of geometric intuition and argues that if we wish the number system like $\mathbb{Q} $ to represent all points on a straight line then we are in deep trouble. The existence of a point corresponding to square root of $2$ is guaranteed by Pythagoras theorem but such points (including all points realized via geometric constructions) are not the only ones on the number line which do not belong in $\mathbb {Q} $ rather there are many more of various kinds.
For example we can try to imagine the existence of a point $a$ such that $a^3=2$. Such a number is not available in $\mathbb {Q} $. But instead of solving $a^3=2$ we can look at inequations $a^3<2$ and $a^3>2$. This leads us to study the partition of $\mathbb {Q} $ into two non-empty disjoint subsets $A$ and $B$ each corresponding to numbers satisfying these inequalities. Dedekind's idea is that as we try to take larger and larger numbers in $A$ and smaller and smaller numbers in $B$ their cubes get closer and closer to $2$. And then Dedekind realizes that the key here is not the algebraic equations and the related inequalities but rather the partitioning of $\mathbb {Q} $ into two sets $A, B$ such that they are non empty, disjoint and exhaustive and further every member of $A$ is less than every member of $B$.
He studies such partitions in detail and shows that there are only three possibilities when we make such a partition:
- $A$ has a greatest member
- $B$ has a least member
- Neither $A$ has a greatest member nor $B$ has a least member.
These possibilities are mutually exclusive and exhaustive. The first two possibilities show that as we move from set $A$ to set $B$ based on ordering there is boundary point which lies at the end of $A$ or the start of $B$ and this boundary point is such that all numbers less than it lie in $A$ and all those greater than it lie in $B$. The third possibility gives us no such boundary point.
Dedekind then says that this is a defining characteristic of the idea of a geometric straight line in the sense that if we cut the line into two parts via a point then exactly one of the two parts must include that division point. This is not exactly a theorem derived from the axioms of Euclidean geometry but Dedekind feels that this is what should be the intrinsic nature of a straight line if it is supposed to be made up of a series of points such that one can go from one point of the line continuously to another point of the line. This is based on the belief that a line is connected / continuous / has no gaps.
And as mentioned above the system of rationals is not continuous / connected / gap-less in the way a straight line is and thus can't represent all points of a line. Dedekind says that the first two possibilities while partitioning the rationals correspond to the rational number which is boundary point of the partition. And the third possibility leads us to a new kind of a number called irrational number which is supposed to act as a boundary point.
Dedekind gives a name to such partition of rationals into two sets: a cut. And he develops the notions of order relations and algebraic operations on such cuts. The arithmetic which evolves out of all this exercise matches the arithmetic of rationals when the cuts correspond to rationals. And thus we already have an expansion of numbers because there are cuts which don't correspond to rationals. This is how Dedekind constructs the real number system $\mathbb{R} $ as a set of cuts.
And then he shows that the final goal of the expansion is achieved. When one tries to make a cut by partitioning the reals into two sets $A$ and $B$ in analogous manner then there is always a boundary point between the two. And the system does not have gaps like $\mathbb {Q} $ had and it can be used to represent all points of a straight line.
Most modern presentations of Dedekind's approach (especially those which appear in real-analysis textbooks) are totally unmotivated and are written as if author is highly disinterested and is doing so only as a formality.
The writing of Dedekind shows how all this is developed from scratch and gives lot of intuitive explanations. IMHO understanding the construction of real numbers from scratch (ideally before you have heard of any calculus related terms like limits) is essential for a thorough study of calculus/real-analysis and the effort is very rewarding.