Why should we believe in the axiom of regularity?
Regularity (aka Foundation) can be seen philosophically as an axiom of restriction. It is not necessarily saying “all the things you consider as sets must be well-founded”. It can be read saying “for the purposes of this set theory, we restrict our universe of discourse to just the well-founded objects”. It’s clarifying what we mean by sets, in a similar way as the extensionality axiom does.
You may find this explanation unsatisfying, since it’s fairly similar to what Maddy gives. But the point is that if you are philosophically unsure about it, the question to ask is not “Are all sets really well-founded?” but “Is it really convenient/harmless/natural to restrict attention to the well-founded sets?”
A precise statement which can be seen as justifying this is the fact that within (ZF – Regularity), one can prove that the class of well-founded objects is a model of ZF.
Edit: see this followup question and its answer for:
a rather stronger sense in which regularity is harmless, in the presence of choice: ‘Over (ZFC – regularity), regularity has no new purely structural consequences’
a counter-observation that in the absence of choice, over (ZF – regularity), it’s not so clearly harmless; it has consequences that can be stated in purely structural terms, such as ‘every set is isomorphic to the set of the children of some element in some well-founded extensional relation’.
I think of the axiom of regularity along with the axiom of extensionality as formalizing what I mean by "set". Once upon a time, before paradoxes, one could think of sets as just any collection of things. Unfortunately, axioms based on that picture, in particular the unrestricted comprehension axiom, led to contradictions, so it became clear that the original, contradictory notion of "set" must be replaced by something clearer. (People might have thought the original notion was perfectly clear, but the paradoxes show that it isn't.) The clearer picture that emerged (in a development beginning with Russell's type theory, and continuing through simple type theory) is of a cumulative hierarchy, in which sets are obtained as follows.
Begin with some non-set entities called atoms ("some" could be "none" if you want a world consisting exclusively of sets), then form all sets of these, then all sets whose elements are atoms or sets of atoms, etc. This "etc." means to build more and more levels of sets, where a set at any level has elements only from earlier levels (and the atoms constitute the lowest level). This iterative construction can be continued transfinitely, through arbitrarily long well-ordered sequences of levels.
This so-called cumulative hierarchy is what I (and most set theorists) mean when we talk about sets. A set is anything that is formed at some level of this hierarchy. This meaning of "set" has replaced older meanings.
The axiom of regularity is clearly true with this understanding of what a set is. It expresses the idea that the stages of the cumulative hierarchy come in a well-ordered sequence. (Without well-ordering, the instructions for each level, namely "form all sets whose elements are at earlier levels," would not be an inductive definition but a circularity.)
Although there are set theories that contradict regularity, I would say that any such theory (and also any theory that contradicts extensionality) is not about sets but about some different (though presumably similar) entities.
I feel that the Regularity is close in spirit to the Extensionality, and together they convey the idea that identity of a set is determined only by its elements. With the Extensionality alone (without Regularity) there could exist sets $x=\{x\}$ and $y=\{y\}$ such that $x\ne y$. Both sets have seemingly identical structure $\Big\{\big\{\{...\}\big\}\Big\}$, but still are not equal. How many different sets with this structure exists? Seven? A proper class? Who knows... This would be very strange and counter-intuitive universe. The Regularity rules out such things.
I'm reading Logical Foundations of Mathematics and Computational Complexity by Pavel Pudlák (DOI 10.1007/978-3-319-00119-7), and a similar reasoning can be found there on p. 170, "Cleaning Up the Universe". Let me quote:
$\hspace{.5cm}$ But let’s get back to set theory. The cleaning process that we are going to consider has little to do with that theory. It is rather related to the well-known Occam razor which suggests getting rid of all unnecessary concepts. Following Cantorian tradition, it is unpopular to prohibit something in set theory. If a set can exist, then in “Cantor’s Absolute” the ideal world of sets, it does exist. Hence, by forbidding some sets, we get narrow-minded, and decide to study only a part of reality. Still there are sets which most set theorists give up voluntarily. Consider, for example, a set $x$ which has a unique element which is itself; so $x = \{x\}$. Let $y$ be another set with the same property. By extensionality they are different because they contain different elements $x\ne y$. If we take the elements of their elements, it is the same and so on. Structurally they are the same, but still they are different. The axioms considered so far do not exclude such sets, but such sets will never appear in the cumulative hierarchy of sets $\{V_\alpha\}_{\alpha\in ON}$, where $ON$ denotes the class of all ordinal numbers. On the other hand, those which are in the hierarchy are nice, as they are in some sense constructed from the canonical set $\varnothing$. Therefore, we prefer to have:
The Axiom of Foundation There are no sets outside the cumulative hierarchy.