How do we know that analytic continuation agrees with UV regulators?
Your example has no counterterms to cancel the 1/r singularity! This explains the discrepancy.
In a correctly regularized expression, the counterterms are not introduced in an ad hoc way but by renormalizing some constants in the original problem that gives rise to the series. These modify all terms in the sum and make the divergent part vanish for particular choices, so that the singular limit can be taken in the result. Having to discard an infinity without having been able to cancel it properly is always a sign of having made a mistake.
You might be interested in reading my tutorial paper ''Renormalization without infinities'' - a tutorial that discusses renormalization on a much simpler level than quantum field theory.
In general, to know what the result should be one must start with a well-defined expression whose limit is sought and then transform the parameters in this expression in a way that after the transformation the limit exists. Then the result should be regulator independent.
But if one doesn't have a finite context to start with there is no way to know what one should do. Detached from physical content, a divergent series is just that - divergent and hence meaningless. The series $\sum_{i=0}^\infty (−1)^i$ can be given any integer value by appropriately reordering and taking partial sums, and all these values are different from the value you get by interpreting it as a limit of a geometric series.
Thus it is only the context that makes the value of a divergent series possibly well-defined.
Note that there are some rigorous theorems (see Can we get full non-perturbative information of interacting system by computing perturbation to all order? for Borel summation) that when a $C^\infty$ function satisfies certain requirements then certain summation methods produce from the Taylor series with zero convergence radius the correct function. But to know that such a theorem applies one needs to know some nonperturbative theory that allows one to check the requirements of such a theorem.
Note also that every asymptotic series is known to be the Taylor series of uncountably many $C^\infty$ functions. Thus there is no unique way of summing an asymptotic power series; the result must depend on the properties assumed in addition to the asymptotic series.
In fact, analytical continuation in some sense is subtracting off the divergence using local counter terms. A very clear treatment has been made in Terry Tao's blog: https://terrytao.wordpress.com/2010/04/10/the-euler-maclaurin-formula-bernoulli-numbers-the-zeta-function-and-real-variable-analytic-continuation/ (See section 2)
Note the result of the regularization does dependent on the regulator you choose, and this is essentially explicit in Zhengyan Shi's answer. (The $f(0)=1 $ condition) In your case, the analytical continuation can be thought of using a compactly supported function $\eta(x)$ on $(0,1)$,with $\eta(0)=1$.Thus one can consider the sum $$\sum_{n=0}^{\infty}n^s\eta(\frac{n}{N})$$
In Terry Tao's blog, he treats it with general complex $s$, with $Re(s)>-1$, and it turns out that with $\eta(0)=1$, one can actually analytically continue zeta function by subtracting of the divergent piece in the asymptotic expansion of $N$, i.e. $$\zeta(-s)=\lim_{N\to \infty}(\sum_{n=0}^{\infty}n^s\eta(\frac{n}{N})-N^{s+1}\int_0^{\infty}x^s\eta(x)dx)$$ and it does not depend on what $\eta(x)$ one chooses as long as $\eta(0)=1$. One can see in this sense analytical continuation is subtracting of the divergence using counter terms.As mentioned above,the result depends on what $\eta(0)$ is.
For example, if we let $s=0$,and consider the sum $\sum_{n=0}^{\infty}\eta(\frac{n}{N})$, which has asymptotic expansion $$N\int_0^{\infty}\eta(x)dx-\frac{1}{2}\eta(0)+\mathcal{O}(\frac{1}{N})$$
The first term is the divergent piece and thus is subtracted off by "counter terms".The second term is the regulated sum.
For zeta regularization $\eta(0)=1$ and thus the answer is $-\frac{1}{2}$. For your $(1-r)^n$ regularization, in this language (you can off course cut your function off beyond $x=1$), is $\eta(x)=(1-r)^x$ for given $r$, and $\eta(0)=1$ as well. So if you consider the finite part of $\lim_{N\to \infty}\sum_{n=0}^{\infty}(1-r)^{\frac{n}{N}}$, you would indeed get $-\frac{1}{2}$ in agreement with the zeta regularization.
But instead of looking at $N\to \infty $ limit with finite $r$, the second method in your question looks at $r\to 1$ limit with finite $N$. The function $x^y$ is not continuous at $(0,0)$ so you are bound to get different asymptotic behaviour depending on which limit you take first.
So as a summary, 1) Zeta function regularization can be thought of subtracting countqerterms 2) Depending on how you choose the regulator and take the limit, you can get different answers, and there is no right answer a priori. The "right" regularization must take into account of the actual physical context. (such as not breaking various symmetries the theory has.)
I don't know much about QFT at all, but I have read some discussions of Casimir's effect in Matthew Schwartz's QFT textbook and I hope I can offer some valuable ideas about why the different regulators agree in some cases but not others.
Essentially, in Schwartz's treatment, there is a class of regulators that yield the same physical result, namely the $-\frac{1}{24r}$ term in the energy. More precisely, he derived the necessary constraints on the regulators.
Here are the details. Suppose the cutoff angular frequency in the hard-cutoff regulator is given by $\omega_{cutoff} = \pi \Lambda$ (the energy will be shown to be independent of the cutoff!). Then a general regulator $f$ turns the original sum:$$ E(r) = \sum_n \frac{\omega_n}{2} \quad \omega_n = \frac{n \pi}{r}$$
Into the sum: $$E(r) = \sum_n \frac{\omega_n}{2} f(\frac{\omega_n}{\omega_{cutoff}})$$
Using the general expression to calculate the total energy, we find two constraints that are necessary. First of all, if $$ \lim_{x ->\infty} xf(x) = 0$$ Then we end up with a nice and clean expression for the total energy (the detailed calculation is a straightforward application of Euler-MacLaurin Series): $$E_{total} = E(r) + E(L-r) = \frac{\pi}{2} \Lambda^2 L - \frac{\pi f(0)}{24r} + ...$$ If we then add the constraint that $f(0) = 1$, then the $r$ dependent energy term matches the experimental results. And the two constraints are not very strong at all. A wide variety of regulators satisfy these constraints. For example:
The heat kernel regulator: $f(x) = e^{-x}$
The Gaussian regulator: $f(x) = e^{-x^2}$
The Hard cut-off: $f(x) = \theta(1-x)$
For your case of $S$, you used the regulator $(1-r)^n$. Maybe you can write down more general regulators and derive some constraints to ensure consistency? That's a just a thought.
P.S. In the Casimir case, if you just started with analytic continuation on the series: $$ E(r) = \sum_n \frac{\omega_n}{2}$$ The regulator is simply $f(x) = 1$. That does not satisfy the vanishing constraint mentioned above. But a direct application of analytic continuation still gives you the correct result... this is confusing for me too. Hopefully, people who actually know QFT can add much more insightful answers.