How do concepts such as limits work in probability theory, as opposed to calculus?

There are several notions of convergence in probability.

Your example is an instance of the law of large numbers which has a weak form and a strong form.

The weak form states that $H/(H+T)$ "converges in probability" to $0.5$. Formally, for any $\epsilon > 0$, $$\lim_{t \to \infty}P\left(\left|\frac{H}{H+T} - 0.5\right| > \epsilon\right) = 0.$$

The strong form states that $H/(H+T)$ "converges almost surely" to $0.5$. Formally, $$P\left(\lim_{t \to \infty} \frac{H}{H+T} = 0.5\right) = 1.$$ Almost sure convergence implies convergence in probability, hence the strong law implies the weak law (but is harder to prove).


Response to comment:

If you imagine flipping a coin many many times, then you can keep track of the sequence $\frac{H}{H+T}$ as $t$ increases. In your words, this sequence is random, since it "depends on what comes up," so in different parallel universes this sequence will be different. You are correct that it is possible that you always flip tails, in which case the sequence is $0,0,\ldots$. But given a particular sequence, you simply have a sequence of real numbers, so the limit in $\lim_{t \to \infty} \frac{H}{H+T}$ is the usual limit you are familiar with (which may not exist for some sequences). The law of large numbers states that with probability $1$, this sequence not only has a limit, but that limit is $0.5$. So scenarios like the one you mentioned (always flipping tails) happen with probability $0$.


In probability, you work with distributions. When you increase the number of trials and take the average, you get a distribution that tends to a Gaussian, with a narrower variance.

The conclusions that you can draw are based on this probabilistic inference. For an infinite number of drawings, the distribution tends to a Dirac delta.

enter image description here