Why is it incorrect to say, $\lim_{x\rightarrow a}f(x)\notin\mathbb{C}$?

Short Answer

There are times when it is appropriate to write $\lim_{x\to a} f(x) \not\in X$, where $X$ is some set of interest. However, if $f$ is a complex function, it is not possible for $\lim_{z\to a} f(z)$ to exist but not be in $\mathbb{C}$, hence the notation $\lim_{z\to a} f(z) \not\in \mathbb{C}$ is confusing and ambiguous. I would avoid this notation.

Discussion

Depending on how, precisely, the notation $\lim_{x\to a} f(x)$ was defined, there may be nothing wrong with writing $\lim_{x\to a} f(x) \not\in \mathbb{C}$ as a sort of rough synonym of "the limit does not exist as a complex number." This might be okay in the right context. However I, personally, don't like this use of notation, and I think that it is likely to cause some confusion. To explain in more detail, let's start with a basic definition:

Definition: Let $a, L \in \mathbb{C}$ and suppose that $f$ is a function which is defined on some ball centered at $a$ (though not necessarily at $a$ itself). If for any $\varepsilon > 0$ there exists some $\delta > 0$ such that $$ |z - a| < \delta \implies |f(z) - L| < \varepsilon, $$ then we say that the limit of $f(z)$ as $z$ approaches $a$ is $L$, and write $$ \lim_{z\to a} f(z) = L. $$

The notation is defined only in cases when the limit actually does exist. Hence when I write $\lim_{x\to a} f(x)$, I am already assuming that this limit exists. Of course, if for any $L \in \mathbb{C}$ I can find some $\varepsilon$ such that no $\delta > 0$ does the job required in the definition, then I can say that the limit does not exist, which I might write as $$ \lim_{z\to a} f(z) \text{ DNE} \qquad\text{or}\qquad \lim_{z\to a} f(z) \text{ does not exist.} $$ This is kind of an abuse of notation, but it is perfectly understandable in most contexts. Since the goal of mathematical writing is clear communication, we let it stand. Indeed, we already overload the notation a bit by considering infinite limits and limits at infinity[1], so it is entirely reasonable to use the notation $\lim_{z\to a} f(z)$ even when the limit does not exit in the sense defined above.

On the other hand, the notation $$ \lim_{z\to a} f(z) \not\in \mathbb{C} $$ implies something other than "the limit does not exist." Rather, it seems to say that the limit exists, but is not a complex number. In principle, such a statement could hold. For example, consider the sequence of rational numbers $$\left( a_0 = 1, a_1 = 1 + \frac{1}{2}, a_2 = 1 + \frac{1}{1+\frac{1}{2}}, a_3 = 1 + \frac{1}{1+\frac{1}{1+\frac{1}{2}}}, \dotsc \right). $$ In each term, replace the fraction $\frac{1}{2}$ with $1/(1+\frac{1}{2})$. Each term in this sequence is rational. However $$ \lim_{n\to\infty} a_n = \varphi = \frac{1+\sqrt{5}}{2}, $$ which is not a rational number. So the limit of this sequence exists, but is not a rational number. Therefore $$ \lim_{n\to\infty} a_n \not\in\mathbb{Q}. $$ This notation implies that the sequence has a limit, but that this limit doesn't live in the set $\mathbb{Q}$. Similarly, if we write $\lim_{z\to a} f(z) \not\in\mathbb{C}$, this implies that the limit exists, but is not a complex number.

But this is nonsense!

The complex numbers for a complete metric space. I'm not going to go into details about what this means, but it implies that if $\lim_{z\to a} f(z)$ exists, then it must be a complex number[2]. Therefore, per the definition written above, it is not possible for $\lim_{z\to a} f(z) = L$ to exist, but for $L$ to not be a complex number. As such, the notation $\lim_{z\to a} f(z) \not\in \mathbb{C}$ is confusing and ambiguous. On the one hand, it asserts that the limit exists. On the other hand, it asserts that the limit is not a complex number. These two statements contradict each other, which is confusing. Thus it is best to avoid this notation.


[1] ...and then we learn more mathematics, learn about the extended real numbers, the Riemann sphere, the Alexandrov compactification, and other topological ideas which cure this overloading, but that is neither here nor there.

[2] As I have defined the limit, if $|f(z)|$ grows without bound as $z \to a$, then the limit does not exist. In this case, we might write $\lim_{z\to a} f(z) = \infty$ and say that the limit is infinite. However, per the definition written above, the limit does not exist.