What makes academic failure different from failure in another career?
Many other careers are not structured around "success" and "failure" in the same way that academia is, and the jobs that are are acknowledged to be high stress. I agree strongly with what Anyon mentions about the pyramid structure of academia, as well as survivorship bias. Jack Aidley also nicely captures the difference that "In most careers you do not 'fail' you simply cease being promoted." Here are a few other ways that the idea of failure is omnipresent in academia. (This is from a U.S. perspective, and I do not know how widely it pertains elsewhere.)
First, success or failure is very personalized and visible in academia. Academics are not very interchangeable: like artists and few other professions, everyone does the job differently and produces creative output. It is often hard to separate yourself from your work (whether being critiqued or praised). Further, on the research side each person is essentially an entrepreneur who is their own publicity, branding, fundraising, management (if there's a lab or students to supervise), R&D, and administrative department. There's a lot of personal investment in this job, and failure cuts deeper into the sense of self than in other jobs.
Second, in other jobs, most people are rarely evaluated by strangers, and often the people evaluating them are invested in their success. Performance evaluations in offices are usually with co-workers and they are often two-way. Managers and HR folks often care about improving the workforce they have, rather than going through the hiring process again. In contrast, the entire tenure process seems designed to give people anxiety (which a professor complained to me about when I was a grad student--he was worried another professor was working against his tenure case). The tenure process is a years-long audition, after which you can be essentially told that you no longer have a job in whatever tiny college town you uprooted yourself to live in. In most industry or government jobs, there's much more flexibility to move between jobs (not just one big annual switch), less stigma about switching jobs, and often lots of small evaluations rather than a few huge hurdles.
Third, academics need to "hustle," constantly seeking new opportunities for funding, research materials, ideas, and labor. Both applying for funding and submitting papers are processes with routine failure built in, and it can be somewhat opaque what is being sought. (Not to mention that the content of research may turn out not to lead anywhere interesting, after all.) However, many jobs don't have such stark success and failure. Annual sales income is a continuous number, or the quality of a project is within a range of acceptability. In many occupations you can gain or lose clients, but usually no one client is crucial. In the industries where grants or contracts are key sources of funding, there are large teams devoted to the project, and often a lot of projects in the pipeline. In contrast, academics are often pretty much alone in their success or failure, and for someone "on the tenure clock," how a particular project turns out--which may have a large random component to it--may change their tenure case substantially.
Fourth, professors are hired not only for what they can do right now, but an entire track record of what they have done and how they are likely to adapt over time in the future. Especially in combination with the large number of people with PhDs competing for a small number of tenure track jobs, this rewards people who have very linear career paths, moving from success to success. Many people worry about deviations from the norm, as well as about failures.
Some of these components are present in other jobs. I mention these other jobs not to compare who has it better, but to point out that things like high-stakes evaluation based on partly random outcomes is not a universal trait of many jobs, while it characterizes investment banking and (I argue) academic research.
Investment bankers are often individually held responsible for their performance, a lot of which relies on investment markets, which have a large random component.
Writers face a lot of the same problems with submissions and rejections, but writing is very rarely their main job. (That has its own problems...)
Salespeople have to deal with a lot of rejection and often work on commission. Their job and their salary are contingent on their success. However, there are low hurdles to switch in and out of sales, and often people can hedge their bets by working part-time at another job. (The precarity that goes with sales and service jobs is also part of what adjunct instructors--aspiring academics--face.)
Doctors need to pass a lot of hurdles on the way to their profession, and not getting past a hurdle may mean that one is saddled with lots of debt without a lucrative career. Then, depending on which field of medicine, failure may mean a patient dies or has bad complications. These sorts of consequences have a legal and financial complications, on top of the large emotional toll. Doctors may also have to relocate nationally or internationally if/when they switch jobs.
- Founders of start-up businesses will have many of the same pressures as professors (the same people must develop and execute ideas while securing funding, with large personal investment). However, failure is a celebrated part of the culture in, say, Silicon Valley, and the potential financial rewards are tremendous. (The restaurant industry also has serial entrepreneurs, though with less potential payoff.)
While each of those jobs mentioned above has a lot of judgment of and consequences around failure, those descriptions hopefully remind you that these extreme conditions around failure are largely NOT present in many other common jobs (service industry, teacher, programmer, manager, IT person, mechanic, engineer, nurse, technician, counselor, administrator, banker, accountant, office clerk, interpreter...).
I think the reason is simply that a lot of people go into academia with the, often unrealistic, goal of becoming a professor. No matter the circumstances, if you don't get to the point where you want to be, well, you can call it a failure, can't you?
This is aggravated by the pyramid structure of academia, where there are fewer positions available for every stage of the career, and most early-career positions are temporary. So you are statistically unlikely to reach your goal in the first place, and if you don't manage to do it during the standard time period, well, going to e.g. industry is your best bet. Outside of academia, on the other hand, if you don't get that managerial promotion or raise this time, maybe there's another chance coming a couple of years down the line. And you can keep working on similar things in the meantime.
So, the 'failure' in industry is likely less permanent, causing less anxiety. On top of that, you have the survivorship bias due to those who got their permanent positions. It affects the academic atmosphere, and can influence what is seen as success and failure. This is also affected by the field: Compare say petroleum engineering and mathematics PhD candidates, and you'll likely see different attitudes about private jobs.
I don't think anyone will be able to answer this question for certain. Here's my personal guess.
Many academics have never left academia before. Instead, they've gotten where they are by excelling in the traditional school system. They topped their high school class, their undergraduate class, and then their graduate class. At all stages they saw less-competent (for want of a better word) colleagues drop out and do something else. To fail to become a professor, then, is to acknowledge that one is also "less competent". For people who've always been the best among their peers, this is unacceptable. Hence they feel academic failure is different.
I suspect that people who have left academia before are much less concerned about academic failure. They have already left academia before, and are less afraid of doing it again.