Writing thesis that rebuts advisor's theory
There is no predicting personality without close study, but if the person is a true scholar they will welcome an advance, even if it refutes something they did earlier.
No, hostility isn't an unreasonable possibility. But whether it is likely depends on the person. Impossible to say how likely. You are in a better position to judge, knowing them. Are they insecure? Do they welcome a challenge? Are they fair and honest?
But it would probably be better to approach them with a questioning attitude rather than one making bold claims. After all, you might be wrong, as you say. And you should be sure of the wider literature. Has the theory been validated by others? Accepted as fact?
The danger lies in the collateral damage that your refutation (and better, probably simpler hypothesis) does: Your prof might have other projects running which rely on his original theory.
Check if that's the case, and discuss with the people who are working on it.
Your prof and the others will either stubbornly defend their original approach (because their work&paychecks are threatened), or they see that it's dangerous to go on with a bad approach, which might end in rejected papers, failed theses, and loss of followup projects, and are glad you found out and want you to help fix it. (All assuming you're right with your analysis.)
If there is not much money riding on it currently, the prof might just be delighted to advise you on a thesis. If the old theory was widely know and used, this will mean he (+you) can write a lot of new papers which will surely get highly cited.
Or of course he might tell you to brush off with your stupid ideas and find another advisor. Well, at least then you know about that.
My answer doesn't speak directly to the situation with your potential advisor: I think that depends on your presentation of your approach, and the advisor's personality (I agreed that it does have the potential to backfire). I want to discuss the challenges you are likely to experience with this situation
his theory is relatively unexplainable ... [my] alternative theory ... would explain the exact same phenomena, but in a more justifiable way
If the two frameworks explain the exact same phenomena, then which version you (or your potential advisor) prefer is a matter of taste, or style. "More justifiable" is a value judgement. As an example, this issue is IMO) largely what drove the loud arguments in the evolutionary biology community over Nowak, Tarnita and Wilson (2010)'s theoretical reframing of the evolution of eusociality (species with extreme cooperation where only a small number of individuals in the group reproduce). Nowak et al. say their method
represents a simpler and superior approach, allows the evaluation of multiple competing hypotheses, and provides an exact framework for interpreting empirical observations
... but they do not say "explains previously unexplained phenomena" or "makes different empirically testable predictions". Their approach may indeed be "better" on some aesthetic grounds (mathematical rigor, simplicity, explainability [at least for some subset of potential users]), but if there is no concrete advantage over the previous approach, then many researchers will resist giving up the old way (and not without some justification).
Disclaimers:
- I have not read the Nowak/Tarnita/Wilson work, or the rebuttals, carefully enough to form my own opinion about which I prefer. However, I think my characterization of the political/philosophical issues here is reasonably accurate
- reframings that give a different perspective on the same set of empirical observations can be interesting and valuable, and may lead to deeper/novel insights; it's just hard to argue for them a priori
Nowak, Martin A., Corina E. Tarnita, and Edward O. Wilson. “The Evolution of Eusociality.” Nature 466, no. 7310 (August 2010): 1057–62. https://doi.org/10.1038/nature09205.