How shall I report a critical error in a given paper?
Talk to an experienced researcher in the field before getting too excited. For instance, you could talk to the professor of your course, if they are an active researcher in this area.
The question of what constitutes a "critical error" is subjective and often hard for a student to evaluate. For instance, perhaps the authors are just using conventions that you are not familiar with, or you have misunderstood what they are saying.
It can also happen that although the text of the paper is literally incorrect, an experienced researcher can see immediately what was intended, and won't be confused by it. These sorts of errors are typically not taken seriously and are not considered to warrant a formal correction to the paper, which requires a lot of work for the authors and editors.
It's good that you've discussed this with your TA, but your TA is likely a graduate student who doesn't yet have a lot of experience with research and publication in this area.
I think it's pretty common for students to get very excited about finding errors of any kind in published work, and to feel that "the world needs to know". I've had this experience myself. But community standards may not be what you think they are, and if you try to raise a big fuss about something that the research community regards as trivial, it'll just waste time and make you look a bit silly.
If an expert agrees that the error really is critical, then they can also advise you on how to proceed. Typically you would begin by contacting the authors, and they would talk to the journal editor about publishing a correction (or "corrigendum"). In the unlikely event that they don't acknowledge the error, you can also raise the issue with the journal editor directly. If the discovery of the error is itself something of significant scientific interest, then you could think about writing a new paper about it. But don't cross that bridge until you come to it.
At first, you could talk to your TA and confirm that these errors really exist. According to your description, this should be a rather quick and easy task since you have identified the error already.
As a next step, you (together with your TA) should contact the corresponding author of the paper in a friendly manner. Do not try to sound accusatory but rather be interested in their work.
However, I do not expect that such an error could easily be fixed in the paper. Once a paper is published no modifications are allowed usually. Perhaps they might add an erratum note or even retract the paper (if the error is significant).
I have verified and consistently proved that the error exists, as I can reproduce the paper results including said error. What are my options now?
I recommend co-authoring a manuscript with your TA that highlights and fixes the noteworthy issues that you have found. (Fixing the issues makes the contribution far stronger, but perhaps isn't absolutely necessary.) I recommend sharing this manuscript with the original authors prior to any publication/submission (as noted by J-Kun); the shared manuscript needn't be in publishable state, it just needs to be comprehensible enough for the authors to comment on.
I want someone to take notice and fix this paper, since I have proved it contains wrong results.
I recommend that you fix this paper, perhaps in collaboration with the original authors (this would reduce the burden on you). Your results demonstrate that the original work is unsuitable, so fixing the paper is a publishable result.
Answers to issues raised in comments:
Couldn't the manuscript just be an email? I don't agree that an erratum is a publishable result (if by that, you mean a research article worthy of journal publication).
The following issues have been noted:
- a mathematical model was wrongly written;
- flaws...of fundamental nature...in the model; and
- a pure error in one of the basic assumptions of the model [which] invalidates the reactor model and the subsequent optimization.
The first sounds like it belongs in an erratum, the second doesn't (albeit the accompanying fix "F = mv instead of F = ma" sounds like it does), and the third seems to suggest a non-trivial fix is required to derive a new reactor model and new optimizations. Thus, it seems like a new article is required. As an alternative, the original article could be extended (in collaboration with the original authors) and published as a technical report.