Reviewing papers at a journal where your own work is currently submitted
I think this is normal.
Because of the power imbalance
I think the power imbalance is a bit overstated here. Editors want good papers for their journal; a good editor is not primarily just farming out the job of peer review with as little effort as possible.
If I...
1) submitted a paper,
2) received a request to review a paper,
3) denied that request for good reason, and then
4) had my paper rejected by the editor,
...either those outcomes are unrelated and the paper should be submitted elsewhere, or the journal is crap and the paper should be submitted elsewhere.
it seems as though the author might feel compelled to agree to review the paper even if they didn't otherwise want to.
I think this is not really a very crucial ethical issue, although I am not in mathematics where maybe the burden of review is greater. Certainly if this compelled someone to review a paper they were not qualified to review, that would be a problem, but I don't see this being a substantial issue.
If you are submitting papers you should also be reviewing them. If all papers were single-author papers, then the number of papers you review should be greater than the number you submit. If a journal is in your area of study such that you submit papers to it, you are among the people in the best position to review other submissions applicable to that journal.
Or maybe they will feel an instinct to say the paper is terrible if they believe fewer other papers being accepted might increase their own chances
I can see this being an issue for grants where there is a limited pool of money and a zero-sum game. I don't know of any journal where it is a practice to rank order articles and then reject one for publication because it wasn't the 12th best article submitted this month. Journals just don't operate on that time scale, and if a journal did decide to be more or less permissive due to overall submission and acceptance rates, that would be over a longer time scale than the review of a single paper.
In summary, feel free to feel annoyed: feelings don't necessarily require justification. However, I don't think there is an ethical violation here.
Let's take the null hypothesis that this is ethically fine, and then examine reasons why the hypothesis might be wrong. You raise two:
Because of the power imbalance, it seems as though the author might feel compelled to agree to review the paper even if they didn't otherwise want to.
This is a legitimate concern. However I think it's fair to say that most editors have seen so many reviewers decline to review that they won't bat an eye at yet another reviewer who declines.
Of course you should give a reason for declining. Something like "I currently already have 2 manuscripts to review for other journals" is more likely to lead to empathy than derision. If you don't feel qualified to review that's also a fair reason. I'm confident most editors will prefer a good review over a poor one, even if they have to work a bit harder to get that good review.
Is it possible the editor will reject the paper because the author didn't want to review another paper? It's conceivable, but not likely: ultimately declining to review is quite minor, and the editor will want good papers. It would take a pretty annoyed editor to reject a good paper because he doesn't like the author (see this question and the reactions of most people to the would-be annoyed editor).
Or maybe they will feel an instinct to say the paper is terrible if they believe fewer other papers being accepted might increase their own chances.
This on the other hand isn't really justifiable. If a paper is rejected it's far more likely that it didn't meet the minimum standards required for publication in the journal than it is because the journal has run out of space. In the latter scenario, it's much more likely the journal will hold the paper to the next issue, and if the next issue is also full, then it might consider increasing the number of issues a year.
tl; dr: I don't think there's a ethical concern here. If it troubles you as an editor then don't, but there are reasonable people who aren't troubled, both as an author and as an editor.
This is normal, and even expected, especially for conferences.
Because of the power imbalance, it seems as though the author might feel compelled to agree to review the paper even if they didn't otherwise want to.
Few people really want to review papers; doing it well is a lot of work for relatively little direct payoff (esp. compared to just reading a published paper). However, it's understood to be an important part of participating in the scholarly community, and the golden rule (classic definition) applies. By submitting a paper to the journal, an author is clearly expressing a desire to be part of whatever very specific scholarly community that journal serves. It is quite reasonable for an editor to request that a submitting author become a reviewer, if the editor believes they could do a reasonably good job of the review. (If the author's submission was crackpot nonsense or way off topic, they may not be a good reviewer candidate.)
Or maybe they will feel an instinct to say the paper is terrible if they believe fewer other papers being accepted might increase their own chances.
If they want to be persuasive, they will need to say specifically why the paper is "terrible" and if you reject what they are reviewing it should be because those are valid points. Similarly, if you as an editor reject what that reviewer authored, it should be because of valid shortcomings pointed out in the reviews of that paper, and a completely independent decision made without considering whether they agreed to review other papers.
Authors submitting to or published in a particular journal have an incentive to set the bar for that publication to a high level (so as to enhance the reputation of their own work, by association). Some authors may wish to qualify this, setting the bar high but not so high that their own work can't get in. Ultimately, it's up to you as the editor to decide where the quality bar is for that venue, and the reviews are just input to guide you.
I am an editor myself and I never do this on principle.
As long as the venue does not have any specific policy against doing this, I would have no qualms about it, as long as I know my editorial decisions are separated from authors' responses to review requests, and that the content-based reasons for the editorial decisions are clearly expressed in the decision letter.
As an author it has happened to me on a number of occasions, and I'm not sure how justified I am in finding it so annoying.
As an author, I don't find any extra annoyance from a review request because it comes from a venue where I've submitted. The opposite is actually more likely, especially when I'm a new submitter to a venue that I believe to be high-quality: I take it as a step of being welcomed into that scholarly community as a qualified peer.
I'm also considerably more likely to accept review requests from venues where I've submitted, because I've already reached the decision that this is a venue worth supporting. I'm also not nearly as likely to lump such a request in with the academic spam clogging my inbox with requests to review or edit for what I assume are generally low-quality venues looking for good names they can use to build their reputation and/or further a scam.
Bonus question: Is asking for a "quick opinion" (rather than a full review) in this context more or less appropriate than asking for a full referee report?
It depends: is that a normal step/activity in the venue?
You might get better full referee reports from reviewers who also have work submitted at that journal, and/or who recently received good-quality reviews (positive or negative) from a submission to that journal; such reviewers may be more cognizant of the golden rule when doing their review.
As an editor, I'd be more inclined toward requesting full referee reports where possible and appropriate, only asking for the "quick opinion" reports if I were needing more of those reports for a step in the venue's process, and lacking in reviewer candidates likely to give those but unlikely to do full referee reports.