Why does editor reject when reviewers recommend acceptance?
First, a note: the editor (or editors, or editorial committee) is solely responsible for the decision to publish or reject a submitted manuscript. Reviewers are often (not always) called in and their reports provide help to the editor in evaluating the manuscript. But, at the end of the day, it is the editor who makes the call, so the situation you describe is possible and not necessarily unethical.
It is, however, very unusual, both for the editor not to follow the unanimous recommendation of the reviewers, and even more so not to explain their rejection any further. You can definitely (and, in my opinion, should) contact the editor to:
- express your surprise at the rejection, given the contents of the reviews;
- ask whether it may have been an administrative error (with the nice streamlined web-based editor interface, a simple misplaced click might have lead to the current situation);
- if the decision is deliberate, ask the editor if he may expand on the reason behind the rejection.
Of course, be professional and polite.
If after contacting the editor you are not satisfied with the answer, the only way forward is to appeal the decision to the editor-in-chief or the full editorial board. Details on how to do so should be found on the journal's website. This should not be done lightly, but if you do not get a decent reply from the editor, it seems warranted by the facts of your case. You may, however, want to consider sending the manuscript to another journal, as it will take much less effort than the appeal…
I should also point out that one issue could be related to being submitted to " journal special issue related to that conference".
While that is the perfect place for the paper, keep in mind that in a regular issue, an accepted paper appears in the first issue which has space. For the special issue, all papers must appear in that issue. If the editors ended up with 5000 pages of accepted articles, they had to trim them down and reject some despite the good reviews.
As your paper had good reviews, it should probably be easy to publish it in some journal, maybe the regular issues of the same one?
I fully agree with F'x but would like to add the following. As stated editors are free to make the judgement they see fit with the journal reputation in mind etc. However, an editor should also wed out manuscripts that do not fit the journal's scope etc. already before the review stage. It therefore seems even more strange that a manuscript reeives such a drastic result from a seemingly good response from reviewers.
Another issue is that reviewers provide comments directly to the editor. It is thus possible that the review seems good but the reviewer may voice some concern that he/she feels the editor must act upon and which they therefore cannot voice in the open review. The editor can react to this. I would still expect the editor to then provide more open concerns so as to provide you with feedback.
All in all F'x's suggestion to contact the editor is the way to go. Describe that you do not understand the resulting "verdict" and that you would like to get more feedback so as to possibly improve a seemingly already good manuscript or possibly be given the opportunity to revise for a new final decision.