Paper rejection based on unfair review
I feel the only way out is to resubmit it for another conference in near future.
That about sums it up. Negative reviews and rejections are hard to swallow. They are common in academia and you cannot let them get to you. The best thing to do is step away from the manuscript. If you need to cry, cry. If you need to scream, scream. After a while, generally for me a week or so, try and look at the reviews again. If they still piss you off, step away from them again. Once you can read the reviews objectively, start to tackle them. This might mean agreeing with the reviewer, or seeing where the reviewer got lost, or that the reviewer is simply a fool. Once you have taken everything from the reviews that will improve the manuscript, file it away and if necessary cry and scream again. Then resubmit the manuscript to the next conference.
The possibility to have unfair reviews is one of the reasons why you receive multiple reviews. I have received multiple reviews where the reviewers clearly didn't review the paper carefully, and it's indeed quite frustrating.
That being said, when I looked back at the papers, I realised that they were not rejected because of one unfair review, but because I didn't manage to convince the fair reviewers: if you get four accept, then it does not matter if you have an unfair reject. Besides, it could also be the case that you've received an unfair accept review (i.e., someone who accepted the paper without really reading), so, in the end, it might balance things out.
Dealing with rejection is a huge step of the academic process, and as it has been said here in the past, if you never get papers rejected, you are probably not aiming high enough. As long as you have only one unfair review, ignore it, and move on to the next conference (and if you only receive unfair reviews, well, you are probably submitting at the wrong venue).
As you said, the paper was on the borderline after 4 (fair) reviews. I would advise first to discuss the paper and the results with your supervisor / colleagues, and if you are still determined to continue, rework it significantly, taking in account the criticism of four referees, that you consider as fair. Make sure the result moves far from the borderline (and in the right direction). Then re-think the remarks of the last ("unfair") referee and decide what you want to do with them. Probably, you should not trash them away completely — try to find a motivated comments and suggestions apart from the "unfair" ones.
When a new version of a paper is ready, you may wish to show it to your supervisor and send to several colleagues outside your group asking them for some pre-submission comments — this may save you a lot of time.