How to prevent a specific branch from being merged in git?

It's possible with Git Hooks. Put the following POC script to .git/hooks/pre-receive on your remote (server-side) repository and give it right permission to execute.

Configure the branch you want to protect, for example master

$ git config hooks.protected-branch.refs master

File: .git/hooks/pre-receive

#!/bin/sh

read old_value new_value ref_name

refs=$(git config hooks.protected-branch.refs)

for ref in $refs; do
    if [ "$ref_name" == "refs/heads/$ref" ]; then
        if [ "$old_value" == "0000000000000000000000000000000000000000" ]; then
            continue
        fi

        if ! git merge-base --is-ancestor "$ref_name" "$new_value"; then
            echo "$ref_name is protected branch"
            exit 1
        fi
    fi
done

When you try to reset master by force-push, You will get similar output like this:

Counting objects: 12, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (12/12), 920 bytes | 153.00 KiB/s, done.
Total 12 (delta 4), reused 0 (delta 0)
remote: refs/heads/master is protected branch
To ../demo
 ! [remote rejected]   master -> master (pre-receive hook declined)
error: failed to push some refs to '../demo