Why can't I use git merge --squash with --no-ff?

It probably doesn't let you because such a command wouldn't make sense.

The documentation for --squash says (emphasis mine):

--squash
Produce the working tree and index state as if a real merge happened (except for the merge information), but do not actually make a commit or move the HEAD, nor record GIT_DIR/MERGE_HEAD to cause the next git commit command to create a merge commit. This allows you to create a single commit on top of the current branch whose effect is the same as merging another branch (or more in case of an octopus).

The --no-ff flag does:

Create a merge commit even when the merge resolves as a fast-forward.

You are essentially asking git to make a commit and NOT make a commit at the same time.

If you want to preserve all of the history of your branch, you should use the --no-ff flag. Commit d is a merge commit that has two parents, a and c.

a ------ d -- ....
 \      /
  b -- c  

If you want all of the commits on that branch to be treated as a single unit of work, then use --squash. Commit d is a regular commit that contains all of the changes from commits b and c but has only one parent, a.

a ---- d -- ....
 \      
  b -- c  

What does --squash do? - it squashes all the commits together, creates your index state such that you can commit all the changes from the branch as a single commit. The change in your code will be like you merged the branch, but your history will be like you made one huge commit.

What does --no-ff do? - it says, even if the branch can be fast-forwarded, do treat it as a merge and create a merge commit.

As such, --squash is about getting a history without a trace of merge. --no-ff is about forcing a history with merge, even when it is being forwarded. As such, they are mutually exclusive and cannot be used together.

I think the question arises from a misunderstanding of fast-forward. It is not a merge; it is just about forwarding a branch directly to another commit from the merged branch because the commits were made directly on the commits in the former branch. No fast forward stops this behaviour and forces a merge commit on top. When you are squashing, it doesn't matter if you are asking it to not fast-forward. It would be the same either way.

This answer is not related to your actual question:

Why can't I use git merge --squash with --no-ff?

as that has been properly answered already.

But rather to make clear that what you were after is indeed possible with a slightly different approach: you would rebase interactive your feature branch first (e.g. to squash all your commits) and after that you could git merge --no-ff the branch on master.