What are the drawbacks to setting git's submodule.recurse config option to true?

This option was introduced in commit 046b482, initially for working tree manipulating commands (read-tree/checkout/reset)

git grep/fetch/pull/push soon followed.
However, as the documentation mentions, unlike the other commands below clone still needs its own recurse flag: git clone --recurse-submodules <URL> <directory>.
See this recent discussion:

This was a design decision once it was introduced, as the git clone might be too large.
Maybe we need to revisit that decision and just clone the submodules if submodule.recurse is set.

As the number/size of the submodules invoved can be potentially large, the default behavior is, for now, to not include them recursively by default.

Them main drawback is the possible time overhead introduced by having to go recursively in each submodule (and their own respective submodules).
If you have many of them, and don't need all of them, it is best leaving that option off, and specifying --recursive when you need it.

Yet, one advantage is to avoid seeing "Untracked files" when switching branches, as seen in this discussion.