Is using SOFTFAIL over FAIL in the SPF record considered best practice?
Well, it was certainly not the intent of the specification for it to be used instead - softfail is intended as a transition mechanism, where you can have the messages marked without rejecting them outright.
As you've found, failing messages outright tends to cause problems; some legitimate services, for example, will spoof your domain's addresses in order to send mail on behalf of your users.
Because of this, the less draconian softfail is recommended in a lot of cases as a less-painful way to still get a lot of the help that SPF offers, without some of the headaches; recipient's spam filters can still take the softfail as a strong hint that a message may be spam (which many do).
If you're confident that no message should ever come from a node other than what you've specified, then by all means, use fail as the SPF standard intended.. but as you've observed, softfail has definitely grown beyond its intended use.
In my understanding, Google relies not only on SPF, but also on DKIM and ultimately DMARC to evaluate e-mails. DMARC takes into account both SPF and DKIM-signing. If either is valid, Gmail will accept the e-mail but if both fail (or softfail), this will be a clear indication that the e-mail may be fraudulent.
This is from Googles DMARC-pages:
A message must fail both SPF and DKIM checks to also fail DMARC. A single check failure using either technology allows the message to pass DMARC.
I therefore think it would be recommended to use SPF in softfail-mode in order to allow it to enter into the greater algorithm of mail analysis.
-all should always be used NO EXCEPTION. To not use it is opening yourself up to someone spoofing your domain name. Gmail for instance has a ~all. Spammers spoof gmail.com addresses all the time. The standard says we must accept emails from them because of ~all. I personally don't follow the standard on this, because i've realized most of you have setup your SPF records incorrectly. I enforce ~all, ?all, just as i would -all. SPF Syntax SPF mistakes