Bash scripts extremely slow to start on OSX

I had the same issue on my work machine, running macOS Sierra 10.12.5 and 10.12.6, and I confirmed that @charles-duffy is right. My company's anti-malware is the problem. They are using a daemon called Confer by Carbon Black for endpoint security.

Run this command to see if the problem stops:

launchctl unload /Library/LaunchDaemons/com.confer.sensor.daemon.plist

I think Confer is taking a fingerprint of every new program it sees and tries to compare it to a blacklist before allowing it to run. It looks like it times out after a second (trying to register the fingerprint online?), and something is making it try over and over again, yielding longer delays of integer seconds. This makes newly modified programs launch slowly, although speeds return to normal after the fingerprint has been taken.

---

Example runs

Here's how the problem manifested on my work laptop. As time went on, sometimes the delay increased, but always by an integer number of seconds at a time. If Confer remembers the program from before, there will be no delay.

bash scripts

$ bash --norc --noprofile -l  # make sure rc scripts don't interfere

bash-3.2$ echo exit > exit.sh && chmod +x exit.sh

bash-3.2$ time ./exit.sh

real    0m1.004s
user    0m0.001s
sys 0m0.002s

bash-3.2$ time ./exit.sh

real    0m0.002s
user    0m0.001s
sys 0m0.001s

There were no problems running commands provided to bash -c:

bash-3.2$ time bash -c exit

real    0m0.008s
user    0m0.002s
sys 0m0.003s

awk scripts

But the problem did affect awk scripts:

bash-3.2$ printf '%s\n' '#!/usr/bin/env awk -f' 'BEGIN { exit 0; }' > test.awk && chmod +x test.awk

bash-3.2$ time ./test.awk

real    0m4.010s
user    0m0.002s
sys 0m0.001s

bash-3.2$ time ./test.awk

real    0m0.005s
user    0m0.002s
sys 0m0.001s

Compiled binaries

It even affected compiled C and Golang code:

bash-3.2$ printf '%s\n' '#include "stdio.h"' 'int main() {' 'printf("Hello, world!\n");' 'return 0;' '}' > hello.c && gcc -o hello hello.c

bash-3.2$ time ./hello
Hello, world!

real    0m4.006s
user    0m0.001s
sys 0m0.001s

bash-3.2$ time ./hello
Hello, world!

real    0m0.004s
user    0m0.001s
sys 0m0.001s

bash-3.2$ printf '%s\n' 'package main' 'import "fmt"' 'func main() {' 'fmt.Println("test")' '}' > test.go && go build -o test ./test.go

bash-3.2$ time ./test
test

real    0m4.018s
user    0m0.001s
sys 0m0.003s

bash-3.2$ time ./test
test

real    0m0.005s
user    0m0.001s
sys 0m0.002s

Starting with 10.15, macOS might actually be talking to Apple every time a new script is executed. I found this in a recent changelog for TextMate:

Fixed: Delay when re-executing (bundle) commands on macOS 10.15. Starting with macOS 10.15 Apple will “call home” each time a new script/binary is executed, this can have a delay of more than a second, depending on internet connectivity/location. The result seems to be cached per inode, previously TextMate would use temporary files when executing scripts or shell commands (giving them a new inode on each run), it now re-uses these files to avoid the delay on repeated executions.