MySQL Error: Authentication plugin 'caching_sha2_password' cannot be loaded

So, I found the running process of mysqld by sudo lsof -i tcp:3306 then I killed it using sudo kill -9 <PID>.

After this I tried mysql_secure_installation again, but ran into a new error :

Error: Can't connect to local MySQL server through socket '/tmp/mysql.sock' (61)

After trying a few fixes for getting mysql.sock to work, I started the MySQL server with sudo mysql.server start then went ahead with mysql_secure_installation to set password for the root user.

This worked for me finally.

Note: Homebrew asks you to first do mysql_secure_installation before starting the MySQL server for the first time but that just made me run into endless loop of errors.