SQL Server Management studio slow connection or timeout when using Windows Authentication

Try running a trace with SQL Profiler while you, and then your coworker, connect to the server.
Select RPC, SQL Statement & PreConnect - Starting/Completed.
Select Save Results To Table option, then compare the 2 tables to find the bottleneck.

Or, since you're connecting by IP, it could be doing a Reverse DNS lookup. If so, add a entry in your hosts file.

What you should check first is your server or client DNS settings

It's not the rare that your SQL Server have the problem connecting to Active Directory. If you try with local Windows account I am sure that you will not have the at issue. It's not unusual that server is configured with public internet DNS and when SQL Server connects to DC to check the credentials and verify it, it will try top contact the public DNS instead of the DNS server of the AD. Since this information is not stored on the public DNS it will fail to verify and this will cause the delay until it manage to contact the proper DNS server or DC via the NTLM

Since you are not experiencing the problem with other SQL Servers, than almost certainly the issue is not related to AD or DC configurations

Fire the IPConfig.exe /all command from the cmd to check the configured DNS servers. You should have only AD's DNS servers configured. Remove all public DNS servers, and leave just AD's DNS servers.