Can I specify a port for certain IP address in SSH's config?

Sure, you can put these settings in ~/.ssh/config.

Host 192.168.1.45
Port 2222

You can take this opportunity to give the machine a nickname, and set other options, in particular the remote user name. For example, the following declaration makes ssh foo equivalent to ssh -p 2222 [email protected] (Host foo means that the following options apply when you run ssh foo, and the HostName directive makes foo a nickname for 192.168.1.45 with all the other options):

Host foo
HostName 192.168.1.45
Port 2222
User matt

You could add the host to your ssh configuration file. Run man ssh_config from a terminal for some nice reading.

In short place

Host <hostname>
Port 2222

in your ~/.ssh/config