Rate limit login attempts: count by IP or username

I would say that you should do both.

If you only rate limit on IP address, an attacker controlling a bot net could brute force an account with a weak password. If you have 10 000 computers with unique IPs and each one is allowed four attempts per hour you can try almost a million passwords per day.

If you only rate limit on username, an attacker with a list of existing usernames could brute force accounts with a weak password. Chances are that a few of your users use one of the top 10 passwords, so if you try those on all accounts you will get in somewhere.

Of course you could do all sorts of combinations. For instance you might only block a username once X different IPs have failed to log in with it, so that an attacker trying to block a user from logging in needs to work a bit for it.

Threat Model

It really depends on your threat model:

• How bad is a DOS attack against users in comparison to the compromisation of a user account?
• Does an attacker want to attack one specific user, or just collect as many valid credentials as possible? (in the first case you would want to block by username, in the second by IP).
• How easy is it to unblock a blocked username for a legitimate user?
• ...

Examples

Lets take ebay as an example:

• A DOS attack would be pretty bad. Someone could prevent all users that are competing on a bid from being able to log in.
• Of course, account compromisation is also not a good thing, but payment is secured (I think, it should be) by requiring some further information (for example, when paying via paypal, you would need to enter your paypal password). Additionally, banning by username only adds security if an attacker wants to attack a specific user. If they just want as many accounts as possible, blocking usernames doesn't limit attackers. Only blocking IPs does.

So in this example - access to the application is somewhat critical, and an attacker does not attack specific user accounts -, limiting by IP would be the right decision, and limiting by username would be damaging.

Lets take another example: An admin login for a web application:

• You likely don't need to worry about DOS. Yes, it would be a minor annoyance, but the admin likely has simple methods of regaining their account. Maybe the login page is also hidden, so an attacker would need to guess that first.
• An attacker only wants to attack that one admin account and they may have quite a few proxies. Blocking that account makes sense here.

So here, blocking by IP has a low effect on the attacker, while blocking by username would be the correct choice.