Why does redis report limit of 1024 files even after update to limits.conf?

You should edit your files inside /etc/pam.d/ directory.

In your case when you run sudo -u ubuntu /usr/local/bin/redis-server, you should add following line to /etc/pam.d/sudo or to /etc/pam.d/common-session-noninteractive in case /etc/pam.d/sudo includes this one:

session required pam_limits.so

This should help setting configuration provided inside /etc/security/limits.conf.

Another suggestion for users finding this post whose issue is not related to pam_limits.so. In my case, Redis was being launched via supervisord. In this case, the method by which supervisord switched user contexts caused the new session to be run with the system default limits, not those I'd configured for the user.

The solution in that case was found here. The short of it is, you need to define the limit via ulimit -n as part of the startup command in supervisord.

Example:

[program:redis-a]
command=bash -c "ulimit -n 32768; exec /usr/local/bin/redis-server /etc/redis/a.conf"

As well as increasing the open file limit. You need to increase maxclients in your redis.conf. It is only 10000 by default.

# Once the limit is reached Redis will close all the new connections sending
# an error 'max number of clients reached'.
maxclients 1024000