How do linux file descriptor limits work?

You want to look at /proc/sys/fs/file-max instead

From recent linux/Documentation/sysctl/fs.txt:

file-max & file-nr:

The kernel allocates file handles dynamically, but as yet it doesn't free them again.

The value in file-max denotes the maximum number of file- handles that the Linux kernel will allocate. When you get lots of error messages about running out of file handles, you might want to increase this limit.

Historically, the three values in file-nr denoted the number of allocated file handles, the number of allocated but unused file handles, and the maximum number of file handles. Linux 2.6 always reports 0 as the number of free file handles -- this is not an error, it just means that the number of allocated file handles exactly matches the number of used file handles.

Attempts to allocate more file descriptors than file-max are reported with printk, look for "VFS: file-max limit reached".

EDIT: the underlying error is probably not the system running out of global filedescriptors, but just your process. It seems likely that the problem is the maximum size limit of select.

It does not look like you are hitting the system file desriptor limit. See this answer.

Perhaps your server process uses select and is thus limited to 1024 descriptors? If you switch to another mechanism, e.g. poll you will not be limited to 1024 descriptors anymore.

select() works with fd_sets

This is from the POSIX documentation of select.h:

The following shall be defined as a macro:

FD_SETSIZE

Maximum number of file descriptors in an fd_set structure.

Try to find or output FD_SETSIZE on your system.

If you find FD_SETSIZE is too low for you, I would rather try to move away from select than trying to increase FD_SETSIZE which is typically harder.