Follow-up from #7956.

Your system information

• Steam client version (build number or date): beta 2021-08-04, version 1628114775
• Distribution (e.g. Ubuntu): Debian 11
• Opted into Steam client beta?: yes
• Have you checked for system updates?: yes

Please describe your issue in as much detail as possible:

Steam launches games with the RLIMIT_NOFILE resource limit's rlimit_cur (soft limit) member set to more than FD_SETSIZE (1024).

This could cause games that call select(2) to crash or hang if many file descriptors are allocated. File descriptors with numeric value FD_SETSIZE or more cannot fit in a fd_set, and attempting to add them to the mask with FD_SET will cause an out-of-bounds write.

Similarly, if a game allocates memory proportional to the number of file descriptors allowed, it could allocate a large amount of memory; or if a game loops over all file descriptors up to the soft limit, like pressure-vessel used to do, it will loop for longer than is necessary.

If games need more than 1024 fds, and the game developer knows that the game and the libraries it uses do not have any of these anti-patterns, then the game should raise the soft fd limit itself, like Proton's branch of Wine does, and then lower it back down to 1024 before executing subprocesses that are not under their control (e.g. after fork() but before execve()).

Steps for reproducing this issue:

1. Launch any game (I used Floating Point)
2. Inspect its rlimits with prlimit -p (I used prlimit -p $(pgrep Floating), which is valid for single-process games like this one)
3. Look at NOFILE - max number of open files
4. Expected result: SOFT column says 1024; HARD column says something OS-dependent and more than 1024, normally 4096 (legacy), 1024×1024 (Debian derivatives) or 512×1024 (non-Debian-derivatives that use systemd)
5. Actual result: NOFILE SOFT column is 2348 (HARD column is 1024×1024 as expected on my Debian system)