Your system information

• Steam client version (build number or date): 1627607186
• Distribution (e.g. Ubuntu): Debian 11
• Opted into Steam client beta?: yes
• Have you checked for system updates?: yes

Please describe your issue in as much detail as possible:

Downloading Civ VI gets paused after a while with "Disk write error". Looking at ~/.steam/root/logs/content_log.txt I see this:

[2021-07-30 18:47:14] Downloading 35959 chunks from depot 533503
[2021-07-30 18:47:26] Increasing target number of download connections to 4 (rate was 0.000, now 53.843)
[2021-07-30 18:47:26] Current download rate: 53.843 Mbps
[2021-07-30 18:47:26] Created download interface of type 'SteamCache' (7) to host cache1-lhr1.steamcontent.com (cache1-lhr1.steamcontent.com)
[2021-07-30 18:48:21] CGenericAsyncFileIOThread::AllocateResource() failed for CFileWriter: errno: 24, File: /home/steam/SteamLibrary/steamapps/downloading/289070/steamassets/base/platforms/windows/audio/751812779.wem
[2021-07-30 18:48:21] Failed to write chunk in file "steamassets\base\platforms\windows\audio\751812779.wem", 266788 bytes at offset 0 (File Not Found)
[2021-07-30 18:48:21] AppID 289070 update canceled : Failed updating depot 533503 while writing chunk, offset 0 (File Not Found) (Disk write failure) "/home/steam/SteamLibrary/steamapps/downloading/289070/steamassets/base/platforms/windows/audio/751812779.wem"
[2021-07-30 18:48:21] CGenericAsyncFileIOThread::AllocateResource() failed for CFileWriter: errno: 24, File: /home/steam/SteamLibrary/steamapps/downloading/289070/steamassets/base/platforms/windows/audio/748790226.wem

errno 24 is EMFILE, "Too many open files", which means RLIMIT_NOFILE is being reached, i.e. Steam is trying to open a very large number of file descriptors. Looking in /proc/$pid/fd/ for the main Steam process, I can see that there are about 1000 fds just before it fails.

By default, Debian sets most user processes to have a RLIMIT_NOFILE soft limit of 1024, and a hard limit of 1024*1024. This is the same as the recommendation in Proton's Wine esync branch, and similar to systemd recommendations (systemd actually defaults to a hard limit of 512*1024, but the exact number doesn't really matter).

The main reason why the soft limit is only 1024 is that programs that use select(2) will fail horribly under any RLIMIT_NOFILE higher than that. A few other programs, notably Java interpreters, also cope badly with high numbers.

There are two ways that Steam can avoid this bug:

• Stop trying to open files before EMFILE is reached, or at the very latest, when EMFILE is reached. A few hundred files open is probably plenty - if Steam is downloading a lot of files in parallel, there are going to be diminishing returns.

-and/or-

• Set a higher RLIMIT_NOFILE for Steam, but not its subprocesses
  • technically it can be anything <= the hard limit, but some older Linux systems set the hard limit to only 4096
  • this only works if Steam always uses poll() or epoll, and never select()
  • if going this route, it's important to drop the soft RLIMIT_NOFILE back down to 1024 when launching any subprocess whose code is not entirely under Valve's control, in particular games and diagnostic tools

Steps for reproducing this issue:

1. Have Debian's default RLIMIT_NOFILE: soft limit 1024, hard limit 1024*1024 (other distributions that use soft limit 1024 and hard limit 512*1024 would probably also trigger this, it's the soft limit of 1024 that matters)
2. Have a large game not installed - I used Civilization VI (289070). It might have to be a game with a large number of small files.
3. Install the game