protonscr

[SteamCMD] XFS inode64 issues

steamopen SteamCMD
ValveSoftware/steam-for-linux#4824 · opened 2017-01-22 by galexrt · updated 2019-12-30 · 7 comments · github
Ggalexrt 2017-01-22 github

Your system information

  • Steam cmd version:
  • Distribution (e.g. Ubuntu): Fedora 24 (Rhel based)
  • Linux Kernel: Linux HOSTNAME 4.9.4-100.fc24.x86_64 #1 SMP Tue Jan 17 19:08:56 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Please describe your issue in as much detail as possible:

strace on root xfs filesystem:
On a root XFS (inode64) partition the steamcmd is working.

[...]
[pid 123] stat64("./zip0.zip", 0xf71c9c40) = -1 ENOENT (No such file or directory)
[pid 123] open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 2
[pid 123] fstat64(2, {st_mode=S_IFDIR|0755, st_size=100, ...}) = 0
[pid 123] getdents(2, /* 7 entries */, 32768) = 152
[pid 123] getdents(2, /* 0 entries */, 32768) = 0
[pid 123] close(2)
[...]

strace on XFS (inode64 non-root) partitions:
Steamcmd failing due to the 64-bit inode returned by getdents() or fstat64().

[...]
[pid 123] stat64("./zip0.zip", 0xf720ac40) = -1 ENOENT (No such file or directory)
[pid 123] open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY|O_CLOEXEC) = 2
[pid 123] fstat64(2, {st_mode=S_IFDIR|0755, st_size=5, ...}) = 0
[pid 123] getdents(2, 0xf7002114, 65536) = -1 EOVERFLOW (Value too large for defined data type)
[pid 123] close(2)
[...]

The same error with XFS (inode64) appears on CephFS and other clustered filesystems that use 64-bit inodes. The problem with XFS occurs "only" on non root partitions that use inode64.

One thing that doesn't want to get into my head is that the steamcmd already uses stat64() instead of stat() calls, but doesn't use getdents64()..

This is a huge problem as 64-bit inodes solve the "issue" with having "too many" files on one partition.
Most srcds games also don't run on a inode64 enabled filesystem, it most definetly is the same issue.
Some games that are also not working on a inode64 filesystem:

  • Garry's Mod
  • Counter-Strike: Source (Not Counter-Strike: Global Offensive)
  • Team Fortress 2 (as mentioned by @joshuacox)
  • Arma 3 (as mentioned by @joshuacox)
  • And other SRCDS games (seems to affect most srcds games that have a srcds version lower than CSGO)

The main "goal" of this issue is to resolve the issue at the "source" of "all", steamcmd or at least raise awareness of this issue and document it for the future.

This https://github.com/ValveSoftware/steam-for-linux/issues/4791 issue may be related with this, depending on if Amazon EFS uses a inode64 filesystem too.

Steps for reproducing this issue:

  1. Server with two xfs fs partitions (example: 30G xfs /, 100G xfs /data; both are mounted with the defaults options that includes the inode64 option).
  2. Go to /data and download and extract the steamcmd tar.gz (the default manual installation process).
  3. Run the steamcmd.sh.
  4. The steamcmd script loops the steamcmd showing only this as the error message "Failed to remove temporary unzip folder, continuing anyway". The steam error logs are empty. The "real" error than can only be "found" by using strace.
Jjoshuacox 2017-01-25 github

Just adding TF2, and Arma3 to this list of problematic game servers

Ttrinitronx 2018-03-04 github

This still appears to be an issue affecting latest steam 32-bit games on Linux x86_64 with XFS.

I am able to run Steam Client fine from Linux 64bit stored on XFS. I'm using large XFS RAID volume to store the Steam Library (I symlink ~/.steam from /home mount to a directory on this XFS RAID volume mount).

From my testing, I am able to install, launch and use Steam Client on Linux x86_64 using this large XFS filesystem. However, many 32bit hl2_linux games and others built for only 32bit x86 platforms still appear affected by crashes related to this inode64 problem. That is to say: ValveSoftware/Source-1-Games#1685 still is not fixed. Looks like it might be easy to fix with a recompile using stat64 & update.

Games known to be affected:

  • Kerbal Space Program (affected, 64bit version exists KSP.x86_64 but Steam always launches 32bit)
  • Half-Life
  • Half-Life 2 Deathmatch
  • Half-Life 2
  • Team Fortress 2
  • Portal
  • Half-Life 1 Source Deathmatch
  • Day of Defeat Source
  • Counter-Strike Source
  • GarrysMod

Workaround:

Move affected game files to a standard ext4 volume mount, then create symbolic links from steamapps/common directory to their new location on the ext4 mount.

Related issues:

  • bug in steam-for-linux appears fixed: #2610
  • bugs in many x86 32bit games appears not fixed: ValveSoftware/Source-1-Games#1685
Ttahoward 2019-03-18 github

Has there been any movement on this? Tried setting up a Rust server and ran into same issue as well.

Eezombie 2019-04-21 github

actual in Fedora 29\30 with kernel 5.x.x

# uname -a
Linux home.local 5.0.8-300.fc30.x86_64 #1 SMP Wed Apr 17 16:55:36 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

/dev/md127 on /mnt/md0 type xfs (rw,noatime,attr2,inode64,sunit=1024,swidth=2048,noquota)
Wwarkanum 2019-05-05 github

I'm still getting this issue on Garrysmod and Black Mesa. 64bit, 5.0.11-arch1-1-ARCH on 8TB inode64 ext4 partition, moved it to another drive. This issue is going to bug us more in the future when everyone gets bigger drives or switching to inode64.

CCyclic3 2019-06-24 github

I use CephFS to back my cluster (which uses 64 bit inodes), and so I have to copy literal gigabytes through the network onto a ramdisk to get the server to load.

Has there been any movement?

Ffrancoisjacques 2019-12-30 github

Civilization V is also affected.

EDIT: mount options (volume is an LVM2 logical volume on an mdraid partition.)

/dev/mapper/funtoo-home_folders on /home type xfs (rw,noatime,attr2,inode64,noquota)

EDIT2: Linux version is 4.19.27

Upstream links