horribly high system/io load
I'm also experiencing this issue.
Your system information
- Steam client version (build number or date): Also current Beta
- Distribution (e.g. Ubuntu): Arch Linux
- Opted into Steam client beta?: Yes
- Have you checked for system updates?: Yes
My setup also runs a LUKS encrypted LVM setup on SSDs which I thought might be related, since steam both has to decrypt and/or decompress the files and then lets the kernel encrypt them when writing.
I have been able to slightly mitigate the issue by throttling the download speed to 3MB/s, but the system is still severly crippled by the downloads.
Other things I have tinkered with:
- Setting different IO-schedulers (none, mq-deadline, Kyber, BFQ)
- Experimenting with allow-discard boot parameter (TRIM)
- Experimenting with sysctl settings (swappiness, vm.dirty(_background)_ratio, vm.vfs_cache_pressure)
- Experimenting with installing to separate LUKS/LVM devices not part of the / (root) or /home device
Noteworthy is this only seems to happen when High IO is created by the steam client when downloading/installing games
I'm recently also having a very similar issue. I don't know exactly when it started, but it cannot be that long ago, a couple of weeks maybe.
What happens is, that sometimes, but not always, when I start a game and it is starting "Processing Vulkan shaders", it gets stuck.
I/O is at 100% and nothing is going to happen for up to 30 minutes or even longer. CPU load is normal, it's just the disk I/O. I'm not having the issue that my system becomes unresponsive though, it's just the game won't start until steam has finished whatever it is doing.
I have absolutely no idea what's going on and why it takes that long. At some point, which can really take a long time, it finishes, compiles the shaders and everything is good - until next time. I think it also doesn't happen all the time when shaders need to be rebuilt and sometimes it also happens when it's updating a game.
Is there any way I can debug what is going on? I find it very difficult to get a hold on the detailed I/O information.
All that I can see in iotop is:
steam -nominidumps -nobreakpad [CJobMgr::m_Work]
- Steam client version (build number or date): Current Beta
- Distribution (e.g. Ubuntu): Ubuntu 20.04
- Opted into Steam client beta?: Yes
- Have you checked for system updates?: Yes
Edit:
I've found a similar bug report and did what was described here
I get tons of these lines from the process mentioned above:
CJobMgr::m_Work-32296 [020] .... 46424.210060: ext4_mark_inode_dirty: dev 8,49 ino 231761254 caller ext4_truncate+0x1ee/0x460
CJobMgr::m_Work-32296 [020] .... 46424.210061: ext4_mark_inode_dirty: dev 8,49 ino 231761254 caller ext4_evict_inode+0x34f/0x570
CJobMgr::m_Work-32296 [020] .... 46424.234431: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_dirty_inode+0x64/0x80
CJobMgr::m_Work-32296 [020] .... 46424.234462: ext4_mark_inode_dirty: dev 8,49 ino 231735703 caller ext4_unlink+0x2c7/0x380
CJobMgr::m_Work-32296 [020] .... 46424.234464: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_unlink+0x336/0x380
CJobMgr::m_Work-32296 [020] .... 46424.234470: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_evict_inode+0x2e3/0x570
CJobMgr::m_Work-32296 [020] .... 46424.234471: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_ext_truncate+0x2e/0xb0
CJobMgr::m_Work-32296 [020] .... 46424.234476: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_dirty_inode+0x64/0x80
CJobMgr::m_Work-32296 [020] .... 46424.234477: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller __ext4_ext_dirty.isra.0+0x74/0x90
CJobMgr::m_Work-32296 [020] .... 46424.234478: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller __ext4_ext_dirty.isra.0+0x74/0x90
CJobMgr::m_Work-32296 [020] .... 46424.234479: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_truncate+0x1ee/0x460
CJobMgr::m_Work-32296 [020] .... 46424.234480: ext4_mark_inode_dirty: dev 8,49 ino 231765243 caller ext4_evict_inode+0x34f/0x570
CJobMgr::m_Work-32296 [020] .... 46424.234774: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller ext4_dirty_inode+0x64/0x80
CJobMgr::m_Work-32296 [020] .... 46424.234791: ext4_mark_inode_dirty: dev 8,49 ino 231735703 caller ext4_unlink+0x2c7/0x380
CJobMgr::m_Work-32296 [020] .... 46424.234793: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller ext4_unlink+0x336/0x380
CJobMgr::m_Work-32296 [020] .... 46424.234797: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller ext4_evict_inode+0x2e3/0x570
CJobMgr::m_Work-32296 [020] .... 46424.234798: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller ext4_ext_truncate+0x2e/0xb0
CJobMgr::m_Work-32296 [020] .... 46424.234803: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller ext4_dirty_inode+0x64/0x80
CJobMgr::m_Work-32296 [020] .... 46424.234804: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller __ext4_ext_dirty.isra.0+0x74/0x90
CJobMgr::m_Work-32296 [020] .... 46424.234805: ext4_mark_inode_dirty: dev 8,49 ino 231752887 caller __ext4_ext_dirty.isra.0+0x74/0x90
Not sure what to do with that information though. Another thing worth to mention is, that the amount of data that is written is actually very low, below 1 M/s. Yet it is blocking. I've patched my kernel to work with fsync, might that be the issue?
Same issue here. Although I don't lose responsiveness (since my steam library resides in another HDD), I'm also seeing that high I/O when launching a game.
- Steam client version (build number or date): 163992987
- Distribution (e.g. Ubuntu): openSUSE Tumbleweed 20201203
- Opted into Steam client beta?: No
- Have you checked for system updates?: Yes
CJobMgr::m_Work-32296 [020] .... 46424.210060: ext4_mark_inode_dirty: dev 8,49 ino 231761254 caller ext4_truncate+0x1ee/0x460
That would be a kernel function. Is system tracing enabled ?
grep . /sys/kernel/debug/tracing/{tracing_on,events/ext4/ext4_mark_inode_dirty/enable}
Check syslog or /proc/kmsg
One of the things Steam tries to do is enable it. The client used to complain if it couldn't. Problem isn't necessarily with tracing itself but the level of tracing Valve enables is .. silly. It will slam the hell out of syslog and was a basis for them requiring root since you can't enable it as a user (nor should it ever outside a DEVEL environment).
tl;dr - Steam writing it out to a log can get amplified very quickly. The write itself can cause another event which snowballs into a loop. It gets worse when you add on anything else logging klog like syslogd or worse journald.
Lately this was getting really bad. Freezes while finalizing download could cause Firefox to crash(before it was just desktop freeze for few seconds). I have slow QLC nvme drive that I though was TLC when buying.... Thanks Crucial for sending reviewers TLC drives while selling it as QLC drive. Don't buy Crucial P2
Anyway all my problems went away when I switched to "Kyber" I/O scheduler.
yup we're there again, especially right after starting up* the client it catapults my load (on a ryzen5 with ssd!) up to >10 consistently, by deciding to upgrade a bunch of wineprefixes for no reason while apparently unpacking a throttled download with more than a thread per core. *(which of course happens after boot usually, when things are still calming down anyway, and sometimes gets so slow it prevents my desktop from actually loading up)
my problem definitely wasn't kernel tracing or extfs related, given i didn't use extfs since opening this, and steam never runs as root, who gave you that idea @h1z1 :P
also thanks for derailing the issue @smirgol @pablow1422 @h1z1 but can you fix your shader caching in your shader caching ticket maybe?
ok seriously this is ridiculous, can literally anyone at valve care?
there isn't even an option to nicely ask steam not to spawn 50 threads all getting in each others' way at once.
forbidding sync on the FS (essentially the same as using eatmydata which some people suggested in other issues) seems to take away some of the load, still pushing it to around 9 minimum though but also making the patching go about 5 times faster. so yeah um maybe don't literally spam threads trying to get exclusive synchronous access to the disk?
Tried every solution, responsiveness is way better but I'm stuck at 30MB/s -at best- on an NVMe SSD. A Helldivers 2 upgrade takes litteraly HOURS. This is driving me crazy.
can confirm @n00dl3 some games will still get the load absurdly high and be stuck at low actual throughput. Helldivers 2 is one of the worst offenders there. that said, plenty people on all platforms complain about HD2 updates regularly, it's not uncommon for a 100 byte download to cause the whole game to be rewritten on disk for no apparent reason.
@kisak-valve since you have clearly been aware of this ongoing issue since at least 2021, is there any chance that you guys give us a simple input/checkbox to limit the amount of threads? it's getting ridiculous.
i can confirm Helldivers 2 updates/installation is just the worst. The problem is even when you pause the installation. It will take some time until the last chunk in unpacked, however it seems that it doesn't release the I/O from my SSD. So even shutting down the PC becomes a half hour long process.
I am happy to provide input. This issue is really killing us.
want to pop in here and say that this is still an issue, on arch and my poor SSD get absolutely HAMMERED every single time I open steam without fail causing my whole system to grind to a halt
Nothing extracted yet.
Your system information
Please describe your issue in as much detail as possible:
whenever steam downloads/updates a game and accesses the disk (as seen both in steam's graph and htop's "DISK R/W" column) it hogs it so bad everything freezes up from iowait. loadavg shoots up to above 100.
since it got worse after i got a better hdd (with ssd hybrid cache) I suspect it to be hogging some sort of I/O controller or kernel thread (now that it can actually use that high bandwidth throughput), blocking others (swap would be the most obvious victim, but it's even making my mouse cursor laggy and i'm pretty sure X doesn't get swapped out while i use it, so probably "general I/O" or kernel load)
Steps for reproducing this issue:
in case you need any logs/etc, please do tell
WORKAROUND:
the problem is that steam just tries to write to as many files as possible at once, and in a synchronous manner (= disallows the operating system's cache), which makes it so it'll block itself and everything else.
so until valve fixes that, we can turn off sync access on the steam library. multiple options:
eatmydatawill wrap steam in a layer of "faking sync access but not actually doing it"sync=disabledas an option on the datasetfstabso you should probably just stick with eatmydata there, but i found so far:barrier=0should work, but will make it so your ext4 can potentially corrupt itself in a brownoutthis only helps with some games, most notably Helldivers 2 will still get the loadavg to >90 and not exceed some 20-30MB/s harddrive throughput