Hello @ipr, if you run steam from a terminal, does the terminal spew give any hints?
Possibly related to #7630.
Possibly related to #7630.
Yes that seems likely, I'm also having quite a large Steam lib.
src/common/pipes.cpp (837) : fatal stalled cross-thread pipe.
src/common/pipes.cpp (837) : fatal stalled cross-thread pipe.
src/common/pipes.cpp (837) : Fatal assert; application exiting
src/common/pipes.cpp (837) : Fatal assert; application exiting
Installing breakpad exception handler for appid(steam)/version(1621394999)
assert_20210523002108_15.dmp[425377]: Uploading dump (out-of-process)
/tmp/dumps/assert_20210523002108_15.dmp
_ExitOnFatalAssert
assert_20210523002108_15.dmp[425377]: Finished uploading minidump (out-of-process): success = yes
assert_20210523002108_15.dmp[425377]: response: CrashID=bp-81272831-a045-429c-87cb-761922210522
assert_20210523002108_15.dmp[425377]: file ''/tmp/dumps/assert_20210523002108_15.dmp'', upload yes: ''CrashID=bp-81272831-a045-429c-87cb-761922210522''
I'll look into uploading dump files.
I'll look into uploading dump files.
I think it was already uploaded to Valve. All they need is probably the crash ID which is already part of the log you posted.
I'll look into uploading dump files.
I think it was already uploaded to Valve. All they need is probably the crash ID which is already part of the log you posted.
Ok, good.
Looks like new version of Steam client improves starting and I can avoid the unmount+mount things now. Good stuff!
I am guessing the changes to storage management and library management fixed the issue.
I don't think your problem is related to async IO or threading. The stalls you're experiencing happen after the crash event when Steam creates a crash dump and cleans up process memory. The original crash event is probably an OOM issue with large libraries.
But yes, it looks like it received some fixes but I'm still seeing crashes from time to time - and if that happens while in a game, it will immediately crash the game, too, usually resulting in a lot of IO thrashing which feels like the issue might be with IO or threading in the client but really it's the aftermath of the crash when the system cleans up memory and swaps pages back in (which doesn't necessarily mean "swap" but also re-populate caches and page executable code back in for running processes).
I don't think your problem is related to async IO or threading. The stalls you're experiencing happen after the crash event when Steam creates a crash dump and cleans up process memory. The original crash event is probably an OOM issue with large libraries.
No, you got it backwards. It is definitely reporting timeout while scanning disk, crash dump happens AFTER it while there is plenty of memory. And I've disabled the shader cache which was a problem with many other users.
But yes, it looks like it received some fixes but I'm still seeing crashes from time to time - and if that happens while in a game, it will immediately crash the game, too, usually resulting in a lot of IO thrashing which feels like the issue might be with IO or threading in the client but really it's the aftermath of the crash when the system cleans up memory and swaps pages back in (which doesn't necessarily mean "swap" but also re-populate caches and page executable code back in for running processes).
Again, to be absolutely clear about this: problem occurs during starting of Steam CLIENT, not when starting a game from it. Don't confuse these things, they are different.
To tell it as simply as possibly so I can get through to you:
It was never about starting a game in this issue.
The changes made to Steam library and storage likely changed the method of scanning/storing information enough that it is now fast enough to avoid the timeout.
Please read the information before commenting.
src/common/pipes.cpp (837) : fatal stalled cross-thread pipe.
This one can simply mean that the process on the other side of the pipe already died... It's not necessarily a timeout. But only the Steam developers will know.
btw, a "stall" is not a timeout. And the log you presented is not a crash, it's already in the cleanup process of a previous crash, and that crash is probably why the pipe stalled - which is a fatal condition to the Steam client, so it exits. It would be interesting if we could see what was logged before this final death event. Running the generated dump file through the strings command may reveal the real error, the string is most likely located somewhere near the end.
Also, I'm not saying it's the same crash (aka OOM, aside from that it's never a system memory issue, there's always enough memory, but still the client OOMs for no apparent reason), I'm just trying to explain why the "stall" may not be the crash, and why the log does not contain the crash but just the information that the crash dump has been collected and uploaded. So in the end, your issue report tries to suggest the solution with its title but the underlying assumption may be wrong. A better title could be "Steam client stalls and crashes on startup while scanning for games".
The OOM crashes also included the stall message for me sometimes. That's probably why originally it was suggested it may be the same problem as #7630.
I think it is bad software design to define timeouts for processes which may take way longer than expected sometimes - or at least it is bad design to let the software crash with an assertion in this case. I somewhat doubt that Valve did such a thing. Also, while your proposed solution could improve things, it's not a proper fix. In that case, the stall condition must be changed instead, or even better: completely eliminated. Async IO doesn't make things magically faster, neither does better threading. A re-design to use async IO or threading may still be needed to eliminate the stall condition. But that's pure speculation, and may introduce new bugs.
src/common/pipes.cpp (837) : fatal stalled cross-thread pipe.
This one can simply mean that the process on the other side of the pipe already died... It's not necessarily a timeout. But only the Steam developers will know.
Sure, but considering rest of evidence (high disk activity still going and so on) the MOST likely reason is simply timeout. The activity is evidence it has NOT died and is in progress and that only the WAITER side is the one with an actual problem. I think that would have been obvious already by what I listed previously above. It is mostly a matter of eliminating possible causes though different options.
I'm not even bothering to explaing the rest..
I think it is bad software design to define timeouts for processes which may take way longer than expected sometimes - or at least it is bad design to let the software crash with an assertion in this case. I somewhat doubt that Valve did such a thing. Also, while your proposed solution could improve things, it's not a proper fix. In that case, the stall condition must be changed instead, or even better: completely eliminated. Async IO doesn't make things magically faster, neither does better threading. A re-design to use async IO or threading may still be needed to eliminate the stall condition. But that's pure speculation, and may introduce new bugs.
You are badly mistaken if you assume users are happy when software is hanging indefinitely. Timeouts are a basic thing in telling a user "ok, this should have worked by now but I'm giving up": consider a mounted network share where the destination is unavailable. Instead of blocking indefinitely user is given the chance to abort ("I know it isn't there so don't try again") or to continue ("server just booted, up it should work now").
You are ludicrous is claiming they are symptom of "bad" or "good" design: there are many very good reasons to use timeouts. For situations which might take potentially forever better approach is to simply handle it in background and let user continue with whatever is possible then.
There is no reason to continue arguing about this any longer since the problem has been fixed.
Nothing extracted yet.
Your system information
Please describe your issue in as much detail as possible:
With large installs it would be beneficial if Steam client has better threading and/or using asynchronous IO for the scanning. Problems occur with large file amounts (several terabytes across multiple drives) causing Steam client to fail starting.
For one, better threading/async IO would allow using the client in the mean time (chat, community hub etc.).
Second, whole client is killed by OS or crashes when it is unresponsive for too long.
Problem is apparent when starting the client with large number of games installed.
Currently I have to do following
Expected behaviour would be that Steam client could start even with large installs without need to go through umount/mount/readd steps.
All drives are now using ext4. Trying to pre-cache filelist (such as du -sh) is not helping any more. In the past games with large amount of files had worse effect than games with simply few but large files. Size of games is rapidly growing and old workarounds are not working any longer to get client starting cleanly, thus needing the unmount-dance described above.
As a stop-gap measure to fix the immediate problem with starting, client should start first then automatically "add" the configured libraries for the user so that the procedure would be simplified/automated. Since the adding after starting seems to work most of the time that should not be too hard to change in the client and would improve client behaviour rapidly.
Occasionally adding large libraries afterwards (when client has started) will not work either and client terminates: this is where better async IO/threading would prove most beneficial.
Steps for reproducing this issue:
Alternatively: