Happens also with the beta (2021-01-30) just released:

Sat Jan 30 04:14:03 2021 GMT: file ''/tmp/dumps/assert_20210130051402_23.dmp'', upload yes: ''CrashID=bp-bfc1c8d1-fb34-400e-a20a-4d6c12210129''
Sat Jan 30 05:59:18 2021 GMT: file ''/tmp/dumps/assert_20210130065916_227.dmp'', upload yes: ''CrashID=bp-45e82d4b-51b8-433c-ac14-e794f2210129''

But:
23: Assert( Assertion Failed: vecDBs.Count() > 0 ):/data/src/clientdll/shadercachemanager/shadercompilejob.cpp:116
227: Assert( Fatal Assertion Failed: OUT OF MEMORY ):/data/src/tier0/memstd.cpp:2487

I'm not sure why it shows "out of memory": According to my atop history, there were 20+ GB of memory available at the time of the crash:

6:50 - 7:50:
MEM | tot 31.3G | free 3.5G | cache 20.0G | dirty 76.2M | buff 1.2M | slab 1.0G | slrec 544.3M | | shmem 1.2G | shrss 60.1M | shswp 0.0M | | vmbal 0.0M | | hptot 0.0M | hpuse 0.0M |

7:00 - 7:10:
MEM | tot 31.3G | free 3.8G | cache 20.3G | dirty 74.7M | buff 1.2M | slab 980.8M | slrec 541.8M | | shmem 701.6M | shrss 60.1M | shswp 0.0M | | vmbal 0.0M | | hptot 0.0M | hpuse 0.0M |

It seems to crash for seemingly random reasons. I've seen the "out of memory" line before and there was always plenty of memory available.

Since fossilize finished processing all games in the beta, the crashes seem to be gone. So the crashes may either depend on the shader caches, or on the system load.

So now it did crash again:

Thu Feb  4 09:22:28 2021 GMT: file ''/tmp/dumps/assert_20210204102226_41.dmp'', upload yes: ''CrashID=bp-91955c29-02e4-4880-bcd0-5789a2210204''
Assert( Fatal Assertion Failed: OUT OF MEMORY ):/data/src/tier0/memstd.cpp:2487

I'm pretty sure there's no out of memory condition, even with Steam running there's 21 GB memory available (8 GB free/unused). Any way for me to debug this further?

Any update on this?
This is still a thing.

build: 2021-03-23
system: Manjaro 21.0 Ornara
tested kernels: 5.9.16-1, 5.10.23-1, 5.11.6-1

@raQai It's rare currently but still happens...

@kisak-valve It seems to also affect Manjaro, so it's probably not Gentoo-specific.

@kakra thanks for the info.

do you by any chance run docker containers on your system which you do not shut down?
I did some testing and log analysis yesterday and it seems like there have been some interferences with my docker containers and Steam. After shutting down all containers, Steam did not crash for like 3 hours and I no longer got OOM exceptions. Will keep testing.

I'm not using docker because it messes with the network interfaces. Instead, I've setup a MACVLAN bridge interface by which the host system connects to the network (so the physical interface doesn't have an IP but the MACVLAN bridge has). Also attached to the MACVLAN bridge I have some systemd-nspawn containers running for development (autostarted on boot because they do not use many resources), and my qemu VM are attached to this bridge (started on demand). This allows me to operate containers and VMs without NAT through the host, and without needing a switch that supports hairpinning. I also have two OpenVPN interfaces running on this machine to connect to two networks I'm managing, these are always connected but have only specific subnet and host routes, no default route.

I never had problems with this setup. I know that many people have problems with docker and VPN network interfaces in Proton games.

FTR, I also tried many things and thought "oh cool, the problem seems fixed by that" only to discover that after the next Steam reboot the problem returned. If you'd ask me, I'd say the containers have nothing to do with that, stability seems mostly random - sometimes it works, and after the next reboot it doesn't.

If looking at RAM usage, the only thing I see is multiple Chrome processes allocated 33 GB virtual RAM (I know that this is not used RAM just address space but - I think - it puts pressure on the TLB [translation lookaside buffer], the actual used RAM of the whole Chrome process tree is much lower, below 2 GB, according to kernel memory accounting which includes file system cache usage:

# systemctl status --user app-google\\x2dchrome-18f0eaf0ceb0486ea0e67eef32f89931.scope
● app-google\x2dchrome-18f0eaf0ceb0486ea0e67eef32f89931.scope - Google Chrome - Web-Browser
     Loaded: loaded (/usr/share/applications/google-chrome.desktop; transient)
  Transient: yes
     Active: active (running) since Sat 2021-04-03 00:35:05 CEST; 10h ago
      Tasks: 360 (limit: 38367)
     Memory: 1.7G
        CPU: 0
     CGroup: /user.slice/user-500.slice/[email protected]/app.slice/app-google\x2dchrome-18f0eaf0ceb0486ea0e67eef32f89931.scope

Steam is comparable when idle (or even while doing its fossilize thing which does a great job after implementing the suggestions the maintainer and I developed during a long discussion):

# systemctl status --user app-steam\\x2dsilent-autostart.service
● app-steam\x2dsilent-autostart.service - Steam Autostart
     Loaded: loaded (/home/kakra/.config/autostart/steam-silent.desktop; generated)
     Active: active (running) since Sat 2021-04-03 00:34:47 CEST; 10h ago
       Docs: man:systemd-xdg-autostart-generator(8)
   Main PID: 2324 (bash)
      Tasks: 103 (limit: 38367)
     Memory: 1.3G
        CPU: 0
     CGroup: /user.slice/user-500.slice/[email protected]/app.slice/app-steam\x2dsilent-autostart.service
             ├─ 2324 bash /home/kakra/.local/share/Steam/steam.sh -silent
             ├─ 3526 /home/kakra/.local/share/Steam/ubuntu12_32/steam -silent
             ├─ 3609 /home/kakra/.local/share/Steam/ubuntu12_32/steam -silent
             ├─ 3610 /bin/bash /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper.sh -lang=de_DE -cachedir=/home/kakra/.local/share/Steam/config/htmlcache -steampid=3526 -buildid=1617402021 -steamid=0 -cachedir=/home/kakra/.local/share/Steam/confi>
             ├─ 3612 ./steamwebhelper -lang=de_DE -cachedir=/home/kakra/.local/share/Steam/config/htmlcache -steampid=3526 -buildid=1617402021 -steamid=0 -cachedir=/home/kakra/.local/share/Steam/config/htmlcache -steamuniverse=Public -realm=Global -clien>
             ├─ 3637 /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper --type=zygote --no-sandbox --log-file=/home/kakra/.local/share/Steam/logs/cef_log.txt --product-version=Valve Steam Client --lang=en_US.UTF-8
             ├─ 3664 /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper --type=gpu-process --field-trial-handle=12209813370393101961,62 s=MimeHandlerViewInCrossProcessFrame --no-sandbox --log-file=/home/kakra/.local/share/Steam/logs/cef_log.txt -->
             ├─ 3711 /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper --type=utility --field-trial-handle=12209813370393101961,6215161123523965996,1310 ssProcessFrame --lang=de --service-sandbox-type=network --no-sandbox --log-file=/home/kakra/.>
             ├─ 4154 /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper --type=renderer --no-sandbox --log-file=/home/kakra/.local/share/Steam/logs/cef_log.txt --field-trial-handle=12209813370393101961,6215161123523965996,131072 --disable-features>
             ├─ 4189 /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper --type=renderer --no-sandbox --log-file=/home/kakra/.local/share/Steam/logs/cef_log.txt --field-trial-handle=12209813370393101961,6215161123523965996,131072 --disable-features>
             ├─ 4195 /home/kakra/.local/share/Steam/ubuntu12_64/steamwebhelper --type=renderer --no-sandbox --log-file=/home/kakra/.local/share/Steam/logs/cef_log.txt --field-trial-handle=12209813370393101961,6215161123523965996,131072 --disable-features>
             ├─ 7392 /home/kakra/.local/share/Steam/ubuntu12_32/../ubuntu12_64/fossilize_replay /home/kakra/.local/share/Steam/steamapps/shadercache/1222140/fozpipelinesv5/steamapp_pipeline_cache.foz /home/kakra/.local/share/Steam/steamapps/shadercache/1>
             ├─ 8519 /home/kakra/.local/share/Steam/ubuntu12_32/../ubuntu12_64/fossilize_replay /home/kakra/.local/share/Steam/steamapps/shadercache/1222140/fozpipelinesv5/steamapp_pipeline_cache.foz /home/kakra/.local/share/Steam/steamapps/shadercache/1>
             └─10139 /home/kakra/.local/share/Steam/ubuntu12_32/../ubuntu12_64/fossilize_replay /home/kakra/.local/share/Steam/steamapps/shadercache/1222140/fozpipelinesv5/steamapp_pipeline_cache.foz /home/kakra/.local/share/Steam/steamapps/shadercache/1>

The virtual machines are barely noticeable here:

systemctl status machine.slice
● machine.slice - Virtual Machine and Container Slice
     Loaded: loaded (/usr/lib/systemd/system/machine.slice; static)
    Drop-In: /etc/systemd/system/machine.slice.d
             └─override.conf
     Active: active since Sat 2021-04-03 00:34:29 CEST; 10h ago
       Docs: man:systemd.special(7)
         IO: 449.5M read, 859.9M written
      Tasks: 151
     Memory: 366.5M
        CPU: 0
     CGroup: /machine.slice
...

Note to above: CPU is accounted with 0 usage due to using a CK patched kernel which only has a fake CPU cgroup controller.

Networking info (network C is my local network and has the default route):

# networkctl status
●          State: routable
         Address: 192.168.AAA.AA on vpn-MANAGED-A
                  192.168.BBB.B on vpn-MANAGED-B
                  192.168.C.CCC on vmbridge
                  2a02:CCCC:CCC:CCCC::CCC on vmbridge
                  2a02:CCCC:CCC:CCCC:CCCC:CCCC:CCCC:CCCC on vmbridge
                  fe80::AAAA:AAAA:AAAA:AAAA on vpn-MANAGED-A
                  fe80::BBBB:BBBB:BBBB:BBBB on vpn-MANAGED-B
                  fe80::CCCC:CCCC:CCCC:CCCC on vmbridge
         Gateway: 192.168.C.C on vmbridge
                  fe80::CCCC:CCCC:CCCC:CCCC on vmbridge
             DNS: 10.AAA.AA.A
                  10.AAA.AA.AA
                  192.CCC.C.C
                  192.BBB.BB.B
                  192.BBB.BB.B
  Search Domains: sol.LOCAL-C.de
   Route Domains: MANAGED-A.de
                  MANAGED-A.local
                  aaaaaaaa.MANAGED-A.de
                  .
                  MANAGED-B.local

The problem seems to occur less often since I opt-ed back into the Steam beta client.

My system currently sees some problems which look like workqueue congestions in the kernel which causes some microspikes in overall system latency but only if I leave the system running for multiple days (I've already upstreamed a kernel patch for bcache which mitigates this issue, backported to kernels 5.4+): The mouse pointer would micro-block sometimes for a few milliseconds which is barely noticeable. It can only be observed because moving the mouse pointer around to target small desktop elements sometimes puts the mouse pointer into an unexpected nearby spot. When this happens, it's usually in bursts so I if I'd move the mouse pointer in circles, it would stop for a few milliseconds, disturbing the circle movement. Not sure how to describe that better. But what I could conclude from that is that concurrent threads in Steam may get into an unexpected race condition because threads start to get out of expected order. If that would be the case, the Steam client would clearly lack some proper internal locking and synchronization. OTOH, the observation seems to be purely coincidental as I cannot reliably reproduce that. Probably similar to your observation about having docker running.

And TBH, stopping docker to properly run Steam is NOT a solution. If there'd be any bad interaction between those two, this needs to be fixed on either side.

Looking at this problem for quite some time now, it looks like a race condition inside the Steam client to me because the observations and proposed solutions seem purely random. Such behavior is usually a timing or concurrency thing when using threads and workers.

So one of the remaining questions is: Do you use bcache, or btrfs? Do you see micro latency spikes while moving the mouse cursor around? Do you see messages about pending workqueue jobs in dmesg? Do you see OOM messages in dmesg or do you run earlyoom/systemd-oomd? Whatever the underlying problem is that triggers those crashes, it's not a problem in that layer, the Steam client should simply not silently crash but handle that gracefully.

BTW: I just installed earlyoom lately to see if there might be any connection but there are no OOM events logged - so this is not a problem. The only thing I see is the micro latency spikes which I'm not sure how to analyze or trace back to the origin. Nothing in the system seems to cause it as I stopped many services and tasks while seeing them, so it's probably something that piles up in the kernel over time. But maybe we both can find a common denominator in our systems.

Not using bcache or btrfs but I indeed do have the described issues with the latency spikes with my mouse (probably the full system). As you mentioned, it is barely noticable in daily use but becomes really annoying in games. Then again once steam is shut down (or crashes silently) everything is back up to normal.

The problem seems to occur less often since I opt-ed back into the Steam beta client.

I had the feeling it got worse after trying the beta client for a few hours (at least it crashed more often). Still I also noticed less micro lags in games using the beta. Using either release, even native games like L4D seem to have micro lag spikes even though my avg FPS is locked at 300.

And TBH, stopping docker to properly run Steam is NOT a solution. If there'd be any bad interaction between those two, this needs to be fixed on either side.

Totally aggree on this one.

If you need any information about my system to find any kind of correlation, just tell me the commands and I provide the entire output. It seems like you understand far more than I do regarding this subject.

Here's my current Steam info posted to gist.github.com: https://gist.github.com/kakra/a92cdfc0ed8cc212ae981a7d9518b5a6 (taken from the Steam system information menu).

Not using bcache or btrfs but I indeed do have the described issues with the latency spikes with my mouse (probably the full system). As you mentioned, it is barely noticable in daily use but becomes really annoying in games. Then again once steam is shut down (or crashes silently) everything is back up to normal.

Well, for me the micro lags / latency spikes do not come from Steam running (or at least not alone), they persist even when I closed it, only a reboot helps. I think there's currently something going on in the kernel that introduced that. Let's hope some of the core kernel developers have the same experience and will come up with a fix. Since I've got the bcache patches into the kernel, latency spikes in games are mostly gone. But I still experience input lagging visibly (and persistently) behind sometimes, almost 200ms or more. Feels like playing drunken. ;-) I don't think we can attribute that to Steam although it started logging a lot of stuff in loop since some weeks (which may contribute to this problem). Could you look at journalctl and see if Steam repeats the same logs over and over again, sometimes multiple times per second? Then maybe head over to https://github.com/ValveSoftware/steam-for-linux/issues/7734.

What you are seeing may come from the fossilize process of Steam: Did you check if it is running when you're seeing the micro lags? OTOH, fossilize will shut down within a few seconds when you start a game, so the issue should not persist through playing a game then.

So we are still at a theory that these micro latency spikes could cause the crashes. But those are not caused by Steam but probably come from the kernel.

Here is my steam info: https://gist.github.com/raQai/37a4554130250b9be9dab9e1be824e9f

Checking for fossilize the next days. Posting an update if i find anything new.

Steam still crashes while opt in to beta. Found the following messages in my journal right before the crash occurs.

The following set of messages occur during micro stuttering.

Apr 05 13:08:39 rq-workstation steam.desktop[340327]: assert_20210405125100_1.dmp[340327]: Uploading dump (out-of-process)
Apr 05 13:08:39 rq-workstation steam.desktop[340327]: /tmp/dumps/assert_20210405125100_1.dmp
Apr 05 13:08:39 rq-workstation assert_20210405125100_1.dmp[340327]: Uploading dump (out-of-process)
                                                                    /tmp/dumps/assert_20210405125100_1.dmp
Apr 05 13:08:39 rq-workstation audit: BPF prog-id=34 op=LOAD
Apr 05 13:08:39 rq-workstation audit: BPF prog-id=35 op=LOAD
Apr 05 13:08:39 rq-workstation systemd[1]: Started Process Core Dump (PID 340328/UID 0).
Apr 05 13:08:39 rq-workstation audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@8-340328-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 05 13:08:39 rq-workstation audit[250023]: NETFILTER_CFG table=filter family=2 entries=4 op=xt_unregister pid=250023 comm="kworker/u64:3"
Apr 05 13:08:39 rq-workstation audit[250023]: NETFILTER_CFG table=nat family=2 entries=15 op=xt_unregister pid=250023 comm="kworker/u64:3"
Apr 05 13:08:40 rq-workstation steam.desktop[340327]: assert_20210405125100_1.dmp[340327]: Finished uploading minidump (out-of-process): success = yes
Apr 05 13:08:40 rq-workstation steam.desktop[340327]: assert_20210405125100_1.dmp[340327]: response: CrashID=bp-a97af391-3ccf-4cf2-865e-46b282210405
Apr 05 13:08:40 rq-workstation steam.desktop[340327]: assert_20210405125100_1.dmp[340327]: file ''/tmp/dumps/assert_20210405125100_1.dmp'', upload yes: ''CrashID=bp-a97af391-3ccf-4cf2-865e-46b282210405''
Apr 05 13:08:40 rq-workstation assert_20210405125100_1.dmp[340327]: Finished uploading minidump (out-of-process): success = yes
Apr 05 13:08:40 rq-workstation assert_20210405125100_1.dmp[340327]: response: CrashID=bp-a97af391-3ccf-4cf2-865e-46b282210405
Apr 05 13:08:40 rq-workstation assert_20210405125100_1.dmp[340327]: file ''/tmp/dumps/assert_20210405125100_1.dmp'', upload yes: ''CrashID=bp-a97af391-3ccf-4cf2-865e-46b282210405''
Apr 05 13:08:40 rq-workstation systemd-coredump[340330]: Process 294173 (steam) of user 1000 dumped core.
                                                         
                                                         Stack trace of thread 294173:
                                                         #0  0x00000000f7993fea __strlen_sse2_bsf (libc.so.6 + 0x91fea)
                                                         #1  0x00000000ed20f9c3 n/a (/home/raqai/.local/share/Steam/ubuntu12_32/steamclient.so + 0x6279c3)
Apr 05 13:08:41 rq-workstation systemd[1]: [email protected]: Succeeded.
Apr 05 13:08:41 rq-workstation audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-coredump@8-340328-0 comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Apr 05 13:08:41 rq-workstation steam.desktop[294015]: /home/raqai/.local/share/Steam/steam.sh: line 771: 294173 Segmentation fault      (core dumped) $STEAM_DEBUGGER "$STEAMROOT/$STEAMEXEPATH" "$@"

The following entries occur after crashing

Apr 05 12:28:39 rq-workstation steam.desktop[184149]: CCrossProcessPipe::BWrite wrote too few bytes: 32 (Broken pipe).  Continuing.
Apr 05 12:28:39 rq-workstation steam.desktop[184149]: src/common/pipes.cpp (837) : fatal stalled cross-thread pipe (pipe is disconnected).
Apr 05 12:28:39 rq-workstation steam.desktop[184149]: src/common/pipes.cpp (837) : fatal stalled cross-thread pipe (pipe is disconnected).
Apr 05 12:28:39 rq-workstation steam.desktop[184149]: src/common/pipes.cpp (837) : Fatal assert; application exiting
Apr 05 12:28:39 rq-workstation steam.desktop[184149]: src/common/pipes.cpp (837) : Fatal assert; application exiting
Apr 05 12:28:40 rq-workstation steam.desktop[241572]: ERROR: ld.so: object '/home/raqai/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Apr 05 12:28:40 rq-workstation steam.desktop[241573]: assert_20210405122839_2.dmp[241573]: Uploading dump (out-of-process)
Apr 05 12:28:40 rq-workstation steam.desktop[241573]: /tmp/dumps/assert_20210405122839_2.dmp
Apr 05 12:28:40 rq-workstation steam.desktop[184149]: _ExitOnFatalAssert
Apr 05 12:28:42 rq-workstation steam.desktop[241573]: assert_20210405122839_2.dmp[241573]: Finished uploading minidump (out-of-process): success = yes
Apr 05 12:28:42 rq-workstation steam.desktop[241573]: assert_20210405122839_2.dmp[241573]: response: Discarded=1
Apr 05 12:28:42 rq-workstation steam.desktop[241573]: assert_20210405122839_2.dmp[241573]: file ''/tmp/dumps/assert_20210405122839_2.dmp'', upload yes: ''Discarded=1''
Apr 05 12:28:42 rq-workstation steam.desktop[241573]: pid 241573 != 241572, skipping destruction (fork without exec?)

@kakra do you think the dmp file would provide additional information? Also do you think this relates to your issue?

You could run the dump file through the strings command to extract all human-readable data from the dump file.

https://gist.github.com/raQai/2e6caa5c932ba42046866379f9d8474b

Did Steam crash, or did a game crash? It looks like this is a Proton 6.5 GE crash...

steam crashed. this crash was during a gaming session though and the game crashed with it. Also I get the same log output when using Proton 6.3-1

Yeah, I think when Steam crashes, the game will just be killed with it - probably due to a "broken pipe". So maybe you picked the wrong dump and there should be another one which is actually from the Steam process?

Hmmm, do you see anything of interest for this or any other issue in this journalctl extract?
https://gist.github.com/raQai/2653bf1a6bf2207952eaf97e50cd9841

I do not see anything that might me directly linked to the steam crash.

I've been getting these too for a past few months I think, recently these crashes became more frequent. I'm using Arch Linux.
Typical log after crash that happens randomly: either while playing killing game too (any game) or just silently die while in background:

[0405/195204.609450:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
[0405/195204.635024:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
[0405/195204.653993:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
Installing breakpad exception handler for appid(steam)/version(1616532526)
Failed to init SteamVR because it isn't installed
ExecCommandLine: "'/home/eugene/.local/share/Steam/ubuntu12_32/steam'"
System startup time: 1.06 seconds
[0405/195204.949379:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
[0405/195204.951054:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
[0405/195204.954806:INFO:crash_reporting.cc(270)] Crash reporting enabled for process: renderer
BuildCompleteAppOverviewChange: 136
RegisterForAppOverview 1: 4ms
RegisterForAppOverview 2: 4ms
Installing breakpad exception handler for appid(steam)/version(1616532526)
Installing breakpad exception handler for appid(steam)/version(1616532526)
Installing breakpad exception handler for appid(steam)/version(1616532526)
***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Installing breakpad exception handler for appid(steam)/version(1616532526)
assert_20210405202126_26.dmp[458641]: Uploading dump (out-of-process)
/tmp/dumps/assert_20210405202126_26.dmp
_ExitOnFatalAssert
assert_20210405202126_26.dmp[458641]: Finished uploading minidump (out-of-process): success = yes
assert_20210405202126_26.dmp[458641]: response: CrashID=bp-f2878fe1-0353-4b1f-a454-ef0e22210405
assert_20210405202126_26.dmp[458641]: file ''/tmp/dumps/assert_20210405202126_26.dmp'', upload yes: ''CrashID=bp-f2878fe1-0353-4b1f-a454-ef0e22210405''

There is plenty of free memory (32G in total, only around 6-8 is used). I'm using btrfs but I had similar problems on ext4 too. Any info I can provide to help with this issue?

Pretty certain i'm affected by this as well on fedora workstation 32, happy to provide any information that will help diagnose the issue.

I've also had those random OOM crashes on Arch Linux (steam-native) during the last 1-2 months. Steam client is just silently idling in the background, when it suddenly crashes, today after about 5 hours. So far it hasn't crashed during a game.

Installing breakpad exception handler for appid(steam)/version(1618256785)
[2021-04-24 12:25:17] Background update loop checking for update. . .
[2021-04-24 12:25:17] Downloading manifest: https://cdn.akamai.steamstatic.com/client/steam_client_ubuntu12
[2021-04-24 12:25:17] Download skipped by HTTP 304 Not Modified
[2021-04-24 12:25:17] Nothing to do
Installing breakpad exception handler for appid(steam)/version(1618256785)
Installing breakpad exception handler for appid(steam)/version(1618256785)
***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Installing breakpad exception handler for appid(steam)/version(1618256785)
assert_20210424143922_50.dmp[78120]: Uploading dump (out-of-process)
/tmp/dumps/assert_20210424143922_50.dmp
_ExitOnFatalAssert
assert_20210424143922_50.dmp[78120]: Finished uploading minidump (out-of-process): success = yes
assert_20210424143922_50.dmp[78120]: response: CrashID=bp-d7f95581-c5a7-4ddb-bc71-f339b2210424
assert_20210424143922_50.dmp[78120]: file ''/tmp/dumps/assert_20210424143922_50.dmp'', upload yes: ''CrashID=bp-d7f95581-c5a7-4ddb-bc71-f339b2210424''

For me, it's getting really annoying lately: Steam crashes in the middle of a game, and it does it often. I have maybe 30-60 minutes before it crashes. Usually, FPS suddenly drop, and games see short freezes multiple times within a few seconds, then everything crashes back to desktop with Steam gone:

Apr 24 14:57:50 jupiter plasmashell[275010]: Looks like steam didn't shutdown cleanly, scheduling immediate update check
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] Loading cached metrics from disk (/home/kakra/.local/share/Steam/package/steam_client_metrics.bin)
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] Using the following download hosts for Public, Realm steamglobal
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] 1. https://cdn.cloudflare.steamstatic.com, /client/, Realm 'steamglobal', weight was 100, source = 'update_hosts_cached.vdf'
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] 2. https://cdn.akamai.steamstatic.com, /client/, Realm 'steamglobal', weight was 100, source = 'update_hosts_cached.vdf'
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] 3. http://media.steampowered.com, /client/, Realm 'steamglobal', weight was 1, source = 'baked in'
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] Checking for update on startup
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] Suche nach verfügbaren Updates...
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:01] Downloading manifest: https://cdn.cloudflare.steamstatic.com/client/steam_client_publicbeta_ubuntu12
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:02] Download skipped by HTTP 304 Not Modified
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:02] Nothing to do
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:02] Installation wird überprüft...
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:02] Performing checksum verification of executable files
Apr 24 14:57:50 jupiter plasmashell[275010]: [2021-04-24 14:22:05] Verification complete
Apr 24 14:57:50 jupiter plasmashell[275010]: ***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
Apr 24 14:57:50 jupiter plasmashell[275010]: src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
Apr 24 14:57:50 jupiter plasmashell[275010]: src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
Apr 24 14:57:50 jupiter plasmashell[275010]: src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Apr 24 14:57:50 jupiter plasmashell[275010]: src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Apr 24 14:57:50 jupiter plasmashell[275010]: Installing breakpad exception handler for appid(steam)/version(1619231536)
Apr 24 14:57:51 jupiter plasmashell[277663]: assert_20210424145750_54.dmp[277663]: Uploading dump (out-of-process)
Apr 24 14:57:51 jupiter plasmashell[277663]: /tmp/dumps/assert_20210424145750_54.dmp
Apr 24 14:57:51 jupiter plasmashell[275010]: _ExitOnFatalAssert
Apr 24 14:57:52 jupiter plasmashell[275019]: Invalid browser dimensions: 0 x 0
Apr 24 14:57:52 jupiter plasmashell[275019]: Invalid browser dimensions: 0 x 4
Apr 24 14:57:52 jupiter plasmashell[275019]: Invalid browser dimensions: 0 x 0
Apr 24 14:57:52 jupiter plasmashell[277663]: assert_20210424145750_54.dmp[277663]: Finished uploading minidump (out-of-process): success = yes
Apr 24 14:57:52 jupiter plasmashell[277663]: assert_20210424145750_54.dmp[277663]: response: CrashID=bp-e47502eb-6c1a-4ecf-8332-fb3ba2210424
Apr 24 14:57:52 jupiter plasmashell[277663]: assert_20210424145750_54.dmp[277663]: file ''/tmp/dumps/assert_20210424145750_54.dmp'', upload yes: ''CrashID=bp-e47502eb-6c1a-4ecf-8332-fb3ba2210424''
Apr 24 14:57:52 jupiter plasmashell[276285]: src/common/pipes.cpp (837) : fatal stalled cross-thread pipe (pipe is disconnected).
Apr 24 14:57:52 jupiter plasmashell[276285]: src/common/pipes.cpp (837) : fatal stalled cross-thread pipe (pipe is disconnected).
Apr 24 14:57:52 jupiter plasmashell[276285]: src/common/pipes.cpp (837) : Fatal assert; application exiting
Apr 24 14:57:52 jupiter plasmashell[276285]: src/common/pipes.cpp (837) : Fatal assert; application exiting
Apr 24 14:57:55 jupiter plasmashell[277676]: ERROR: ld.so: object '/home/kakra/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
Apr 24 14:57:55 jupiter plasmashell[277677]: crash_20210424145752_2.dmp[277677]: Uploading dump (out-of-process)
Apr 24 14:57:55 jupiter plasmashell[277677]: /tmp/dumps/crash_20210424145752_2.dmp
Apr 24 14:57:55 jupiter plasmashell[276285]: _ExitOnFatalAssert

Why did it try to allocate 1.9 GB of memory anyways? This is highly likely to fail because Steam is a 32 bit app:

# file /home/kakra/.local/share/Steam/ubuntu12_32/steam
/home/kakra/.local/share/Steam/ubuntu12_32/steam: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.24, BuildID[sha1]=da97d79e349497519d62cf33b7cfaa6aff1857eb, not stripped

Something is going crazy inside Steam. It looks like the update check is running right in the middle of a game session.

It's always the exact same size:

Apr 25 05:27:43 jupiter plasmashell[60187]: ***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
Apr 25 05:27:43 jupiter plasmashell[60187]: src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
Apr 25 05:27:43 jupiter plasmashell[60187]: src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
Apr 25 05:27:43 jupiter plasmashell[60187]: src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Apr 25 05:27:43 jupiter plasmashell[60187]: src/tier0/memstd.cpp (2489) : Fatal assert; application exiting

Right, it is always that 747-sized chunk of memory

@kisak-valve Please adjust the labels, this is not Gentoo-specific, it seems to affect any distribution. Thus, it probably also affects officially supported distributions.

Thank you guys for all the detailed discussion in this thread. I'm having a similar issue and can help with information/logs if necessary.

However, someone mentioned Docker here at some point and I had two containers running. After removing the containers (and Docker itself since I actually don't need it in this machine) I didn't experience the issue anymore! I still have occasional stuttering but Steam won't crash anymore.

I'll test later if by just installing Docker and having it running again Steam will crash ¯_(ツ)_/¯

Even before I test Docker... I left the game running for a while and there's the same error again:

Assert( Assertion Failed: m_pInternalPipe->BRead failed ):/data/src/common/pipes.cpp:623

I was the one mentioning the docker issues. Not only with running containers but the plain service running lead to crashes. As a temporary workaround I disabled the service on startup and did not experience most of the issues anymore.

The OOM issue seems unrelated though since I still occasionally get one of those (mostly while the pre-cache process is running as @kakra mentioned)

I think we are seeing two different kind of crashes: One is related to having docker running and may be an issue with the network interfaces it creates (there are multiple reports for games having network issues with docker running), the other is the OOM issue. I'll rename this report to mention the OOM issue. For docker-related issues, you should open another report.

But I can confirm that the frequency of crashes is much lower after the background-fossilize process (shader pre-caching) is done (you can watch this in the settings dialog which shows a live status). After the pre-caching finished, the Steam process ran for 10 hours straight now while it previously would crash every half an hour or so. But it's still only one trigger of the problem, the Steam process may still crash while playing a game, and the frequency has vastly increased since my initial report. Most of the times, the client immediately updates itself when restarting it after a crash so it may be related to update search, too.

I've set it to look for updates only at night now. Let's see if this improves things and narrows down the issue.

@kisak-valve Yes, #7280 looks like a valid duplicate: I can actually see a lot of the same behavior described there. But this also seems to happen when no background fossilize is running at all. So I'll leave this open for a few more days to observe if this is a real duplicate, or if this is related purely to the background update process: After all, I observed logs showing that an update was found often just before the crash, and after restart, it would actually self-update the client.

Does the Steam client link to the fossilize lib, or does it just call it as an external binary? For me it looks like Steam is just running fossilize in external mode, and some pipe is connected to control the external process. Also, fossilize should not run at all while a game is running - except maybe it is also recording the shader state of games running: In that case it could make sense to split the fossilize process into a child process/fork so it doesn't crash the whole gaming session. After all, fossilize seems to be designed around the possibility to shut down unexpectedly (aka crashing) because some shader processing is known to segfault the process, or at least, the database is designed around recovering properly. So I'd expect that fossilize is expected to MAYBE crashing sometimes, and handle that gracefully. The Steam client should be able to handle similar problems gracefully, too.

I wonder why different people see the exact same allocated byte count in the error message (#7280 has a different size in one report, in most others, it's missing). Is this connected to the size of some shader file? Is Steam maybe trying to allocate this memory from shared memory? Because that is usually limited in Linux.

OTOH, maybe Steam should switch to being a 64-bit app. Many distributions already removed support for 32-bit only systems, and even playing modern games on 32-bit is not even an option. In the Steam hardware survey, 32-bit systems are below the 0.1% threshold.

@kakra you may be onto something with that update thing. I restricted the update time outside the time period I normally run/use Steam, and sure enough, yesterday Steam client idled happily in the background the whole time I was using the computer (about 14 hours straight) without crashing. Usually it crashes with the '1970089747' OOM erron in much less than 6 hours.

I don't think #7280 has much relevance in my case. At least the 'Allow backgroud processing of Vulcan shaders' is, and has always been disabled.

To summarize, we probably see three distinct crash situations:

• This one is related to the background update check, as we can rule out fossilize (confirmed by @cyberpunkrocker-zero in the previous comment), the update check doesn't necessarily trigger a download, it just checks for the availability of the self-update or other components which seemingly can be prevented by limiting the time window for update checks in the Steam client settings, it has a distinct indicator in the log of allocating 1970089747 bytes of memory, this can happen while a game is running, resulting in the game dying together with the Steam processes, and the situation may actually only be apparent WHILE a game is running, thus it helps limiting the update checks to intervals when you're not playing games
• #7280 is about crashes introduced by OOM situations while the fossilize background worker is running, the log shows allocation sizes different from 1970089747
• #6751 / #7740 are about crashes probably related to Docker network interfaces not managed by NetworkManager, the log shows the NM API calls are involved, it's probably some NULL pointer exception (can probably prevented by not using NetworkManager at all, I never saw this crash, and I don't have NM installed but use systemd-networkd instead)

@kisak-valve So I think we have another distinct crash situation here. What do you think?

@cyberpunkrocker-zero I think I can confirm this: The client churns along nicely since multiple days now after I restricted updates to nightly time intervals.

@kakra I restricted the update time to 6:00AM-9:00AM. If I start Steam before 9:00AM, it checks the updates, if any, and continues to work for the rest of the day without crashing. So it seems that update checks / updates does not crash Steam if done when the client is not yet started, but eventually OOM crash the client if done while it is running. Also, game updates work normally without crashing Steam if started manually outside the defined update time.

I don't have any update restrictions and I allow downloads during gameplay. I've only had 1 OOM during the past 3 days.
Before this it could crash once an hour or even once every few minutes. So it happens in periods. One week it's good and the other it crashes every day. Do you think it's related to updates?

I also had OOMs in offline mode. Last time i remember was 16th february 2021:

***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
src/tier0/memstd.cpp (2487) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2487) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2487) : Fatal assert; application exiting
src/tier0/memstd.cpp (2487) : Fatal assert; application exiting

I also noticed that these crashes happen during specific actions in games (there are more crashes, including crashes when no games are running, but these happened multiple times):

• going to the next room in The Binding of Isaac (offline mode)
• clicking on menu button in TF2

Both of these are native games, steam overlay is on.
These happened multiple times in exact timing of mentioned events, so I don't think it's update related.

Please look at the console output of the crash (depends on your desktop environment / login manager, it may be logged to xsession.log, or to journalctl, or something else). Then look at https://github.com/ValveSoftware/steam-for-linux/issues/7630#issuecomment-827555620 to decide whether it's this crash situation or one of the other I identified.

In this case it looks like it's the crash that comes from the update check. Do you have the console log at hand the comes just before the crash?

Steam overlay is off here because it interfered badly with the (Vulkan) render surface in the past, leading to game freezes. That's probably fixed by now but I still have it turned off.

Restricting updates to a specific time period is no longer a work-around for me, the Steam client started crashing again, usually more often when new updates of Proton appear, or after downloading some bigger game updates.

Finally got around to seeing the output from a terminal, to check and see what's been causing mine to crash, came across this issues, myself on Arch:

***** OUT OF MEMORY! attempted allocation size: 4120 ****
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Installing breakpad exception handler for appid(steam)/version(1621394999)
_ExitOnFatalAssert

This has been happening for me since when this issue was first reported, and has persisted across new hardware. I keep all of my Steam games on a separate SSD from my root and home partitions. I have had pre-caching in the background disabled for a good while since it was causing Tekken 7 to constantly rebuild its cache any time I ran it, so leaving it to simply keep its existing cache from merely running the game has been fine and stable. It's entirely possible the shadercache was stale but I'm not entirely certain this is the cause. I've just recently (just before posting this) deleted the existing cache and re-enabled pre-caching to see if anything changes.

I know it's been 20 days since the last report, but I figured reporting a number different from the 2GB of memory might raise some flags.

Same experience as kakra, restricting updates used to help but lately it has ceased doing so.

Happens also with the beta (2021-01-30) just released:

But:
23: Assert( Assertion Failed: vecDBs.Count() > 0 ):/data/src/clientdll/shadercachemanager/shadercompilejob.cpp:116
227: Assert( Fatal Assertion Failed: OUT OF MEMORY ):/data/src/tier0/memstd.cpp:2487

I'm not sure why it shows "out of memory": According to my atop history, there were 20+ GB of memory available at the time of the crash:

Is there 32-bit code with malloc() of around 2GB (signed 32-bit int) somewhere?
Edit: looks like this was already found earlier

Another might be if there is not large enough contiguous memory area for allocation (depending of flags, system configuration and what type of memory it is trying to allocate).

Often memory mapping of files would be a better choice that reading into large buffers but without knowing what the memory is used for that is hard to tell if it would apply here too.

I'm observing this crash after Steam (current stable) downloads large Shader caches (i.e. for Horizon Zero Dawn):

***** OUT OF MEMORY! attempted allocation size: 402653184 ****
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal Assertion Failed: OUT OF MEMORY
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
src/tier0/memstd.cpp (2489) : Fatal assert; application exiting
Installing breakpad exception handler for appid(steam)/version(1621394999)
assert_20210606095532_168.dmp[2421]: Uploading dump (out-of-process)
/tmp/dumps/assert_20210606095532_168.dmp
_ExitOnFatalAssert
mash@ryzen:~$ assert_20210606095532_168.dmp[2421]: Finished uploading minidump (out-of-process): success = yes
assert_20210606095532_168.dmp[2421]: response: CrashID=bp-0a1d8c3b-e93f-447f-9244-43d8f2210606
assert_20210606095532_168.dmp[2421]: file ''/tmp/dumps/assert_20210606095532_168.dmp'', upload yes: ''CrashID=bp-0a1d8c3b-e93f-447f-9244-43d8f2210606''

Retrospectively, it makes sense that this seems to be related to shader cache updates...

I am also getting this sort of crash literally once or twice a day now when I start Steam. If Steam wasn't running for a few hours when I start it it then it will run for a few seconds/minutes then crash from some shader updates. But then I can relaunch steam after a few seconds and since the problem update is done it then runs ok and I can start a game. But then if I close steam, leave my PC and come back the next day Steam will crash again within a minute or 2 of starting it up.

This even happens when I have more than 32GB of free RAM on 64GB system. (Vega + RADV)
For me this only started in the last ~1.5 weeks probably starting after the video foz related updates.

Ubuntu 21.04 here, no Docker used, 16GB of RAM, no Beta : Steam Client Jun 8 2021.
About shaders : Enable Shader Pre-caching : checked, Allow background processing : not checked.
About steam games auto-updates : restricted to 1hour/day from 11 AM to 12 PM.
I have daily unattended steam client crashes, always during update time :

journalctl -S "2021-06-19 00:00"|grep steam|grep Seg
juin 19 11:03:17 host steam.desktop[3641]: /home/user/.steam/debian-installation/steam.sh: line 772:  4113 Segmentation fault      $STEAM_DEBUGGER $DEBUGGER_ARGS "$STEAMROOT/$STEAMEXEPATH" "$@"
juin 20 11:14:30 host steam.desktop[3571]: /home/user/.steam/debian-installation/steam.sh: line 772:  4069 Segmentation fault      $STEAM_DEBUGGER $DEBUGGER_ARGS "$STEAMROOT/$STEAMEXEPATH" "$@"
juin 22 11:00:41 host steam.desktop[6459]: /home/user/.steam/debian-installation/steam.sh: line 772:  7164 Segmentation fault      $STEAM_DEBUGGER $DEBUGGER_ARGS "$STEAMROOT/$STEAMEXEPATH" "$@"
juin 23 11:00:33 host steam.desktop[152925]: /home/user/.steam/debian-installation/steam.sh: line 772: 153375 Segmentation fault      $STEAM_DEBUGGER $DEBUGGER_ARGS "$STEAMROOT/$STEAMEXEPATH" "$@"
juin 23 11:46:40 host steam.desktop[172591]: /home/user/.steam/debian-installation/steam.sh: line 772: 172735 Segmentation fault      $STEAM_DEBUGGER $DEBUGGER_ARGS "$STEAMROOT/$STEAMEXEPATH" "$@"
juin 24 11:01:50 host steam.desktop[200936]: /home/user/.steam/debian-installation/steam.sh: line 772: 201080 Segmentation fault      $STEAM_DEBUGGER $DEBUGGER_ARGS "$STEAMROOT/$STEAMEXEPATH" "$@"

Latest Steam beta update mentions "Initial fixes for excess memory usage downloading shader depots". Is that related?

Been several months and nothing has changed. Is there anything else we can do to provide more information on getting this solved? I can't imagine we'd want this still happening with SteamOS 3 and the Deck coming around in a few months.

Arch
kernel 5.14.8-zen1-1-zen
Ryzen 5 3600
RX 5600XT
16GB RAM
Mesa 21.2.2

Really, nothing has changed other than kernel and mesa versions since I last posted about this.

@hikarutilmitt I have almost exactly the same specs as you (on Arch, of course) and I haven't had any problems for several months. This seems to be fixed for anyone else, too. Are you sure you have the latest steam client? (latest client update 2 days ago)

@hikarutilmitt I have almost exactly the same specs as you (on Arch, of course) and I haven't had any problems for several months. This seems to be fixed for anyone else, too. Are you sure you have the latest steam client? (latest client update 2 days ago)

I'm on the beta branch and it updates pretty frequently. I just after posting my last comment cleared the shader cache completely and restarted steam then re-enabled background processing to see if the recent changes just hadn't taken somehow. I'll put the error in again if it happens again.

EDIT:
it's still happening, beta or no beta.

[2021-10-22 14:24:37] Nothing to do
Installing breakpad exception handler for appid(steam)/version(1634158817)
Installing breakpad exception handler for appid(steam)/version(1634158817)
***** OUT OF MEMORY! attempted allocation size: 4120 ****
src/tier0/memstd.cpp (2838) : OUT OF MEMORY
src/tier0/memstd.cpp (2838) : OUT OF MEMORY
src/tier0/memstd.cpp (2838) : Fatal assert; application exiting
src/tier0/memstd.cpp (2838) : Fatal assert; application exiting
Installing breakpad exception handler for appid(steam)/version(1634158817)
DBG DBG ffffffff DUMP_REQUESTED Assert( OUT OF MEMORY ):/data/src/tier0/memstd.cpp:2838

_ExitOnFatalAssert

If there's any other log to tell me what exactly is doing this so either I can fix some potential misconfiguration on my end or help maybe nail down the bug itself for anyone else, I'd be all for it.

That infamous bug is back for me again.

Last night, it crashed out of the blue, the system was idle, and Steam checked for updates:

Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:30] Opted in to client beta 'publicbeta' via beta file
Okt 23 05:38:09 jupiter steam[3047]: You are in the 'publicbeta' client beta.
Okt 23 05:38:09 jupiter steam[3047]: Looks like steam didn't shutdown cleanly, scheduling immediate update check
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] Loading cached metrics from disk (/home/kakra/.local/share/Steam/package/steam_client_metrics.bin)
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] Using the following download hosts for Public, Realm steamglobal
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] 1. https://cdn.cloudflare.steamstatic.com, /client/, Realm 'steamglobal', weight was 100, source = 'update_hosts_cached.vdf'
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] 2. https://cdn.akamai.steamstatic.com, /client/, Realm 'steamglobal', weight was 100, source = 'update_hosts_cached.vdf'
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] 3. http://media.steampowered.com, /client/, Realm 'steamglobal', weight was 1, source = 'baked in'
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] Checking for update on startup
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] Suche nach verfügbaren Updates...
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:31] Downloading manifest: https://cdn.cloudflare.steamstatic.com/client/steam_client_publicbeta_ubuntu12
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:32] Download skipped by HTTP 304 Not Modified
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:32] Nothing to do
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:32] Installation wird überprüft...
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:32] Performing checksum verification of executable files
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:20:47] Verification complete
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:42:51] Background update loop checking for update. . .
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:42:51] Suche nach verfügbaren Updates...
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:42:51] Downloading manifest: https://cdn.cloudflare.steamstatic.com/client/steam_client_publicbeta_ubuntu12
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:42:52] Download skipped by HTTP 304 Not Modified
Okt 23 05:38:09 jupiter steam[3047]: [2021-10-23 00:42:52] Nothing to do
Okt 23 05:38:09 jupiter steam[3047]: ***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
Okt 23 05:38:09 jupiter steam[3047]: src/tier0/memstd.cpp (2838) : OUT OF MEMORY
Okt 23 05:38:09 jupiter steam[3047]: src/tier0/memstd.cpp (2838) : OUT OF MEMORY
Okt 23 05:38:09 jupiter steam[3047]: src/tier0/memstd.cpp (2838) : Fatal assert; application exiting
Okt 23 05:38:09 jupiter steam[3047]: src/tier0/memstd.cpp (2838) : Fatal assert; application exiting
Okt 23 05:38:09 jupiter steam[3047]: Installing breakpad exception handler for appid(steam)/version(1634882040)
Okt 23 05:38:09 jupiter steam[133871]: assert_20211023053809_113.dmp[133871]: Uploading dump (out-of-process)
Okt 23 05:38:09 jupiter steam[133871]: /tmp/dumps/assert_20211023053809_113.dmp
Okt 23 05:38:09 jupiter steam[3047]: _ExitOnFatalAssert

When a crash happens in the middle of a game, the game is simply insta-killed together with Steam. This is very annoying. But I seem to have no log of that occurrence (although that happened maybe just a few hours before) so it might be unrelated to update checks.

The Steam client - as an essential "background" service - really needs to become ultra-stable. Maybe concerns need to be split out from the monolithic client, so the Steam API for games becomes a simple single-concern background service, the GUI should be able to be closed separately (so the game could stay running without having the relatively heavy GUI in the background, so it could even crash without taking down the API and the game), and the client updater should be an out-of-process component that is able to properly shutdown the single components and boot them up again.

This is becoming an at least daily thing, now. It happens less frequently, or at least after a longer period of time, if after installing an update to mesa and/or the kernel itself I uncheck background processing, uncheck automatic processing, shut down steam, completely delete the shadercache folder from my library folder, reboot for my updates, then turn background processing again. It might last a day after this, but once it happens it happens consistently until I run the next update.

In a bid to test if maybe it's a particular game causing this to happen and not just random, I uninstalled Monster Hunter World and its associated hi-res texture packs that add a significant disk and processing under normal circumstances. There's nothing pointing at this, but I figured I'd take a shot at it since the error gives literally no indication of what is causing it.

There was also a post on the community forums earlier this year (mid-April 2021) about this and one suggestion stuck out to me near the bottom regarding system memory cache flushing to disk:
https://steamcommunity.com/app/221410/discussions/0/5506189390975205006/

specifically setting the vm.dirty_ratio and vm.dirty_background_ratios to lower numbers to reduce the amount of paging going on before writing to the disk. I did notice after at least two crashes (between different reboots) that the buffered memory in free was incredibly high, which goes along with the thoughts many of us have had that fossilize is running the cache ragged and that, while the system is fine with usable memory, it's running to many large "chunks" of memory before flushing to disk and causing our OOM issues that aren't system-level OOM issues. The defaults in Arch are vm.dirty_background_ratio = 10 and vm.dirty_ratio = 20, so I'm going to look into lowering them to 5 and 10, respectively, and keep an eye on my I/O so I'm not thrashing the SSD. If it at least slows down but doesn't go away I'll try halving them again (the 3 and 1 in the post seems awfully low, IMO) and try to get it to a less crash-happy state.

I just want this fixed, because it's going off the rails a lot faster and I'd really hope we don't want this to still be a bug with a fast-approaching (even if delayed) product launch.

Mine are set to:

vm.dirty_background_ratio = 20
vm.dirty_ratio = 50

with 32 GB of RAM. But I don't think that is the issue here at least for me... Because:

1. System monitoring shows that there's vast amounts of available memory (really free, not used for caching, actually unallocated and unused) so there's no kernel fighting for memory allocations
2. There's no OOM note in dmesg, no increasing HDD thrashing in advance, no increasing load in advance, just the moment it crashes the disks thrashes because memory is dumped to the crash reporter

The *_ratio values are actually too coarse if you want to go that low, you should use the *_bytes variants instead. Generally, you may want to aim background ratio to be as low as you want outstanding dirty data in a process before the kernel forces dirty data to be flushed without blocking processes. On the other side, you want the non-background ratio have enough space for dirty data before the kernel blocks processes from writing more data until the dirty cache is flushed. Going too high with this value may make your system block for multiple minutes until all dirty data is flushed out to disk.

But I still don't think that the issue here. I rather think it's an address space issue because it usually involves 32bit processes which can allocate 2 GB of RAM at most per PID (minus shared memory buffers mapped into the address space, e.g. if pulseaudio is used).

fossilize is a separate issue when it created a huge amount of dirty data due to bad write patterns of graphics drivers, it was fixed by watching the IO pressure of the kernel (with PSI-enabled kernel only, usually stock kernels) and it will immediately turn down the number of running threads when pressure is increasing, then slowly go up again when pressure releases. Together with adjusted lower IO priority and CPU batch scheduling, it should have much better background behavior now and be barely noticeable - both for memory/IO and CPU. I don't think this issue has anything to do with background processing of shaders except if that's running from the same process (but I don't think so). Watching PSI values was actually an idea by me, and I added the patches for IO and CPU priorities. It turned out pretty well and fixed most problems for me (and that issue report has been silent since).

Other settings that may be involved according to your theory:

vm.dirty_writeback_centisecs = 1000
vm.dirty_expire_centisecs = 6000
vm.vfs_cache_pressure = 75

vm.min_free_kbytes = 237610
vm.admin_reserve_kbytes = 262144
vm.watermark_scale_factor = 200
vm.max_map_count = 262120

# because my kernel barely swaps anything without it, thus running into filesystem thrashing issues:
vm.swappiness = 133

But I don't think this plays into the issue.

Been a month, now, still happening. It's not happening with a specific game, specific driver, anything. Only thing left to point at, as kakra has said, is something in the client and a 32-bit process (of which there are many) that ends up overallocating memory, possibly even the Steam itself.

The almost last ditch effort ever, that I expect will not have results, is switching to using the flatpak of Steam to see if maybe, for some reason, under some odd circumstance, there's a difference between it and the one Arch ships or the runtime that gets updated. I'm giving it a few days/weeks of running to see if it can avoid crashing on me, but I'm not expecting much. It's actually happening more often than before, again, and has happened at least 3 times while playing a game over the weekend (losing precious progress in a roguelite!).

With the Deck now scheduled to ship on Feb 25 I'm hoping we see an update to this problem or a way to fix it by then, or around then if we must.

After over half an year with properly working, non-crashing Steam client on Arch Linux, the OOM issue was re-introduced with the latest Steam client update:

[2022-01-29 08:15:34] Background update loop checking for update. . .
[2022-01-29 08:15:34] Tarkistetaan saatavilla olevat päivitykset...
[2022-01-29 08:15:34] Downloading manifest: https://cdn.akamai.steamstatic.com/client/steam_client_ubuntu12
[2022-01-29 08:15:35] Download skipped by HTTP 304 Not Modified
[2022-01-29 08:15:35] Nothing to do
Opted-in Controller Mask for AppId 1451940: 0
***** OUT OF MEMORY! attempted allocation size: 1970089747 ****
src/tier0/memstd.cpp (2838) : OUT OF MEMORY
src/tier0/memstd.cpp (2838) : OUT OF MEMORY
src/tier0/memstd.cpp (2838) : Fatal assert; application exiting
src/tier0/memstd.cpp (2838) : Fatal assert; application exiting
assert_20220129111530_102.dmp[15096]: Uploading dump (out-of-process)
/tmp/dumps/assert_20220129111530_102.dmp
_ExitOnFatalAssert
assert_20220129111530_102.dmp[15096]: Finished uploading minidump (out-of-process): success = yes
assert_20220129111530_102.dmp[15096]: response: CrashID=bp-f5bf05a3-b07c-49d7-b130-c75fa2220129
assert_20220129111530_102.dmp[15096]: file ''/tmp/dumps/assert_20220129111530_102.dmp'', upload yes: ''CrashID=bp-f5bf05a3-b07c-49d7-b130-c75fa2220129''

The amount of memory allocation attempt is exactly the same as before; '1970089747'. A regression?

This happens about once a day. I'm trying to restrict the update download time as before, to see whether it affects anything.

Yeah, I can confirm... After it worked without crashing for the past few weeks (or at least the crashes were very rare), it now crashes within minutes after booting my desktop. If I restart the client, it usually takes less than 2 hours to crash again. After starting it a third time, it eventually keeps running and does not crash.

I also had these daily crashes, but "thanks" to another critical bug ( https://github.com/ValveSoftware/steam-for-linux/issues/8140 ), I'm forced not to load my 2nd library (on a separate HDD) where most of my games are located.
So : with now only 135 games "ready to play" (instead of 2000+), I have no more daily crashes...
Valve should understand these bugs have a big deterrent effect regarding buying new games on Steam.

I also had these daily crashes, but "thanks" to another critical bug ( #8140 ), I'm forced not to load my 2nd library (on a separate HDD) where most of my games are located.

Shader pre-caching seems to be issue for many (for example, Proton issue 5442 touches on this).
Try to disable shader pre-caching.

Did anyone notice: this thread was started exactly one year ago today... and we are basically back to square one.

Happy anniversary, LOL.

I also had these daily crashes, but "thanks" to another critical bug ( #8140 ), I'm forced not to load my 2nd library (on a separate HDD) where most of my games are located.

Shader pre-caching seems to be issue for many (for example, Proton issue 5442 touches on this). Try to disable shader pre-caching.

@ipr
If you speak about the problem that Steam (since version Oct 6 2021) wakes up every 10min each HDD that stores a Steam library, disabling shader pre-caching does nothing. (I also already tried to symlink shadercache folder to my primary SSD, with no result)

Today's update: I started testing the effect of shader pre-caching on this issue after my last comment.

The cache was back then a bit over 4Gb in size; I disabled caching, went without pre-cache for a few days: no crashes.

Then I re-enabled caching. Steam then proceeded to download about 6Gb (ouch!) worth of shaders in one go (another ouch!)... but, after that, I managed to use Steam for some 3 weeks without any issues, until yesterday exactly the same OOM crashes reappeared.

I have a feeling that the crashes may have something to do with some kind of cache corruption (?). I suppose if I disable and re-enable the shader cache again, I might survive a few weeks without crashes. But the huge download...

Well, it's now been a little over 2 months since switching to the Flatpak and... it works fine. Zero crashes ever. During my time of using it since my last post, I had tried switching back to the Arch runtime and it actually crashed even more than usual. During this time I also changed my GPU from an AMD back to an nvidia, but this is fairly inconsequential in this regard.

So now the real problem is figuring out what part of the flatpak is different from the clients packaged by our distros.

7 months later and I'm bumping this to make sure it doesn't get lost, or if maybe someone has long fixed the issue and can provide the solution.

Flatpak still works over the Steam package. I've doubled my system RAM to 32GB since this started, even. There is no discernable reason to be getting OOM unless a piece of the 32-bit packages shipped with Steam are the cause.

Arch Linux using the zen kernel
AMD Ryzen 5 3600
32GB DDR4 3200
GeForce RTX 3060 12GB
NVMe OS drive
SATA SSD for Steam (and only Steam) games.

All my crashes look like the crash happens in 32-bit components which is likely a result of address space exhaustion.

I'm still seeing those crashes but I cannot actually pinpoint that to updates or fossilize running, although it looks like it is mostly happening due to update checks with a big library:

# cd $steam_library_folder
# ls | wc -l
200
# du -sh .
2,5T    .

Crashes happen seemingly random, sometimes even while playing a game (which is probably when Steam explicitly should not be checking for updates, also fossilize shouldn't be running). If such games use the Steam API, they likely crash with it or stop responding properly. Other games simply continue to run but the FPS overlay disappears.

OTOH, Steam happily updates many games in a row, sometimes up to 50, without a single crash. That's why I think the crash isn't really related to updating games per-se, that procedure seems to work just fine. It's something else working in the background, with the OOM condition more likely being triggered while update checks are running.

I can also confirm that sometimes, the clients runs just fine for weeks, and then again it crashes randomly from twice a day to every second day.

This Problem still is present for me on ubuntu 20.04.5 fully patched and last steam client.

Linux 5.4.0-67-lowlatency #75-Ubuntu SMP PREEMPT x86_64 x86_64 x86_64 GNU/Linux
Radeon 580
32gb ddr3 ram
intel i7

deleting and deactivation shader caching allows me to use steam for more that 2 min. now - we'l see - Problem arised 2 weeks ago ¯_(ツ)_/¯

After the Jan 5 beta update that specifically mentioned crashing under Linux, I went back to the normal client instead of the Flatpak version. Everything was going nicely until now. I'm going to give it a bit more time to see if it's the same error or not (I was in the middle of playing a game demo), but I'm not hopeful.

Closing as this didn't happen in a long time for me.

Almost two years later...
OS: Manjaro KDE 25.0.3 Zetar
Steam non-beta up to date.
32GB RAM

Steam (actually steam webhelper) is OOM Killed with a >20GB RAM peak for no reason
https://pastebin.com/3QdgsWmV

Fixed for me since I run it with CEF sandbox disabled: Edit your shortcut (or autorun shortcut) to use steam -no-cef-sandbox %U.