I agree that this is not the same problem as #10412, despite the superficially similar symptoms. In #10412, we don't get as far as the container runtime starting. In this issue, the container starts up fine and hands over control to steamwebhelper, but the steamwebhelper is crashing.

This in steamwebhelper.log looks bad, and perhaps distinctive:

[0125/020544.602831:ERROR:nss_util.cc(357)] After loading Root Certs, loaded==false: NSS error code: -8018

8018 seems to be SEC_ERROR_UNKNOWN_PKCS11_ERROR. Is there anything unusually set up on your system involving PKCS11 or certificates, perhaps?

My home directory is mounted via NFS ... The other issue (with using flock on NFS)

I should mention here that one possible route towards solving #10412 is to use the flock(1) utility to take out a lock on a file or directory in ~/.steam/root/ubuntu12_64, to prevent concurrent access - so perhaps preemptively check that you can do that.

I would recommend putting the Steam installation (usually ~/.local/share/Steam) on a local filesystem that is well-supported by Linux (ext4, xfs, btrfs, that sort of thing), and doing the same for the Steam library that contains your compatibility tools (all versions of Proton and Steam Linux Runtime) if different, even if you also have a secondary Steam library on NFS. Some of the things that the container runtime framework needs to do are metadata operations that have really bad performance on remote filesystems.

Steam Beta is borked, I can reproduce this crash on Arch Linux with clean Steam (removed ~/.steam and ~/.local/share/Steam) and then after enabling Beta

XRRGetOutputInfo Workaround: initialized with override: 0 real: 0xe3432dc0
XRRGetCrtcInfo Workaround: initialized with override: 0 real: 0xe3431500
steamwebhelper.sh[119628]: === piektdiena, 2024. gada 26. janvāris, 22:46:21 EET ===
steamwebhelper.sh[119628]: Starting steamwebhelper under bootstrap sniper steam runtime at ~/.local/share/Steam/ubuntu12_64/steam-runtime-sniper
Steam Runtime Launch Service: starting steam-runtime-launcher-service
Steam Runtime Launch Service: steam-runtime-launcher-service is running pid 119719
bus_name=com.steampowered.PressureVessel.LaunchAlongsideSteam
CAppInfoCacheReadFromDiskThread took 515 milliseconds to initialize
steamwebhelper.sh[119917]: === piektdiena, 2024. gada 26. janvāris, 22:46:31 EET ===
steamwebhelper.sh[119917]: Starting steamwebhelper under bootstrap sniper steam runtime at ~/.local/share/Steam/ubuntu12_64/steam-runtime-sniper
steamwebhelper.sh[120126]: === piektdiena, 2024. gada 26. janvāris, 22:46:42 EET ===
steamwebhelper.sh[120126]: Starting steamwebhelper under bootstrap sniper steam runtime at ~/.local/share/Steam/ubuntu12_64/steam-runtime-sniper
src/steamUI/steamuisharedjscontroller.cpp (545) : Failed creating offscreen shared JS context
src/steamUI/steamuisharedjscontroller.cpp (545) : Failed creating offscreen shared JS context
01/26 22:46:45 Init: Installing breakpad exception handler for appid(steam)/version(1706155871)/tid(119545)
assert_20240126224645_30.dmp[120310]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240126224645_30.dmp
assert_20240126224645_30.dmp[120310]: Finished uploading minidump (out-of-process): success = yes
assert_20240126224645_30.dmp[120310]: response: CrashID=bp-1e48b135-8144-4efd-a813-f1b892240126
assert_20240126224645_30.dmp[120310]: file ''/tmp/dumps/assert_20240126224645_30.dmp'', upload yes: ''CrashID=bp-1e48b135-8144-4efd-a813-f1b892240126''
[2024-01-26 22:46:54] Shutdown

Current workaround is to switch back to non-Beta:

$ rm -f ~/.local/share/Steam/package/beta

@smcv I'm not sure what to look for with regard to the certificates. I don't believe anything is non-standard there. Pointers as to what to check would be appreciated. The machine is my daily-driver and I've not seen anything else fail relating to certs/TLS/etc.

As to the flock issue: The core problem is NFS doesn't support flock(), that only works for local filesystems. The previously mentioned workaround was a tiny C program someone contributed to #5788 called "fakeflock.c".
I just tried that same workaround with this beta, and it did not change the ultimate outcome. However, I did notice a dialog with a moving progress bar that I didn't see yesterday. In the other ticket you mentioned that there likely going to be another update to re-add it. I'm guessing that code has been updated and it's not due to my preloading the libfakeflock library.

I'd rather not move everything to a local filesystem as I have the NFS server for convenient backups and rolling snapshots (every 15 minutes) for the entire family (thankfully, they're not on the beta). Snapshot rollbacks have saved each of us from various catastrophes numerous times. I've been using it this way for well over a decade and it's worked very well for us so far. If NFS is the issue, we're likely not the only ones that will be impacted by this due to the popularity of various DIY/home NAS solutions out there.

In an attempt to test it out, I unmounted, unshared, and relocated my steam filesystem on the server to prevent automatic mounting, then created a Steam subdirectory on a local filesystem with enough space, linked ~/.local/share/Steam back to this location, removed ~/.steam*, the relaunched Steam to force a full re-install. I then logged into Steam, switched over to the beta, at which point it segfaulted (I grabbed the logs, below). I then restarted Steam again and the beta came right up.
So it seems we've managed to narrow it down to NFS.
steam-logs-local.tar.gz

Crash text:

crash_20240126194512_39.dmp[451629]: Uploading dump (out-of-process)
/tmp/dumps/crash_20240126194512_39.dmp
crash_20240126194512_39.dmp[451629]: Finished uploading minidump (out-of-process): success = yes
crash_20240126194512_39.dmp[451629]: response: CrashID=bp-9cc63e34-48bf-4e6b-bbf3-a3ec02240126
crash_20240126194512_39.dmp[451629]: file ''/tmp/dumps/crash_20240126194512_39.dmp'', upload yes: ''CrashID=bp-9cc63e34-48bf-4e6b-bbf3-a3ec02240126''
/var/cache/fscache/Steam/steam.sh: line 798: 450752 Segmentation fault      (core dumped) "$STEAMROOT/$STEAMEXEPATH" "$@"

@davispuh Is your home directory (or Steam installation path) mounted via NFS?
If not, what filesystem are you using? What mount options? I think we're making progress on narrowing this down.

@davispuh Is your home directory (or Steam installation path) mounted via NFS? If not, what filesystem are you using? What mount options? I think we're making progress on narrowing this down.

Well it's bit complicated :joy: My home directory is not using NFS but local btrfs but I do have it exported with NFS and I'm also using bind mounts, subvolumes, btrfs raid1 and md raid6.

Here's summary of mounts:

/dev/nvme1n1p2 on / type btrfs (rw,noatime,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/Arch)
/dev/sdp       on /home type btrfs (rw,noatime,compress=lzo,space_cache=v2,subvolid=593,subvol=/home,x-systemd.automount)
/dev/md127     on /mnt/Data type btrfs (rw,noatime,compress=zstd:3,space_cache=v2,subvolid=257,subvol=/Data,x-systemd.automount,x-systemd.mount-timeout=10m)
/dev/sdp       on /mnt/RAID type btrfs (rw,noatime,compress=lzo,space_cache=v2,subvolid=395,subvol=/RAID,x-systemd.automount,x-systemd.mount-timeout=10m)
/dev/sdp       on /srv/nfs/RAID type btrfs (rw,noatime,compress=lzo,space_cache=v2,subvolid=395,subvol=/RAID,x-systemd.automount)

That last /dev/sdp is bind mount, fstab looks like this:

UUID=ee7e665c-3de5-43e3-80b8-d312bdf58dae  /  btrfs  rw,noatime,noautodefrag,ssd,space_cache=v2,subvol=Arch  0  0
UUID=cf489774-f2f9-4d80-9cb7-08ebad25bfb3  /home  btrfs  rw,noatime,noautodefrag,space_cache=v2,compress=lzo,subvol=home,noauto,x-systemd.automount  0  0
UUID=cf489774-f2f9-4d80-9cb7-08ebad25bfb3  /mnt/RAID  btrfs  rw,noatime,noautodefrag,space_cache=v2,compress=lzo,subvol=RAID,noauto,x-systemd.automount,x-systemd.device-timeout=10m,x-systemd.mount-timeout=10m  0  0
UUID=502744d5-e441-47d1-ab41-bcf2eb800e2f  /mnt/Data  btrfs  rw,noatime,space_cache=v2,compress=zstd,subvol=Data,noauto,x-systemd.automount,x-systemd.device-timeout=10m,x-systemd.mount-timeout=10m  0  0
/mnt/RAID                                  /srv/nfs/RAID  none   bind,noauto,x-systemd.automount  0  0

And exports

/srv/nfs/RAID       172.16.0.0/25(rw,fsid=100,insecure,no_subtree_check,anonuid=20000,anongid=20000) 172.24.0.0/25(rw,fsid=100,insecure,no_subtree_check,anonuid=20000,anongid=20000)
/home/Dāvis         172.16.0.0/28(rw,fsid=101,insecure,no_subtree_check,anonuid=20000,anongid=20000) 172.24.0.0/28(rw,fsid=101,insecure,no_subtree_check,anonuid=20000,anongid=20000)

PS. These are just relevant excerpts

@davispuh Unfortunately, I know very little about brtfs, so I don't know where the overlap between its capabilities and NFSv4 would be.

@smcv I forgot to mention that I'm running NFSv4, which has different locking mechanisms than NFSv3 (neither of which natively support flock).
I just tried flock_to_setlk from #5788 by @DataBeaver (compiled into both 32 bit and 64 bit versions, both preloaded via LD_PRELOAD) with no luck. While flock is triggered many times, this workaround (which is more functional than the fakeflock one) did not solve the problem.
Further down in that issue discussion, @eqvinox has an excellent description of the limitations of NFSv4 and which calls to use.

I did some testing due to getting pinged. After switching to the beta, at first the Steam UI didn't show up at all and I also didn't get this "not responding" dialog. This reproduced a couple of times. I then tried running Steam on a local drive (ext4 filesystem), which worked. After that running on the NFS mount worked too. To further confirm the functionality I tried it on another computer, with the same NFS mount. There I got the "not responding" dialog, but the Steam UI showed up and worked as well. After choosing to restart either steam or just steamwebhelper the dialog did not reappear. I have to get on with other things, but maybe I'll try a clean install on NFS later.

Both computers are running Debian unstable and have Nvidia GPUs with driver version 525.147.05.

Edit: I ran the local Steam installation by changing HOME to point at a local mount. It's not impossible that it could have affected the installation on NFS, though it definitely did run from the local drive.

Following DataBeaver's lead, I redid my fresh local install per my above description (using a link).
I then copied (via tar to preserve links) the installation back to NFS to see if it was only the installer with the issue, or if it was a post-installation problem.
I launched Steam and ended up with the same dialog, so I believe the installer is ruled out.

However, one thing I noticed was a large number of errors when running a diff between the local and NFS copies. I believe all of these were dead links.

~/.steam$ find root/ -xtype l -exec test ! -e {} \; -ls | wc -l
411

The confusing part is, this is true of both installations, so I'm not even sure how sniper can run in the local installation if this is causal in the NFS installation (so this very well may be a red-herring).

Digging into them in more detail, most appear to be dead links to the /run/ hierarchy, so can be ignored.

~/.steam$ find root/ -xtype l -exec test ! -e {} \; -ls  | grep -v ' /run/' | wc -l
101

Filtering down to absolute path links, I see the certificate issue you mentioned. It appears to be looking for different filenames than I have on my system, as I see a mixture of near-misses (eg: different numbers) and completely missing ones (eg: Staat*).
From my below findings, this is likely not relevant, but I'm leaving this here for context.

Next I see that it's linking to font configs that don't exist in /usr/share/fontconfig/conf.avail/ (I have 29 total in that directory, perhaps I'm missing a package?).
Again, this is likely not relevant, given the below.

Digging into a few of the oddballs at the end of the list, it's become apparent to me that this runs in some sort of chrooted environment with multiple filesystem overlays or something along those lines (sorry, I'm unfamiliar with exactly what the steam-runtime-* installations do), as I'm finding the absolute paths stuffed under various different hierarchies in the Steam installation.
Examples:
/etc/python3.9/sitecustomize.py is found under ubuntu12_64/steam-runtime-sniper/sniper_platform_0.20240125.75305/files/
/usr/lib/i386-linux-gnu/libXaw.so.7 is found under ubuntu12_32/steam-runtime/

Extrapolating this back to the certs, and it looks like they are actually there too, but in yet another subdirectory. For example,
/usr/share/ca-certificates/mozilla/Staat_der_Nederlanden_EV_Root_CA.crt is in ubuntu12_64/steam-runtime-sniper/var/tmp-74JLI2/

Backing up and filtering for relative paths, I find a few additional causes.
Some are simply missing targets (eg: 4 occurrences of an selinux link that point to a non-existing entry in its parent) where the file doesn't exist in the entire installation.
The certs seem to be pointing to the wrong directory. For example the links in the */usr/etc/ssl/certs/ directory point to the current directory, while the files are actually found in */etc/ssl/certs/ instead (/etc instead of /usr/etc, but both the link and extant file paths are under the ubuntu12_64/steam-runtime-sniper/var/tmp-74JLI2 path).
The os-release link is specified one level too deep (would resolve to */usr/usr/lib/os-release).
This appears to also be true of all of the dead library links.

Here's the list of dead links with the /run/ links stripped out:
dead-links.txt

I can't tell at this stage whether the problem that @glabifrons is having is to do with NFS or not, so this might all be a red herring. But, we are going to need this sooner or later, so...

The core problem is NFS doesn't support flock()

Does it support POSIX process-associated record locks (fcntl F_SETLKW) and/or Linux open-file-description locks (fcntl F_OFD_SETLKW)?

We are going to need to put some sort of locking into place, otherwise we get bizarre failure modes like one process deleting a temporary runtime that another process is still using. Sorry, but avoiding that is more important than supporting NFS. If flock(1) and flock(2) are unavailable, a different locking mechanism is a possibility, but having no locking at all is not really an option.

At the moment, the container runtime tries to use the Linux-specific fcntl F_OFD_SETLKW, falling back to POSIX fcntl F_SETLKW on ancient kernels. You could test this: with Steam not running,

adverb="$HOME/.steam/root/ubuntu12_64/steam-runtime-sniper/pressure-vessel/bin/pressure-vessel-adverb"
ref="$HOME/.steam/root/ubuntu12_64/steam-runtime-sniper/.ref"

"$adverb" --write --wait --lock-file="$ref" -- sleep 600 &
"$adverb" --write --lock-file="$ref" -- true
"$adverb" --write --wait --lock-file="$ref" -- true

(If necessary, copy the whole steam-runtime-sniper directory into a temporary location on NFS, and adjust the paths accordingly.)

The "$adverb" --write --lock-file="$ref" -- true command (without --wait) should fail with error message "E: Unable to lock ... for writing: file is busy".

The last command (with --wait) should block, with no output, until you kill the sleep process (or wait 10 minutes for it to exit on its own), at which point the last command should exit successfully.

The previously mentioned workaround was a tiny C program someone contributed to https://github.com/ValveSoftware/steam-for-linux/issues/5788 called "fakeflock.c"

Disabling locking like this is not a solution. This will lead to concurrent processes all believing that they have the lock at the same time, and overwriting or deleting files that the other concurrent processes were using.

@davispuh:

We do not have enough information on this issue to be able to guess whether the failure mode you are seeing with the beta is the same as @glabifrons is seeing, or the same as #10412, or some different thing. Please look at the logs in ~/.steam/root/logs/, especially steamwebhelper.log and webhelper.txt.

If we try to handle multiple different problems on the same issue number, it quickly becomes really confusing, which makes it take longer to solve any of the problems that were reported; so we should reserve this issue number for the specific problem that @glabifrons is experiencing (which unfortunately we have not yet been able to identify). If we can identify that something different is going wrong for you, please open a separate issue for that, with a title that is as specific as possible.

My home directory is not using NFS but local btrfs but I do have it exported with NFS and I'm also using bind mounts, subvolumes, btrfs raid1 and md raid6.

I don't know whether any of these will interfere with the container runtime. My first guess would be that RAID shouldn't matter, because that's at a lower level than anything we're doing, but the others might. If you can try launching Steam on the same system but from a home directory that is as "ordinary and boring" as possible (perhaps by creating a temporary user whose home directory is on local disk and is not NFS-exported, and logging in as that user) then that will help to narrow down whether any of these less-usual configurations are involved.

@DataBeaver:

There I got the "not responding" dialog, but the Steam UI showed up and worked as well

Unfortunately, I think this is normal if it takes an unusually long time for the steamwebhelper to start. Steam cannot currently distinguish between multiple different reasons why the steamwebhelper might fail to start, and it also cannot currently distinguish between "it's taking a long time, but might still work" and "it will never work, however long we wait".

If your NFS mount has enough latency to make small metadata operations like link(2) and chmod(2) unexpectedly slow, then it's going to take a while to start. We've seen this before with Steam libraries on other network filesystems like SMB.

Back to @glabifrons:

it's become apparent to me that this runs in some sort of chrooted environment with multiple filesystem overlays or something along those lines

Yes, it's the Steam container runtime, which has quite a lot of code in common with Flatpak. It's normal that some of the files below steam-runtime-sniper/ are symbolic links to filenames that don't exist on your host system. As long as those symlinks work correctly inside the container, everything is fine.

Looking at your list of dangling symlinks, the majority of them are very likely to work as intended inside the container. I do notice one bug, but it's a bug that will only affect developers who are running this stuff in a non-default configuration that isn't relevant to end-user systems.

If you are copying Steam installations between filesystems, you can delete all of steam-runtime-sniper/var/ instead of copying it: the subdirectories in there are temporary, and are deleted and re-created automatically. In fact, you could even delete all of steam-runtime-sniper/, because Steam will automatically unpack it from steam-runtime-sniper.tar.xz.

You can verify that steam-runtime-sniper/ has the expected contents by running:

~/.steam/root/ubuntu12_64/steam-runtime-sniper/pressure-vessel/bin/pv-verify

This checks both metadata and content of all of the files in there, so expect it to take up to 30 seconds on HDD, and perhaps longer on NFS.

You can also get an interactive shell inside the container by running:

~/.steam/root/ubuntu12_64/steam-runtime-sniper/run -- xterm

Inside that xterm, you should find that all the symbolic links in /etc/ssl/certs are working (they have a valid target, and if you use ls --color they will typically appear in cyan rather than red).

It would be useful for me to see a detailed log from the container runtime framework, which you can get by running:

STEAM_LINUX_RUNTIME_LOG=1 \
STEAM_LINUX_RUNTIME_VERBOSE=1 \
~/.steam/root/ubuntu12_64/steam-runtime-sniper/run -- xterm

(You can just exit from the xterm when it has opened, or run a simpler command like true.)

The log file will appear in steam-runtime-sniper/var/, with a symbolic link slr-latest.log that points to it.

I'd rather not move everything to a local filesystem as I have the NFS server for convenient backups and rolling snapshots

I'm sure that's desirable, but remote filesystems have functionality and performance characteristics that are very much unlike local filesystems, and we can't support every possible scenario.

As currently implemented, the whole steam-runtime-sniper/ directory is actually "expendable": it needs to exist while Steam is running, but it does not contain any user data, so the only thing that is lost when it's deleted (assuming it isn't in active use) is some time. If we can set up some sort of mechanism for redirecting this from its normal location onto a local disk, then that would make it faster and more robust for you, and would also avoid it wasting space and time in your backups.

At the moment the way it's implemented doesn't allow for it to be a symlink or a mount point, but I'll see whether that can become possible in future.

The core problem is NFS doesn't support flock()

Does it support POSIX process-associated record locks (fcntl F_SETLKW) and/or Linux open-file-description locks (fcntl F_OFD_SETLKW)?

A quick look at the relevant manpages tells me that while NFS doesn't support flock natively, Linux can emulate it using fcntl locks, albeit with slightly different semantics. There's no mention if whether it's F_SETLKW or F_OFD_SETLKW, nor does the fcntl manpage say whether those two have any difference over NFS. I assume they work the same, since the only difference is that OFD locks will also block other file descriptors from the same process, and from the NFS protocol's point of view it doesn't matter if it's two different processes on the same client or two fds in the same process.

The "$adverb" --write --lock-file="$ref" -- true command (without --wait) should fail with error message "E: Unable to lock ... for writing: file is busy".

The last command (with --wait) should block, with no output, until you kill the sleep process (or wait 10 minutes for it to exit on its own), at which point the last command should exit successfully.

This works for me on my NFS home directory. As does Steam itself. So I think NFS is at most a contributing factor, not the root cause. It will be interesting to see glabifrons's results for the lock test. Could be that we have some configuration differences.

There I got the "not responding" dialog, but the Steam UI showed up and worked as well

Unfortunately, I think this is normal if it takes an unusually long time for the steamwebhelper to start. Steam cannot currently distinguish between multiple different reasons why the steamwebhelper might fail to start, and it also cannot currently distinguish between "it's taking a long time, but might still work" and "it will never work, however long we wait".

Understandable. It's a relatively minor annoyance, but if you want to do something about it, maybe add an option to wait a bit longer without restarting anything? Or even keep checking for responsiveness while the dialog is up, and hide it if steamwebhelper starts responding after all.

Hmm I thought this is only issue for steamwebhelper crash due to new Beta but looks like there are several issues causing crashes...

My crash is not #10412 because my steam-runtime-sniper is complete without missing files and I also tried

$ rm -rf ~/.steam/root/ubuntu12_64/steam-runtime-sniper

but that didn't change anything and

$ ~/.steam/root/ubuntu12_64/steam-runtime-sniper/run-in-sniper vkcube

works fine without issues.

In logs nothing in particular stands out

$ cat ~/.steam/steam/logs/steamwebhelper.log
steamwebhelper.sh[43847]: Starting steamwebhelper with sniper steam runtime at /mnt/Games/SteamLinux/ubuntu12_64/steam-runtime-sniper
exec ./steamwebhelper --no-sandbox --no-sandbox -lang=en_US -cachedir=/mnt/Games/SteamLinux/config/htmlcache -steampid=42835 -buildid=1706390103 -steamid=xxx -logdir=/mnt/Games/SteamLinux/logs -uimode=7 -startcount=2 -steamuniverse=Public -realm=Global -clientui=/mnt/Games/SteamLinux/clientui -steampath=/mnt/Games/SteamLinux/ubuntu12_32/steam -launcher=0 -no-restart-on-ui-mode-change --enable-media-stream --enable-smooth-scrolling --password-store=basic --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --disable-quick-menu --disable-features=DcheckIsFatal
[0129/194030.399588:ERROR:context.cc(100)] The browser_subprocess_path directory (./steamwebhelper) is not an absolute path. Defaulting to empty.
[0129/194030.426813:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 0
[0129/194030.426864:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1706389061
[0129/194030.426867:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0129/194030.426870:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0129/194030.426873:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0129/194030.427856:INFO:crash_reporting.cc(239)] Crash reporting enabled for process: browser
[0129/194030.429320:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0129/194030.430817:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0129/194030.457887:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: xxx
[0129/194030.457969:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1706390103
[0129/194030.457973:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0129/194030.457976:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0129/194030.457980:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0129/194030.461011:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: xxx
[0129/194030.461099:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1706390103
[0129/194030.461102:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0129/194030.461105:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0129/194030.461108:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0129/194031.302189:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq: No such file or directory (2)
[0129/194031.302222:ERROR:file_io_posix.cc(144)] open /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq: No such file or directory (2)

$ cat ~/.steam/steam/logs/webhelper.txt
[...]
[1970-01-01 03:00:00] Client version: no bootstrapper found
[1970-01-01 03:00:00] Startup - webhelper launched with: ./steamwebhelper --no-sandbox -lang=en_US -cachedir=/mnt/Games/SteamLinux/config/htmlcache -steampid=42835 -buildid=1706390103 -steamid=xxx -logdir=/mnt/Games/SteamLinux/logs -uimode=7 -startcount=2 -steamuniverse=Public -realm=Global -clientui=/mnt/Games/SteamLinux/clientui -steampath=/mnt/Games/SteamLinux/ubuntu12_32/steam -launcher=0 -no-restart-on-ui-mode-change --enable-media-stream --enable-smooth-scrolling --password-store=basic --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --disable-quick-menu --disable-features=DcheckIsFatal
[1970-01-01 03:00:00] Disabling sandbox due to a previous crash in CefInitialize with the sandbox enabled
[1970-01-01 03:00:00] Browser - launching child process with: /mnt/Games/SteamLinux/ubuntu12_64/steamwebhelper --type=zygote --no-zygote-sandbox --no-sandbox --user-agent-product=Valve Steam Client --lang=en_US.UTF-8 --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --crashpad-handler-pid=43969 --buildid=1706390103 --steamid=xxx
[1970-01-01 03:00:00] Browser - launching child process with: /mnt/Games/SteamLinux/ubuntu12_64/steamwebhelper --type=zygote --no-sandbox --user-agent-product=Valve Steam Client --lang=en_US.UTF-8 --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --crashpad-handler-pid=43969 --buildid=1706390103 --steamid=xxx


[1970-01-01 03:00:00] Client version: no bootstrapper found
[1970-01-01 03:00:00] Startup - webhelper launched with: ./steamwebhelper --no-sandbox --no-sandbox -lang=en_US -cachedir=/mnt/Games/SteamLinux/config/htmlcache -steampid=42835 -buildid=1706390103 -steamid=xxx -logdir=/mnt/Games/SteamLinux/logs -uimode=7 -startcount=2 -steamuniverse=Public -realm=Global -clientui=/mnt/Games/SteamLinux/clientui -steampath=/mnt/Games/SteamLinux/ubuntu12_32/steam -launcher=0 -no-restart-on-ui-mode-change --enable-media-stream --enable-smooth-scrolling --password-store=basic --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --disable-quick-menu --disable-features=DcheckIsFatal
[1970-01-01 03:00:00] Disabling sandbox due to a previous crash in CefInitialize with the sandbox enabled
[1970-01-01 03:00:00] Browser - launching child process with: /mnt/Games/SteamLinux/ubuntu12_64/steamwebhelper --type=zygote --no-zygote-sandbox --no-sandbox --user-agent-product=Valve Steam Client --lang=en_US.UTF-8 --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --crashpad-handler-pid=44120 --buildid=1706390103 --steamid=xxx
[1970-01-01 03:00:00] Browser - launching child process with: /mnt/Games/SteamLinux/ubuntu12_64/steamwebhelper --type=zygote --no-sandbox --user-agent-product=Valve Steam Client --lang=en_US.UTF-8 --log-file=/mnt/Games/SteamLinux/logs/cef_log.txt --crashpad-handler-pid=44120 --buildid=1706390103 --steamid=xxx

$ cat ~/.steam/root/logs/cef_log.txt
[0129/194027.772949:INFO:crash_reporting.cc(239)] Crash reporting enabled for process: browser
[0129/194027.774154:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0129/194027.775646:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0129/194030.427856:INFO:crash_reporting.cc(239)] Crash reporting enabled for process: browser
[0129/194030.429320:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0129/194030.430817:WARNING:task_impl.cc(32)] No task runner for threadId 0

And here are backtraces but I don't know how to get symbols for it?

#0  0x000070f2f7d3f003 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x8b3f003)
#1  0x000070f2f7d2dc3f n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x8b2dc3f)
#2  0x000070f2f372235c n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x452235c)
#3  0x000070f2f3c811c5 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x4a811c5)
#4  0x000070f2f372269f n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x452269f)
#5  0x000070f2f3724dec n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x4524dec)
#6  0x000070f2f142cc3d n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x222cc3d)
#7  0x000070f2f14a84ac n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x22a84ac)
#8  0x000070f2f4fa63b0 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x5da63b0)
#9  0x000070f2f4fa7670 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x5da7670)
[#10](/issue/ValveSoftware/steam-for-linux/10) 0x000070f2f4fa7454 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x5da7454)
[#11](/issue/ValveSoftware/steam-for-linux/11) 0x000070f2f4fa4c41 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x5da4c41)
[#12](/issue/ValveSoftware/steam-for-linux/12) 0x000070f2f142bc32 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x222bc32)
[#13](/issue/ValveSoftware/steam-for-linux/13) 0x000070f2f142b995 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x222b995)
[#14](/issue/ValveSoftware/steam-for-linux/14) 0x000070f2f13fdac5 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x21fdac5)
[#15](/issue/ValveSoftware/steam-for-linux/15) 0x000070f2f13fd654 n/a (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x21fd654)
[#16](/issue/ValveSoftware/steam-for-linux/16) 0x000070f2f137336b cef_initialize (/mnt/Games/SteamLinux/ubuntu12_64/libcef.so + 0x217336b)
[#17](/issue/ValveSoftware/steam-for-linux/17) 0x00000000005d077a CefInitialize(CefMainArgs const&, CefStructBase<CefSettingsTraits> const&, scoped_refptr<CefApp>, void*) (steamwebhelper + 0x1d077a)
[#18](/issue/ValveSoftware/steam-for-linux/18) 0x0000000000519b1a CCEFThread::Init(int, char**, scoped_refptr<CefApp>) (steamwebhelper + 0x119b1a)
[#19](/issue/ValveSoftware/steam-for-linux/19) 0x0000000000517839 InitializeCef(int, char**, scoped_refptr<CefApp>) (steamwebhelper + 0x117839)
[#20](/issue/ValveSoftware/steam-for-linux/20) 0x0000000000590323 main (steamwebhelper + 0x190323)
[#21](/issue/ValveSoftware/steam-for-linux/21) 0x000070f2ee924cd0 n/a (/run/host/usr/lib/libc.so.6 + 0x27cd0)

#0  0x00007fefe313f003 n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x8b3f003)
#1  0x00007fefe312dc3f n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x8b2dc3f)
#2  0x00007fefe03a5494 n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x5da5494)
#3  0x00007fefe03a66ff n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x5da66ff)
#4  0x00007fefe03a7428 n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x5da7428)
#5  0x00007fefe03a4d46 n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x5da4d46)
#6  0x00007fefe03a4e3a n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x5da4e3a)
#7  0x00007fefdc82c391 n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x222c391)
#8  0x00007fefdc7fd517 n/a (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x21fd517)
#9  0x00007fefdc773260 cef_execute_process (/mnt/Game/SteamLinux/ubuntu12_64/libcef.so + 0x2173260)
[#10](/issue/ValveSoftware/steam-for-linux/10) 0x00000000005d069f CefExecuteProcess(CefMainArgs const&, scoped_refptr<CefApp>, void*) (steamwebhelper + 0x1d069f)
[#11](/issue/ValveSoftware/steam-for-linux/11) 0x000000000058fc7e RealMain(int, char**) (steamwebhelper + 0x18fc7e)
[#12](/issue/ValveSoftware/steam-for-linux/12) 0x0000000000590e38 main (steamwebhelper + 0x190e38)
[#13](/issue/ValveSoftware/steam-for-linux/13) 0x00007fefd9d24cd0 n/a (/run/host/usr/lib/libc.so.6 + 0x27cd0)

Hmm I thought this is only issue for steamwebhelper crash due to new Beta but looks like there are several issues causing crashes...

The steamwebhelper has had several major changes in the new beta. Some issues caused by those changes (like #10412) are to do with the fact that it is now running inside a container runtime, like Counterstrike 2 and Dota 2 do. Others could be to do with changes inside the steamwebhelper itself.

At the moment, unfortunately I don't see enough information here to be able to say whether you are experiencing the same crash as the original reporter of this particular issue or not.

$ cat ~/.steam/steam/logs/webhelper.txt

In the beta that was active over the weekend, the log was truncated every time the steamwebhelper restarted, which was unhelpful because it meant that previous error messages could be lost. Please try updating Steam to the current beta 1706390103, which has stopped truncating the log every time, so should get you better logs.

You might need to do this by swapping to the stable branch (completely exit from Steam, rm ~/.steam/root/package/beta, start it again), and then back to the beta.

And here are backtraces but I don't know how to get symbols for it?

The general public cannot get debug symbols for the proprietary parts of Steam (and neither can I), but Valve can. In your original report, you quoted log output that said Steam has uploaded a crash dump, CrashID=bp-1e48b135-8144-4efd-a813-f1b892240126. That could be useful information to a Valve developer. If you can get a similar CrashID from the current beta, that will also help.

All information in my comment is with latest Beta version (today installed), there you can see steamwebhelper was ran with -buildid=1706390103 parameter. Also I deleted logs folder before run so there isn't old info.

Looks like it crashes very early in startup as there isn't any other log entries after Browser - launching child process with.

In non-Beta version I see

[2024-01-29 21:16:20] CreateBrowser 512367304 type:12 flags:0  (-2147483648, -2147483648) 0x0

But this is not present in Beta we never reach it.

Crash seems to be inside libcef.so - Chromium Embedded Framework (CEF) which is open source so I think even with Valve's modifications it might be possible to match it up with relevant functions with some reversing but that's not our job.

So they need to look into this. My latest CrashID=bp-5aaf6174-a286-49b8-a3b3-eb2fe2240129

@smcv I tried the adverb commands both on the local filesystem and on the NFS installation, and surprisingly it worked for both. So it looks like it's not a locking issue.

Just in case the information is useful though, this post has the best description of the limitations of the NFSv4 calls (IIRC, there's one for read and one for write, but none for both):
https://github.com/ValveSoftware/steam-for-linux/issues/5788#issuecomment-511219089
Two posts below that, @eqvinox has a short explanation of how the Linux kernel translates the calls (so it does add that, but they don't look like the ones you asked about).

Another thing you mentioned is compatibility for old kernels. I may actually have the opposite problem, as I'm running the HWE kernel: 6.5.0-15-generic. I wonder if there might be negative interaction with the newer kernels. @davispuh what kernel are you using?

I ran pv-verify in both installations, and on my NVME drive it took 2.8s, and on NFS it took 5.5s.
Both came back verified on both checks it performs.

I tried your other command to launch an xterm within sniper and verified all symlinks look good.

I tried to generate a log for you as requested, but no logs ever appear in either the var under steam on NFS or the local installation. Only the .ref file and a temp subdirectory (tmp-$randomchars) in my local installation and several of those subdirs in my NFS one.
I updated the beta to the latest (released on the 26th... how do you identify the version number?) and the results were the same - no log. I tried exporting the variables (thinking maybe there was another subshell being triggered), same results, no log in steam-runtime-sniper/var.
Since I couldn't generate those logs (in local or NFS installations), I re-collected the main logs as initially requested after the above mentioned upgrade to the latest beta.
steam-logs-jan26beta.tar.gz

I like your thoughts on relocation.
Sniper is well under 1GB, so /tmp or /var/tmp would likely work for most (and /tmp would make the most sense).
It could be a manual process with links done by experienced users, or maybe even something from a path entry dialog down the road labeled something like "Copy temporary runtime to local filesystem" available only if the system detects that it's running in a remote filesystem.

I do have one thought that I hope is a stupid question: Steam doesn't attempt to launch anything as root, does it?
I ask as if it does so in my home directory, it will fail as root is "squashed" on NFS by default (its a security precaution).
Any actions a root from a client show up as user "nobody" from the server's perspective, so will fail any operation on any path that's not world-read/execute (at a minimum) perms all the way up.

One other thing... you mentioned you don't have access to debug symbols, but Valve can... with as much effort as you're putting into this and as knowledgeable as you are on how the inner workings and even the development direction, I thought you were an employee of Valve!

I was thinking about NFS and root-squash and the overlays and I think I figured out at least part of what's going on.
From within the sniper shell (the xterm launched per your instructions), all device files (the entire output of ls -l /dev/) are shown as owned by nobody:nogroup.

I don't pretend to understand what type of container sniper is using or how it's overlaying filesystems, but I find this really strange as up until the 20th, not only did it work, but many of the games I play use Proton (I recall you saying sniper is related in the other issue thread) and I even use Proton Experimental for some of them, like Space Engineers).

It's still difficult to tell from the information available, but the best diagnosis I can make so far is that @glabifrons might be seeing a steamwebhelper crash that is not directly related to the introduction of the container runtime, similar to @davispuh. If that's the case, then a Steam developer will have to take over investigating this.

@glabifrons, if you can find a CrashID in recent Steam output (it will look similar to the one @davispuh provided) then that would probably be useful information for a Steam developer. Equally, if you can't find a CrashID anywhere, that would also be an interesting data point - it might imply that I'm wrong about this being a steamwebhelper crash.

I still think that [0129/213808.640966:ERROR:nss_util.cc(357)] After loading Root Certs, loaded==false: NSS error code: -8018 in your cef_log.txt might be significant, but I don't know why that would happen to you but not to others. It sounds as though the CA certificates in the sniper container are set up correctly (their symbolic links are not broken in the container environment). A container runtime verbose log might help to figure out what is different about your system.

@davispuh, am I correct to think that you don't see NSS error code: -8018 in your cef_log.txt or other logs?

I tried to generate a log for you as requested, but no logs ever appear in either the var under steam on NFS or the local installation

Sorry, I was forgetting which layer is responsible for implementing slr-latest.log. The command I should have suggested is:

STEAM_LINUX_RUNTIME_LOG=1 \
STEAM_LINUX_RUNTIME_VERBOSE=1 \
~/.steam/root/ubuntu12_64/steam-runtime-sniper/_v2-entry-point -- xterm

This should record a log like I said.

Another way to get logs would be to exit from Steam completely, and then run it as:

STEAM_LINUX_RUNTIME_LOG=1 \
STEAM_LINUX_RUNTIME_KEEP_LOGS=1 \
STEAM_LINUX_RUNTIME_VERBOSE=1 \
steam

which should record one log in steam-runtime-sniper/var each time it tries to restart the steamwebhelper, with slr-latest.log always pointing to the newest one.

the .ref file and a temp subdirectory (tmp-$randomchars) in my local installation and several of those subdirs in my NFS one

This is interesting. Normally (if you don't use STEAM_LINUX_RUNTIME_KEEP_LOGS=1), the container runtime is meant to delete old var/tmp-XXXXXXX subdirectories during startup - so at any given time, you should usually only have one.

It sounds as though your installation on a local disk is working correctly, but the old subdirectories are not being garbage-collected on your NFS installation. I'd be interested to see why not. If you can get a log file, it should tell us why.

how do you identify the version number?

Normally you would use Help → About Steam, but when your issue is that the UI isn't starting, obviously that isn't an option.

As @davispuh said, one good indication is that each time Steam runs the steamwebhelper, it passes it an argument -buildid=1706390103 which gives the build ID. Many of the other logs also mention the build ID, for example [2024-01-29 21:36:16] Client version: 1706390103 in console_log.txt and [2024-01-29 21:37:22] Download skipped: /steam_client_publicbeta_ubuntu12?t=3744699689 version 1706390103, installed version 1706390103, existing pending version 0 in bootstrap_log.txt.

I tried the adverb commands both on the local filesystem and on the NFS installation, and surprisingly it worked for both

OK, good. It sounds as though we cannot rely on flock(1) or flock(2) on NFS, but the fcntl locking that we already use should be safe. I'll bear that in mind for future work on this topic.

From within the sniper shell (the xterm launched per your instructions), all device files (the entire output of ls -l /dev/) are shown as owned by nobody:nogroup.

This may seem weird, but is normal. When an unprivileged user creates a new user namespace, as we do in the Steam container runtime, the kernel will only allow us to create one user ID mapping (our own user ID) and one group ID mapping (our own primary group ID). Everything else is mapped to the "overflow uid" and "overflow gid" (normally nobody:nogroup), very similar to NFS root-squashing. Files owned by root and files owned by any other user will show up inside the container as though they were owned by the overflow uid, which you should interpret as meaning "owned by someone who is not me".

Flatpak apps have the same behaviour, for the same reason.

Steam doesn't attempt to launch anything as root, does it?

Not usually, and not on the critical path for basic UI functionality. In some situations (mainly either related to VR, or on the Steam Deck) it will try to run commands via pkexec.

Another thing you mentioned is compatibility for old kernels. I may actually have the opposite problem, as I'm running the HWE kernel: 6.5.0-15-generic.

My main test environments for new container runtime releases are Ubuntu 22.04 (with the same HWE kernel you're using) and Arch (with a very new kernel, currently 6.7), so it's heavily tested on modern kernels.

I thought you were an employee of Valve

I'm a consultant helping them with the Steam Runtime and related topics. If your particular issue is a problem with the container runtime, my team might be able to fix it; if it's a problem with steamwebhelper itself, probably someone else will need to take over.

I just launched the beta in the foreground again and got two crash-IDs.

assert_20240130192454_27.dmp[1678661]: response: CrashID=bp-76d14b36-100b-48a3-aec4-615e12240130
assert_20240130192454_27.dmp[1678661]: file ''/tmp/dumps/assert_20240130192454_27.dmp'', upload yes: ''CrashID=bp-76d14b36-100b-48a3-aec4-615e12240130''

assert_20240130192527_134.dmp[1679205]: response: CrashID=bp-722467e1-6953-4b8a-9ad0-01f382240130
assert_20240130192527_134.dmp[1679205]: file ''/tmp/dumps/assert_20240130192527_134.dmp'', upload yes: ''CrashID=bp-722467e1-6953-4b8a-9ad0-01f382240130''

I can provide more output for context if needed.

I was able to get a log using the _v2-entry-point, so thank you for the correction.
Interestingly, it deletes the old log when it creates a new one, but still doesn't clean up the tmp-$random directories.
This one was just using true as the target executable.
slr-non-steam-game-t20240130T203006.log

Thank you very much for the overflow uid explanation. That's a huge relief that it's not what I was afraid of, as that would have meant that solution wasn't NFS compatible. I'm glad you mentioned Steam's VR... I guess I'll be putting off playing with that for a while (was looking at it recently due to a deal on woot that I almost bought).

Hopefully the above log is helpful, but if not, we now have crashdumps as well.

Sifting through the log, I find the error about not finding libvdpau.so.1 to be interesting, as it placed a copy into Steam/ubuntu12_64/steam-runtime-sniper/var/tmp-92DII2/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/.
I'm guessing it copied it from Steam/ubuntu12_64/steam-runtime-sniper/sniper_platform_0.20240125.75305/files/lib/x86_64-linux-gnu/libvdpau.so.1.0.0 (note the extra ".0.0" on the end, I'm not sure if it's expecting an *.so.1 in that location or where else it's looking where it's not seeing it).

I saw it clean out the tmp dirs then give a errors that they're not empty. Most were empty by the time I looked. I did an rmdir * in the var under sniper and it removed most of them (7 remain of 19 that were there before). This could be purely a timing/sync issue with a file being removed server-side. A 1 second sleep should be more than enough to solve that. As issues go, that's incredibly minor.

Other than that, I don't see anything jumping out at me in that log. You'll know better than I what to look for though.

Sifting through the log, I find the error about not finding libvdpau.so.1 to be interesting, as it placed a copy into Steam/ubuntu12_64/steam-runtime-sniper/var/tmp-92DII2/usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu

Probably it found that your host system had a 64-bit libvdpau.so.1 but not a 32-bit libvdpau.so.1. Ideally you will want both of those on Nvidia-based systems, to get hardware acceleration for video encoding and decoding in both 64- and 32-bit processes. On Ubuntu this means installing both libvdpau1:amd64 and libvdpau1:i386. This is worth fixing, but probably not what is causing your crash, since the steamwebhelper is a 64-bit process that will be unaffected by the absence of a 32-bit library.

. I'm guessing it copied it from Steam/ubuntu12_64/steam-runtime-sniper/sniper_platform_0.20240125.75305/files/lib/x86_64-linux-gnu/libvdpau.so.1.0.0

No, anything that is created in usr/lib/pressure-vessel/overrides is a symbolic link pointing to graphics-stack-related files from your host system (via the /run/host mount point inside the container).

I saw it clean out the tmp dirs then give a errors that they're not empty.[...] This could be purely a timing/sync issue with a file being removed server-side. A 1 second sleep should be more than enough to solve that.

Sorry, I am not going to slow down each container startup for every Steam-on-Linux user just to benefit NFS users. If it's leaving behind nearly-empty directories, then the disk space cost is trivially small.

I suspect that what might be happening here is that we're deleting the directory tmp-XXXXXX while the file tmp-XXXXXX/.ref is open and locked. POSIX guarantees that we can delete files while they are still open, but NFS implements deletion of open files by renaming them to some weird name that will be removed when the file is eventually closed, and then that means the directory isn't empty, so rmdir() refuses to delete it.

Please could a Valve developer look up the backtraces for the two crash IDs referenced by @glabifrons in https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-1918241124

assert_20240130192454_27.dmp[1678661]: response: CrashID=bp-76d14b36-100b-48a3-aec4-615e12240130
assert_20240130192527_134.dmp[1679205]: response: CrashID=bp-722467e1-6953-4b8a-9ad0-01f382240130

and check whether they are the same thing that @davispuh is experiencing, which is this?

CrashID=bp-5aaf6174-a286-49b8-a3b3-eb2fe2240129

From the log in https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-1918241124, we are likely to be using the container's root CA certificates (derived from Debian 11's ca-certificates package, and available in /etc/ssl/) rather than the host's root CA certificates. This makes it odd that @glabifrons is getting an error message relating to root CA certificates, but I'm not: if there was a problem with the container's /etc/ssl, I would expect it to affect everyone, including me.

@glabifrons is using Ubuntu, which is Debian-derived, so this is not a mismatch between Debian and e.g. Fedora search paths for root CA certificates, or anything like that.

A potentially interesting factor is that we have pulled in /lib/x86_64-linux-gnu/libnvidia-pkcs11-openssl3.so.545.29.06 as part of the graphics stack, which has OpenSSL 3 /lib/x86_64-linux-gnu/libcrypto.so.3 as a dependency. This means that code inside the container can see all of these OpenSSL libraries:

• libssl.so.1.1 from sniper (actually from Debian 11, which sniper is based on)
• libcrypto.so.1.1 from sniper (actually from Debian 11)
• libcrypto.so.3 from Ubuntu 22.04
• whatever version of OpenSSL is statically linked into steamwebhelper, if any

so one possible factor for Valve developers to investigate would be whether these can somehow collide?

[Edited to add: the fact that https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-1924106225 didn't solve this, according to https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-1925158103, suggests that this probably wasn't the problem.]

@glabifrons, if you are comfortable with using unreleased software, one thing you could try is:

• confirm that the bug is still reproducible on your system
• rename ~/.steam/root/ubuntu12_64/steam-runtime-sniper/pressure-vessel out of the way
• replace it with the result of unpacking this: https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/jobs/439261/artifacts/raw/_build/pressure-vessel-bin.tar.gz (which is a test-build taken from steam-runtime-tools!663)
• try again

If that makes it work, then my theory about libnvidia-pkcs11-openssl3.so.* in https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-1919256799 is probably going in the right direction. If it still fails with that modified version of pressure-vessel, then my theory was probably wrong, but that could still be a useful data point for someone with more insight into the proprietary parts of Steam.

@smcv Thank you very much for coming up with more ways to narrow this down.
Unfortunately, I tried it and still got the crash dialog.

It looks like the change that causes this crash was just pushed to release, as my entire family is now getting this crash screen and they do not run the beta.
I just updated Steam and now I'm getting the crash in the released version too.

Crash dumps (2) from my kid's system:

assert_20240227194154_28.dmp[4071927]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240227194154_28.dmp
assert_20240227194154_28.dmp[4071927]: Finished uploading minidump (out-of-process): success = yes
assert_20240227194154_28.dmp[4071927]: response: CrashID=bp-1c93b6a7-b308-4df1-9465-5fdf92240227
assert_20240227194154_28.dmp[4071927]: file ''/tmp/dumps/assert_20240227194154_28.dmp'', upload yes: ''CrashID=bp-1c93b6a7-b308-4df1-9465-5fdf92240227''
BRefreshApplicationsInLibrary 1: 2ms
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
src/common/html/chrome_ipc_client.cpp (1111) : Failed to connect to master html process, created shared memory (spawn time 60.02)
src/common/html/chrome_ipc_client.cpp (1111) : Failed to connect to master html process, created shared memory (spawn time 60.02)
02/27 19:42:26 Init: Installing breakpad exception handler for appid(steam)/version(1708985249)/tid(4071705)
assert_20240227194226_38.dmp[4072137]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240227194226_38.dmp
assert_20240227194226_38.dmp[4072137]: Finished uploading minidump (out-of-process): success = yes
assert_20240227194226_38.dmp[4072137]: response: CrashID=bp-ce3828f7-3c00-4890-ae03-e08652240227
assert_20240227194226_38.dmp[4072137]: file ''/tmp/dumps/assert_20240227194226_38.dmp'', upload yes: ''CrashID=bp-ce3828f7-3c00-4890-ae03-e08652240227''
[2024-02-27 19:42:50] Shutdown

Crash dumps (3) from my wife's system:

assert_20240227192652_27.dmp[725933]: Finished uploading minidump (out-of-process): success = yes
assert_20240227192652_27.dmp[725933]: response: CrashID=bp-f3691d6b-88e6-4e81-a9e2-28b6e2240227
assert_20240227192652_27.dmp[725933]: file ''/tmp/dumps/assert_20240227192652_27.dmp'', upload yes: ''CrashID=bp-f3691d6b-88e6-4e81-a9e2-28b6e2240227''
BRefreshApplicationsInLibrary 1: 2ms
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
src/common/html/chrome_ipc_client.cpp (1111) : Failed to connect to master html process, created shared memory (spawn time 60.02)
src/common/html/chrome_ipc_client.cpp (1111) : Failed to connect to master html process, created shared memory (spawn time 60.02)
02/27 19:27:24 Init: Installing breakpad exception handler for appid(steam)/version(1708985249)/tid(725773)
assert_20240227192724_39.dmp[726038]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240227192724_39.dmp
assert_20240227192724_39.dmp[726038]: Finished uploading minidump (out-of-process): success = yes
assert_20240227192724_39.dmp[726038]: response: CrashID=bp-ac33e824-b4d4-4235-a03e-f6d682240227
assert_20240227192724_39.dmp[726038]: file ''/tmp/dumps/assert_20240227192724_39.dmp'', upload yes: ''CrashID=bp-ac33e824-b4d4-4235-a03e-f6d682240227''
src/clientdll/configstore.cpp (128) : ConfigStore (UserLocalConfigStore) is dirty, and being destroyed, we're discarding data
src/clientdll/configstore.cpp (128) : ConfigStore (UserLocalConfigStore) is dirty, and being destroyed, we're discarding data
assert_20240227192809_42.dmp[726105]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240227192809_42.dmp
assert_20240227192809_42.dmp[726105]: Finished uploading minidump (out-of-process): success = yes
assert_20240227192809_42.dmp[726105]: response: CrashID=bp-8b4472ce-df9b-4a5d-b93f-3a9842240227
assert_20240227192809_42.dmp[726105]: file ''/tmp/dumps/assert_20240227192809_42.dmp'', upload yes: ''CrashID=bp-8b4472ce-df9b-4a5d-b93f-3a9842240227''
[2024-02-27 19:28:10] Shutdown

I can generate more from mine too if it will help.

Looks like it activated on Fedora 39 also last night and steamwebhelper doesn't start anymore.

From steamwebhelper.log

steamwebhelper.sh[62847]: === Wed Feb 28 09:59:03 AM EET 2024 ===
steamwebhelper.sh[62847]: Starting steamwebhelper under bootstrap sniper steam runtime at /data/home/user/.local/share/Steam/ubuntu12_64/steam-runtime-sniper
exec ./steamwebhelper --no-sandbox -lang=en_US -cachedir=/data/home/user/.local/share/Steam/config/htmlcache -steampid=57800 -buildid=1708985249 -steamid=76561198176790325 -logdir=/data/home/user/.local/share/Steam/logs -uimode=7 -startcount=14 -steamuniverse=Public -realm=Global -clientui=/data/home/user/.local/share/Steam/clientui -steampath=/data/home/user/.local/share/Steam/ubuntu12_32/steam -launcher=0 -use_safe_shutdown_workaround -use_xcomposite_workaround -no-restart-on-ui-mode-change --enable-smooth-scrolling --disable-gpu-compositing --disable-gpu --password-store=basic --log-file=/data/home/user/.local/share/Steam/logs/cef_log.txt --disable-quick-menu --disable-features=DcheckIsFatal
[0228/095904.170782:ERROR:context.cc(100)] The browser_subprocess_path directory (./steamwebhelper) is not an absolute path. Defaulting to empty.
[0228/095904.186298:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 0
[0228/095904.186333:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1708962035
[0228/095904.186337:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0228/095904.186340:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0228/095904.186343:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0228/095904.186606:INFO:crash_reporting.cc(239)] Crash reporting enabled for process: browser
[0228/095904.187351:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0228/095904.188050:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0228/095904.204115:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 76561198176790325
[0228/095904.204179:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1708985249
[0228/095904.204183:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0228/095904.204147:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 76561198176790325
[0228/095904.204186:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0228/095904.204193:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0228/095904.204192:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1708985249
[0228/095904.204197:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0228/095904.204200:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0228/095904.204203:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux

The only ERROR I could find was

steamwebhelper.log:[0228/095904.170782:ERROR:context.cc(100)] The browser_subprocess_path directory (./steamwebhelper) is not an absolute path. Defaulting to empty.

webhelper.txt states

[1970-01-01 02:00:00] Client version: no bootstrapper found
[1970-01-01 02:00:00] Startup - webhelper launched with: ./steamwebhelper --no-sandbox -lang=en_US -cachedir=/data/home/user/.local/share/Steam/config/htmlcache -steampid=57800 -buildid=1708985249 -steamid=76561198176790325 -logdir=/data/home/user/.local/share/Steam/logs -uimode=7 -startcount=14 -steamuniverse=Public -realm=Global -clientui=/data/home/user/.local/share/Steam/clientui -steampath=/data/home/user/.local/share/Steam/ubuntu12_32/steam -launcher=0 -use_safe_shutdown_workaround -use_xcomposite_workaround -no-restart-on-ui-mode-change --enable-smooth-scrolling --disable-gpu-compositing --disable-gpu --password-store=basic --log-file=/data/home/user/.local/share/Steam/logs/cef_log.txt --disable-quick-menu --disable-features=DcheckIsFatal
[1970-01-01 02:00:00] Disabling GPU acceleration due to --disable-gpu-compositing (browser)
[1970-01-01 02:00:00] Disabling sandbox due to a previous crash in CefInitialize with the sandbox enabled
[1970-01-01 02:00:00] Browser - launching child process with: /data/home/user/.local/share/Steam/ubuntu12_64/steamwebhelper --type=zygote --no-zygote-sandbox --no-sandbox --user-agent-product=Valve Steam Client --lang=en_US.UTF-8 --log-file=/data/home/user/.local/share/Steam/logs/cef_log.txt --crashpad-handler-pid=62977 --buildid=1708985249 --steamid=76561198176790325
[1970-01-01 02:00:00] Browser - launching child process with: /data/home/user/.local/share/Steam/ubuntu12_64/steamwebhelper --type=zygote --no-sandbox --user-agent-product=Valve Steam Client --lang=en_US.UTF-8 --log-file=/data/home/user/.local/share/Steam/logs/cef_log.txt --crashpad-handler-pid=62977 --buildid=1708985249 --steamid=76561198176790325

The funny part seems to be the timestamp of log-entries? 1970 instead the system local time Wed Feb 28 10:05:33 AM EET 2024

Console output from startup below for reference in case it's useful

$ steam
steam.sh[68823]: Running Steam on fedora 39 64-bit
steam.sh[68823]: STEAM_RUNTIME is enabled automatically
setup.sh[68896]: Steam runtime environment up-to-date!
steam.sh[68823]: Steam client's requirements are satisfied
tid(68939) burning pthread_key_t == 0 so we never use it
[2024-02-28 10:35:31] Startup - updater built Feb 26 2024 15:40:54
[2024-02-28 10:35:31] Startup - Steam Client launched with: '/data/home/user/.local/share/Steam/ubuntu12_32/steam'
02/28 10:35:31 Init: Installing breakpad exception handler for appid(steam)/version(1708985249)/tid(68939)
[2024-02-28 10:35:31] Loading cached metrics from disk (/data/home/user/.local/share/Steam/package/steam_client_metrics.bin)
[2024-02-28 10:35:31] Using the following download hosts for Public, Realm steamglobal
[2024-02-28 10:35:31] 1. https://client-update.akamai.steamstatic.com, /, Realm 'steamglobal', weight was 1000, source = 'update_hosts_cached.vdf'
[2024-02-28 10:35:31] 2. https://cdn.cloudflare.steamstatic.com, /client/, Realm 'steamglobal', weight was 1, source = 'update_hosts_cached.vdf'
[2024-02-28 10:35:31] 3. https://cdn.steamstatic.com, /client/, Realm 'steamglobal', weight was 1, source = 'baked in'
[2024-02-28 10:35:31] Verifying installation...
[2024-02-28 10:35:31] Verification complete

Steam logging initialized: directory: /data/home/user/.local/share/Steam/logs

XRRGetOutputInfo Workaround: initialized with override: 0 real: 0xf03e9170
XRRGetCrtcInfo Workaround: initialized with override: 0 real: 0xf03e7880
steamwebhelper.sh[68952]: === Wed Feb 28 10:35:32 AM EET 2024 ===
steamwebhelper.sh[68952]: Starting steamwebhelper under bootstrap sniper steam runtime at /data/home/user/.local/share/Steam/ubuntu12_64/steam-runtime-sni
per
CAppInfoCacheReadFromDiskThread took 37 milliseconds to initialize
Steam Runtime Launch Service: starting steam-runtime-launcher-service
Steam Runtime Launch Service: steam-runtime-launcher-service is running pid 69112
bus_name=com.steampowered.PressureVessel.LaunchAlongsideSteam
steamwebhelper.sh[69455]: === Wed Feb 28 10:35:42 AM EET 2024 ===
steamwebhelper.sh[69455]: Starting steamwebhelper under bootstrap sniper steam runtime at /data/home/user/.local/share/Steam/ubuntu12_64/steam-runtime-sni
per
steamwebhelper.sh[69787]: === Wed Feb 28 10:35:52 AM EET 2024 ===
steamwebhelper.sh[69787]: Starting steamwebhelper under bootstrap sniper steam runtime at /data/home/user/.local/share/Steam/ubuntu12_64/steam-runtime-sni
per
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
02/28 10:35:53 Init: Installing breakpad exception handler for appid(steam)/version(1708985249)/tid(68939)
assert_20240228103553_26.dmp[69925]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240228103553_26.dmp
/usr/share/themes/Adwaita/gtk-2.0/main.rc:733: error: unexpected identifier 'direction', expected character '}'
/usr/share/themes/Adwaita/gtk-2.0/hacks.rc:28: error: invalid string constant "normal_entry", expected valid string constant
assert_20240228103553_26.dmp[69925]: Finished uploading minidump (out-of-process): success = yes
assert_20240228103553_26.dmp[69925]: response: CrashID=bp-f44023a9-c5be-4169-bcde-6f24c2240228
assert_20240228103553_26.dmp[69925]: file ''/tmp/dumps/assert_20240228103553_26.dmp'', upload yes: ''CrashID=bp-f44023a9-c5be-4169-bcde-6f24c2240228''
BRefreshApplicationsInLibrary 1: 1ms
steamwebhelper.sh[70150]: === Wed Feb 28 10:36:02 AM EET 2024 ===
steamwebhelper.sh[70150]: Starting steamwebhelper under bootstrap sniper steam runtime at /data/home/user/.local/share/Steam/ubuntu12_64/steam-runtime-sni
per
steamwebhelper.sh[70483]: === Wed Feb 28 10:36:12 AM EET 2024 ===
steamwebhelper.sh[70483]: Starting steamwebhelper under bootstrap sniper steam runtime at /data/home/user/.local/share/Steam/ubuntu12_64/steam-runtime-sni
per
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
src/steamUI/steamuisharedjscontroller.cpp (546) : Failed creating offscreen shared JS context
reaping pid: 69787 -- unknown
src/steamexe/main.cpp (264) : Assertion Failed: ReapProcess: waitid failed: 'No child processes'. Possibly leaking a zombie.

src/steamexe/main.cpp (264) : Assertion Failed: ReapProcess: waitid failed: 'No child processes'. Possibly leaking a zombie.

02/28 10:36:18 Init: Installing breakpad exception handler for appid(steam)/version(1708985249)/tid(68939)
assert_20240228103618_41.dmp[70821]: Uploading dump (out-of-process)
/tmp/dumps/assert_20240228103618_41.dmp
assert_20240228103618_41.dmp[70821]: Finished uploading minidump (out-of-process): success = yes
assert_20240228103618_41.dmp[70821]: response: CrashID=bp-bf7c1a81-5ec6-4e58-b63d-6f6bc2240228
assert_20240228103618_41.dmp[70821]: file ''/tmp/dumps/assert_20240228103618_41.dmp'', upload yes: ''CrashID=bp-bf7c1a81-5ec6-4e58-b63d-6f6bc2240228''
[2024-02-28 10:36:24] Shutdown

Dmesg also seems to show some errors related to steam

...
[ 9218.130122] traps: steamwebhelper[72547] trap int3 ip:7fd7c552f375 sp:7ffc7c2187f0 error:0 in libcef.so[7fd7bf336000+91a8000]
[ 9228.209527] traps: steamwebhelper[72937] trap int3 ip:7f0847f2f375 sp:7fff65725410 error:0 in libcef.so[7f0841d36000+91a8000]
[ 9238.381259] traps: steamwebhelper[73276] trap int3 ip:7ff54af2f375 sp:7ffe72f79c30 error:0 in libcef.so[7ff544d36000+91a8000]
[ 9248.740349] traps: steamwebhelper[73644] trap int3 ip:7f433fd2f375 sp:7ffdd7cb33c0 error:0 in libcef.so[7f4339b36000+91a8000]
[ 9248.740355] traps: steamwebhelper[73643] trap int3 ip:7f38b252f375 sp:7ffcc27ba750 error:0 in libcef.so[7f38ac336000+91a8000]
[ 9258.766632] traps: steamwebhelper[73977] trap int3 ip:7f276632f375 sp:7ffc006d1fd0 error:0
[ 9258.766633] traps: steamwebhelper[73976] trap int3 ip:7faa1872f375 sp:7ffe7d3b1e40 error:0 in libcef.so[7faa12536000+91a8000]
[ 9258.766636] in libcef.so[7f2760136000+91a8000]

...

Apparently if I just leave the error message window open, the steam appears to be running on the background without GUI and running second client from my laptop with AMD GPU seems to be able to stream games from my workstation with NVidia GPU i.e. it looks like the GUI is broken on NVidia hardware. My laptop is not fast enough to actually play anything, but steam seems to be responding to commands remotely and I seem to be able to launch games just fine.

Yep this is retarded, Steam now is completely broken without any workarounds even when bug was reported month ago in Beta and without fixing it got promoted to prod...

To me it looks like these are this same thing #10538 #10541 #10539

As of this morning i have the exact same after it updated steam (not beta).
Arch Linux, Intel RTX3090

Same problem here!

• Ubuntu 22.04.4 LTS
• GPU: Advanced Micro Devices, Inc. [Radeon RX 6700/6700 XT]
• Kernel driver in use: amdgpu

Are you all using Wayland? I wonder if this is Wayland specific bug...

Are you all using Wayland? I wonder if this is Wayland specific bug...

At least I'm running the old X11

It seems that the regular Steam now also uses the Sniper runtime and Steam has stopped working for me.

I can run the ./run-in-sniper vkcube

But it remains impossible for me to start Steam in Debian Testing using KDE with a Radeon graphics card.

The Flatpak edition also seems to start.

Edit: My laptop with Intel UHD 620 also fails to start with the same error after an update.

It seems that the regular Steam now also uses the Sniper runtime and Steam has stopped working for me.

I can run the ./run-in-sniper vkcube

But it remains impossible for me to start Steam in Debian Testing using KDE with a Radeon graphics card.

My laptop that uses and Intel UHD 620 still starts up the Steam client. The Flatpak edition also seems to start.

Same here, run-in-sniper works, but GUI fails. Laptop with AMD GPU has no issues.

./run-in-sniper vkcube
Selected GPU 0: NVIDIA GeForce RTX 4060 Ti, type: DiscreteGpu

I think I have the same issue

Today at 07:53am CET I contacted the steam support which approx. 60 minutes later pointed/directed me here, because direct help with the steam support team is only provided when problems occur on an Ubuntu LTS-Installation. Since I am working on a Solus 4.5 64-bit System with the GNOME 45 desktop my search for help brought me here.

I have the exact same problem, it seems that the steamwebhelper is stuck in an endless loop. I've tried to start the steam client with various relevant options/parameters, the problem remains.

I can exit the steam client through the tray icon, though.

Steam autoupdated itself at around 05:00pm CET yesterday, that's when the problem occured the first time.

Edit: Filesystem is ext4, AMD ATI Radeon R7 Graphics, Kernel driver in use: amdgpu, Distro: Solus 4.5 GNOME

@glabifrons - thank you for your detailed report. I assume the crash issue does not go away if you pass -cef-disable-gpu on the command line?

@TTimo Thank you for taking a look. Unfortunately, that doesn't seem to make a difference. Here are the crash IDs from launching with that command line option:
CrashID=bp-eef6aa82-944f-4e9c-a63a-986402240229
CrashID=bp-4466f2d5-4310-419d-9b29-d808f2240229

FWIW: I launched it before testing the command line option, saw the crash dialog, and waited while watching top. I saw a fair bit of activity so I left it run for a while. Eventually I saw it download something "Downloading update (67,470 of 67,470 KB)..." and watched it process that (python3 hit the top processes for a bit at this point IIRC).
Once it settled down and no more logs were being written, I selected the "Exit Steam" dialog on the crash screen and clicked "OK".
Upon restarting it again (without the command line option), I saw it unpack and install the update, but still went to the same crash screen (I was hoping).
I waited again for it to settle down, but it was much quicker this time.
This is when I tried your suggested command line option.

@glabifrons - it's probably a good idea to test latest without NFS in the loop. You may have a different issue with the new chromium setup but we'd need to make sure it's not something due to your NFS configuration first.

Yes, I actually managed this before and ended up reverting to a kind of hybrid config to be able to use Steam.
I reconfigured the links in the ~/.steam directory to point to an admittedly kludgey subdirectory under my cache drive, launched Steam to let it install, shut it down, then replaced the steamapps subdirectory with a link pointing back to ~/.local/share/Steam/steamapps (everything under my home directory will be over NFS) and was able to run Steam and launch games.
So yes, it appears to either be related to NFS, or something related to it, but only for the Steam UI, as I have no problem running games, even under Proton from NFS (I played Space Engineers a bit last night this way).

At one point (mentioned in the discussion above somewhere) I noted that I'm using NFSv4. I thought I'd point that out in case it was relevant, as NFSv4 and NFSv3 have different capabilities. That appears to have been ruled out though, as another person mentioned using NFSv4 successfully.
Another person with this problem turned out not to be running his home on NFS, but on an odd stack of VFS layers (I couldn't follow his description, but it involved btrfs, which I've never used).

Edit: While I'm configured for NFSv4, I just checked the output of mount(1) and it shows that NFSv4.1 has been negotiated. I don't know if that should make a difference.

To everyone else joining this issue thread: Please indicate (most important) what filesystem your home directory is using, which video card brand you're using, and which video card drivers you are using. Additionally, what distro you are using.

My home directory is mounted via NFSv4 (v4.1 according to the output of the mount command) from a Solaris server.
My video card is Nvidia and my drivers are "nvidia-driver-545" from Ubuntu's repository.
My distro is Ubuntu MATE LTS 22.04 (Ubuntu LTS 22.04 with the MATE desktop environment pre-installed).

There are several situations that can cause the steamwebhelper not to work: the "Steamwebhelper is not responding" menu is a symptom, not a cause. It might be better if we retitle this issue to be specific to @glabifrons' situation (it seems that having Steam installed on NFS rather than on a local filesystem is now understood to be a key part of the problem for them), and ask anyone else with similar symptoms to take them to a separate issue.

Anyone else who is experiencing the "Steamwebhelper is not responding" dialog, please check the Steam logs for more information (Flatpak users: ~/.var/app/com.valvesoftware.Steam/.local/share/Steam/logs, Snap users: ~/snap/steam/common/.local/share/Steam/logs, everyone else: ~/.steam/root/logs). steamwebhelper.log is likely to be the most important log file.

If your symptoms and logs match symptoms and logs seen by other users, you can subscribe to an existing bug report without adding comments by using the "Notifications" panel.

A brief summary of some common known issues that are not in-scope for this issue report:

One common issue at the moment is that it can cause problems if ~/.config, ~/.cache or ~/.local/share are symbolic links (known to be a problem for ~/.config, speculated to be a problem for the others). This is #10547 and #10552, and is out-of-scope for this specific issue thread. The workaround is to use bind-mounts instead of symbolic links. More details: https://github.com/ValveSoftware/steam-for-linux/issues/10547#issuecomment-1972901294. This can affect the steamwebhelper even if Proton 5.13+ and the various Steam Linux Runtime compatibility tools have worked correctly for you in the past.

Another common issue is that if bubblewrap doesn't work on your system, then neither will steamwebhelper, Proton 5.13+ or the various Steam Linux Runtime compatibility tools (but if Proton 5.13+ and Steam Linux Runtime have worked for you in the past, then this particular thing is not the problem). The required kernel functionality is similar to what's needed by Flatpak (see https://github.com/flatpak/flatpak/wiki/User-namespace-requirements#unprivileged-bubblewrap). On systems where bubblewrap needs to be setuid root (https://github.com/flatpak/flatpak/wiki/User-namespace-requirements#setuid-bubblewrap), in principle the container runtime framework is meant to work, but currently doesn't (https://github.com/ValveSoftware/steam-runtime/issues/650). Again, that's out-of-scope for this specific issue thread.

i have this symptom as well.

i am using nvidia, proprietary driver version 535.154.05. my home directory is an nfs mount, nfs v4.0. i am using gentoo linux, where i've used steam successfully for many years.

it was suggested to install the steam client locally, which i am able to do (i have space available), however it's not clear how to specify a custom installation directory. i've googled around with no satisfaction. i can see in bin_steam.sh that STEAM_DATA_HOME is set to $HOME/.local/share. i haven't tried modifying the script (yet) as it looks like $HOME is called out in several places.

i have temporarily created a local home directory instead of using the nfs mounted location. steam works now, so appears to be something related to having steam installed on an nfs mount.

is it possible to select where to install the steam client?

is it possible to select where to install the steam client?

Try symlinking ~/.local/share/Steam to /mnt/Steam, that's what I do (other reasons not because NFS) and it works fine.

Note you might need to delete ~/.steam folder so it's recreated.

Actually it might be that issue is ~/.config being on NFS and not due to Steam itself. So maybe try bind mounting that to local filesystem.

@davispuh

Actually it might be that issue is ~/.config being on NFS and not due to Steam itself. So maybe try bind mounting that to local filesystem.

It's not the .config directory. My ~/.config/ is on NFS still with my workaround described (above and) below, and it works.
The only thing that must be on a local filesystem appears to be Steam itself (not the steamapps subdirectory, it can be on NFS).
My ~/.steam/ directory was modified as shown:

$ ls -ln ~/.steam/
total 17
lrwxrwxrwx 1 1000 1000   7 Mar  1 20:50 bin -> ./bin32/
lrwxrwxrwx 1 1000 1000  36 Feb 29 23:53 bin32 -> /var/cache/fscache/Steam/ubuntu12_32/
lrwxrwxrwx 1 1000 1000  36 Feb 29 23:53 bin64 -> /var/cache/fscache/Steam/ubuntu12_64/
-rwxrwxr-x 1 1000 1000 659 Mar  1 20:49 registry.vdf*
lrwxrwxrwx 1 1000 1000  24 Feb 29 23:53 root -> /var/cache/fscache/Steam/
lrwxrwxrwx 1 1000 1000  32 Feb 29 23:53 sdk32 -> /var/cache/fscache/Steam/linux32/
lrwxrwxrwx 1 1000 1000  32 Feb 29 23:53 sdk64 -> /var/cache/fscache/Steam/linux64/
lrwxrwxrwx 1 1000 1000  24 Feb 29 23:53 steam -> /var/cache/fscache/Steam/
-rw-rw-r-- 1 1000 1000   8 Feb 29 23:53 steam.pid
prw------- 1 1000 1000   0 Jan 27 13:05 steam.pipe|
-r-------- 1 1000 1000  16 Feb 29 23:53 steam.token

I created the directory /var/cache/fscache/Steam/ and changed the owner and group to my user (fscache is the mount point for my NFS cache, but this does not interfere with it).
I then launched steam and let it install into the new directory.
After it finished, I shut it down, then renamed the new steamapps directory and created a link to replace it that pointed to the existing games in my NFS mounted home:

$ ls -ln /var/cache/fscache/Steam/steamapps
lrwxrwxrwx 1 1000 1000 42 Feb 27 19:20 /var/cache/fscache/Steam/steamapps -> /home/glabifrons/.local/share/Steam/steamapps/

So Steam seems perfectly fine with the steamapps subdirectory being replaced with a link as a workaround (if, like me, you want to keep your games NFS mounted).
I realize there is a configuration change you can make somewhere in Steam to point to a different directory for game installation, but I don't want to reinstall a couple terabytes of games (between me and my family, my kid has over 1TB alone).

thanks @glabifrons. something in your description is missing for me (maybe?). i get that you created a new directory (/var/cache/...) and changed the permissions. how did you install the steam client to that directory?

my client machines have <nfs_server>:/home mounted. so i can access my home directory from any client machine and the server via the same path. did you link ~/.local/share/Steam to /var/cache/fscache/Steam and then install?

i did the above and it's working for me now. linked ~/.local/share/Steam to a directory on a local drive and then launched the client. it did the install and launches ok now. thanks for the help @glabifrons

@rustysack

thanks @glabifrons. something in your description is missing for me (maybe?). i get that you created a new directory (/var/cache/...) and changed the permissions. how did you install the steam client to that directory?

If the ~/.steam/ directory has those links pointing to that new directory, launching Steam will trigger it to install to that new directory.
If your root drive has enough space, you could create (for example) /opt/Steam/ then recreate those links in the ~/.steam/ directory to point to /opt/Steam instead. Once you launch Steam, it'll install there (just make sure your user owns the directory first).

my client machines have <nfs_server>:/home mounted. so i can access my home directory from any client machine and the server via the same path. did you link ~/.local/share/Steam to /var/cache/fscache/Steam and then install?

No, I left ~/.local/share/Steam/ as-is, no changes are needed there. It's the links in ~/.steam/ that tell Steam where it's installed.
Important: I'm using the Steam installer from steampowered.com, not the Ubuntu customized one. It has a different directory structure and I can't for the life of me figure out why they'd do that.

i did the above and it's working for me now. linked ~/.local/share/Steam to a directory on a local drive and then launched the client. it did the install and launches ok now. thanks for the help @glabifrons

That'll work too, but it ends up moving all of your games to the local drive too. The way I did it kept the games on NFS and only moved the Steam installation.

i did the above and it's working for me now. linked ~/.local/share/Steam to a directory on a local drive and then launched the client. it did the install and launches ok now. thanks for the help @glabifrons

That'll work too, but it ends up moving all of your games to the local drive too. The way I did it kept the games on NFS and only moved the Steam installation.

in my case, i have storage settings in steam that put games on a local disk any, in a specific location. i want steam client on the nfs mount so i can access my steam account from any computer. can't work that way like this though. hope valve fixes this properly.

I'm experiencing the very same issue on a system where users' home directories are stored on NFS. However, this mount is on a pretty fast network, so I doubt it's because of excessive latency.

@smcv Is there any workaround to this particular issue at the moment?

This might be a long-shot, but it's worth asking (note: I admit I know next to nothing about .Net).
Is there any part of this that runs under .Net/mono? I know I already mentioned seeing a windows (PE32+) executable running during one of my test runs, so it's plausible (although I do understand that .Net != PE32+).
The reason I ask is I stumbled over something that would prevent a .Net application from running from NFS, a field called loadFromRemoteSources.
I stumbled across it from the last comment in a vaguely similar issue in a different project:
https://github.com/Linux74656/SpaceEngineersLinuxPatches/issues/33#issuecomment-939283334

I'm experiencing the very same issue on a system where users' home directories are stored on NFS. However, this mount is on a pretty fast network, so I doubt it's because of excessive latency.

You say "pretty fast", but latency and throughput aren't the same thing. Some of the setup that the container runtime framework needs to do involves a lot of metadata operations (mkdir(), link(), symlink(), chmod()), some of which are documented to happen atomically with respect to other metadata operations. Each metadata operation is very small, but the filesystem needs to wait for the reply from the server before it can proceed to the next, which means the fastest it can go is (round trip time) × (number of operations), and when there are thousands of metadata operations to be done, the time adds up.

If possible, I would recommend putting the actual Steam installation on a local Unix filesystem like ext4, xfs or btrfs (not a remote filesystem like NFS, and not a non-Unix filesystem like NTFS either). You can still install games on a remote filesystem, if you want, by adding another Steam library (I use ~/SteamLibrary on my test machines) - but it will be most reliable if all of the Steam Linux Runtime and Proton compatibility tools are also installed in a Steam library that lives on a local Unix filesystem.

It is possible to relocate the Steam installation to wherever you want, within reason, as long as you adjust both of the symbolic links ~/.steam/steam and ~/.steam/root so they point to its real path. A few locations like /usr will not work for technical reasons, but anywhere below /home, /srv, /mnt, /opt, /media or /run/media should be OK. If you are moving it to a different filesystem, make sure that you don't lose file permissions: executable files must stay executable.

For ordinary OS-level packages (.deb or similar) the default location for the actual Steam installation is ~/.local/share/Steam, or sometimes ~/.steam/debian-installation.

For Flatpak users it's usually ~/.var/app/com.valvesoftware.Steam/.local/share/Steam, and the symbolic links that need to be adjusted are ~/.var/app/com.valvesoftware.Steam/.steam/{steam,root}.

For Snap users it's usually ~/snap/steam/common/.local/share/Steam, and the symbolic links are ~/snap/steam/common/.steam/{steam,root}.

Is there any workaround to this particular issue at the moment?

I can't tell what specific issue you are having (the "Steamwebhelper is not responding" dialog is a symptom, not a root cause), and you haven't attached any logs that I've seen. steamwebhelper.log is the most important. Until I see a log, I can't tell what point it's failing at - one possibility is that the Steam Linux Runtime container framework is failing to start, and another possibility is that the container is starting up OK but then the steamwebhelper is crashing.

If whatever issue you are having is triggered by having Steam installed on NFS, the workaround would be to try having Steam not be installed on NFS, as above.

This might be a long-shot, but it's worth asking (note: I admit I know next to nothing about .Net). Is there any part of this that runs under .Net/mono?

Not as far as I know. To the best of my knowledge, steamwebhelper is a native x86_64 Linux executable written in mostly C++, with functionality derived from the Chromium codebase via CEF.

Quite a lot of games use .Net (notably, the popular Unity engine is .Net-based), so if you want to play Steam games, any incompatibility between NFS and .Net is likely to be something you will need to solve or work around sooner or later; but that shouldn't matter when you are still at the stage of trying to run Steam itself.

I'm a fedora user.

Try to execute steam from a terminal.
This workaround worked on a Fedora 39.

@gnroses:

I'm a fedora user.

Try to execute steam from a terminal. This workaround worked on a Fedora 39.

If there is a root cause for steamwebhelper crashes that involves NFS, then I would be surprised by this workaround having any effect.

I suspect you might be getting the same symptom, but for a different reason - something that involves the terminal. Please report a separate issue, with logs from the failing situation (steamwebhelper.log is usually the most important log) - and ideally also a log from the working situation where you used your workaround, so we can compare.

I think the script calling a steamwebhelper, point to a location in a variable null or an invalid folder and get a error with absolute path.
But, calling steam on a terminal, get the right path and run flawlessly.

steam.sh[7537]: Running Steam on fedora 39 64-bit
steam.sh[7537]: STEAM_RUNTIME is enabled automatically
setup.sh[7610]: Steam runtime environment up-to-date!
steam.sh[7537]: Steam client's requirements are satisfied
tid(7666) burning pthread_key_t == 0 so we never use it
[2024-03-29 14:55:47] Startup - updater built Mar  6 2024 20:27:25
[2024-03-29 14:55:47] Startup - Steam Client launched with: '/home/wesley/.local/share/Steam/ubuntu12_32/steam'
minidumps folder is set to /tmp/dumps
03/29 14:55:47 Init: Installing breakpad exception handler for appid(steam)/version(1709846872)/tid(7666)
[2024-03-29 14:55:47] Loading cached metrics from disk (/home/wesley/.local/share/Steam/package/steam_client_metrics.bin)
[2024-03-29 14:55:47] Using the following download hosts for Public, Realm steamglobal
[2024-03-29 14:55:47] 1. https://client-update.akamai.steamstatic.com, /, Realm 'steamglobal', weight was 1000, source = 'update_hosts_cached.vdf'
[2024-03-29 14:55:47] 2. https://cdn.cloudflare.steamstatic.com, /client/, Realm 'steamglobal', weight was 1, source = 'update_hosts_cached.vdf'
[2024-03-29 14:55:47] 3. https://cdn.steamstatic.com, /client/, Realm 'steamglobal', weight was 1, source = 'baked in'
[2024-03-29 14:55:47] Verifying installation...
[2024-03-29 14:55:47] Verification complete
UpdateUI: skip show logo
Steam logging initialized: directory: /home/wesley/.local/share/Steam/logs

^[[23~XRRGetOutputInfo Workaround: initialized with override: 0 real: 0xeb1cc170
XRRGetCrtcInfo Workaround: initialized with override: 0 real: 0xeb1ca880
steamwebhelper.sh[7709]: === sex 29 mar 2024 14:55:48 -03 ===
steamwebhelper.sh[7709]: Starting steamwebhelper under bootstrap sniper steam runtime at /home/wesley/.local/share/Steam/ubuntu12_64/steam-runtime-sniper
CAppInfoCacheReadFromDiskThread took 16 milliseconds to initialize
Steam Runtime Launch Service: starting steam-runtime-launcher-service
Steam Runtime Launch Service: steam-runtime-launcher-service is running pid 7793
bus_name=com.steampowered.PressureVessel.LaunchAlongsideSteam
/usr/share/themes/Adwaita/gtk-2.0/main.rc:733: error: unexpected identifier 'direction', expected character '}'
/usr/share/themes/Adwaita/gtk-2.0/hacks.rc:28: error: invalid string constant "normal_entry", expected valid string constant
/usr/share/themes/Breeze/gtk-2.0/widgets/entry:70: error: unexpected identifier 'direction', expected character '}'
/usr/share/themes/Breeze/gtk-2.0/widgets/styles:36: error: invalid string constant "combobox_entry", expected valid string constant
BRefreshApplicationsInLibrary 1: 1ms
BuildCompleteAppOverviewChange: 172 apps
RegisterForAppOverview 1: 5ms
RegisterForAppOverview 2: 5ms
Fossilize INFO: Setting autogroup scheduling.
Fossilize INFO: Setting autogroup scheduling.
wine: using kernel write watches, use_kernel_writewatch 1.
fsync: up and running.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
/bin/sh\0-c\0/home/wesley/.local/share/Steam/ubuntu12_32/reaper SteamLaunch AppId=1086940 -- /home/wesley/.local/share/Steam/ubuntu12_32/steam-launch-wrapper -- '/home/wesley/.local/share/Steam/steamapps/common/SteamLinuxRuntime_sniper'/_v2-entry-point --verb=waitforexitandrun -- '/home/wesley/.local/share/Steam/steamapps/common/Proton - Experimental'/proton waitforexitandrun  '/media/Baldurs/steamapps/common/Baldurs Gate 3/Launcher/LariLauncher.exe'\0
chdir "/media/Baldurs/steamapps/common/Baldurs Gate 3/bin"
ERROR: ld.so: object '/home/wesley/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/wesley/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
ERROR: ld.so: object '/home/wesley/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/wesley/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
ERROR: ld.so: object '/home/wesley/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
wine: using kernel write watches, use_kernel_writewatch 1.
fsync: up and running.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
Setting breakpad minidump AppID = 1086940
Steam_SetMinidumpSteamID:  Caching Steam ID:  76561198294142202 [API loaded no]
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
Fossilize INFO: Overriding serialization path: "/media/Baldurs/steamapps/shadercache/1086940/fozpipelinesv6/steamapprun_pipeline_cache".
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
Fossilize INFO: Overriding serialization path: "/media/Baldurs/steamapps/shadercache/1086940/fozpipelinesv6/steamapprun_pipeline_cache".
GameOverlay: started '/home/wesley/.local/share/Steam/ubuntu12_32/gameoverlayui' (pid 9563) for game process 9506
tid(9563) burning pthread_key_t == 0 so we never use it
minidumps folder is set to /tmp/dumps
03/29 14:57:07 Init: Installing breakpad exception handler for appid(gameoverlayui)/version(20240306202747)/tid(9563)
03/29 14:57:07 Init: Installing breakpad exception handler for appid(gameoverlayui)/version(1.0)/tid(9563)
[2024-03-29 14:57:49] Background update loop checking for update. . .
[2024-03-29 14:57:49] Checking for available updates...
[2024-03-29 14:57:49] Downloading manifest: https://client-update.akamai.steamstatic.com/steam_client_ubuntu12?t=3108872834
[2024-03-29 14:57:49] Manifest download: send request
[2024-03-29 14:57:49] Manifest download: waiting for download to finish
[2024-03-29 14:57:50] Manifest download: finished
[2024-03-29 14:57:50] Download skipped: /steam_client_ubuntu12?t=3108872834 version 1709846872, installed version 1709846872, existing pending version 0
[2024-03-29 14:57:50] Nothing to do
^[[Awine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.
wine: using kernel write watches, use_kernel_writewatch 1.

(process:9563): GLib-GObject-CRITICAL **: 17:40:51.789: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
GameOverlay: started '/home/wesley/.local/share/Steam/ubuntu12_32/gameoverlayui' (pid 18386) for game process 18239
tid(18386) burning pthread_key_t == 0 so we never use it
minidumps folder is set to /tmp/dumps
03/29 17:40:51 Init: Installing breakpad exception handler for appid(gameoverlayui)/version(20240306202747)/tid(18386)
03/29 17:40:51 Init: Installing breakpad exception handler for appid(gameoverlayui)/version(1.0)/tid(18386)
pid 9229 != 9228, skipping destruction (fork without exec?)
Game 1086940 created interface STEAMAPPLIST_INTERFACE_VERSION001 / AppList
Game 1086940 created interface STEAMAPPS_INTERFACE_VERSION008 / Apps
Game 1086940 created interface STEAMHTMLSURFACE_INTERFACE_VERSION_005 / HTMLSurface
Game 1086940 created interface STEAMHTTP_INTERFACE_VERSION003 / HTTP
Game 1086940 created interface STEAMINVENTORY_INTERFACE_V003 / Inventory
Game 1086940 created interface STEAMMUSICREMOTE_INTERFACE_VERSION001 / MusicRemote
Game 1086940 created interface STEAMMUSIC_INTERFACE_VERSION001 / Music
Game 1086940 created interface STEAMPARENTALSETTINGS_INTERFACE_VERSION001 / ParentalSettings
Game 1086940 created interface STEAMREMOTEPLAY_INTERFACE_VERSION001 / RemotePlay
Game 1086940 created interface STEAMREMOTESTORAGE_INTERFACE_VERSION016 / RemoteStorage
Game 1086940 created interface STEAMSCREENSHOTS_INTERFACE_VERSION003 / Screenshots
Game 1086940 created interface STEAMUGC_INTERFACE_VERSION016 / UGC
Game 1086940 created interface STEAMUSERSTATS_INTERFACE_VERSION012 / UserStats
Game 1086940 created interface STEAMVIDEO_INTERFACE_V002 / Video
Game 1086940 created interface SteamController008 / 
Game 1086940 created interface SteamFriends017 / Friends
Game 1086940 created interface SteamInput006 / 
Game 1086940 created interface SteamInput006 / Controller
Game 1086940 created interface SteamMatchGameSearch001 / GameSearch
Game 1086940 created interface SteamMatchMaking009 / Matchmaking
Game 1086940 created interface SteamMatchMakingServers002 / MatchmakingServers
Game 1086940 created interface SteamNetworking006 / Networking
Game 1086940 created interface SteamNetworkingMessages002 / 
Game 1086940 created interface SteamNetworkingSockets012 / 
Game 1086940 created interface SteamNetworkingUtils004 / 
Game 1086940 created interface SteamParties002 / Parties
Game 1086940 created interface SteamUser021 / User
Game 1086940 created interface SteamUtils010 / 
Game 1086940 created interface SteamUtils010 / Utils
Game 1086940 method call count for IClientNetworkingUtilsSerialized::GotLocationString : 1
Game 1086940 method call count for IClientNetworkingSocketsSerialized::GetSTUNServer : 2
Game 1086940 method call count for IClientNetworkingSocketsSerialized::GetCachedRelayTicketCount : 1
Game 1086940 method call count for IClientNetworkingSocketsSerialized::GetCertAsync : 1
Game 1086940 method call count for IClientHTTP::ReleaseHTTPRequest : 2
Game 1086940 method call count for IClientHTTP::GetHTTPResponseBodyData : 2
Game 1086940 method call count for IClientHTTP::SendHTTPRequest : 2
Game 1086940 method call count for IClientHTTP::SetHTTPRequestHeaderValue : 1
Game 1086940 method call count for IClientHTTP::CreateHTTPRequest : 2
Game 1086940 method call count for IClientUtils::RecordSteamInterfaceCreation : 32
Game 1086940 method call count for IClientUtils::GetAPICallResult : 6
Game 1086940 method call count for IClientUtils::GetAppID : 35
Game 1086940 method call count for IClientUtils::GetIPCountry : 1
Game 1086940 method call count for IClientFriends::SetRichPresence : 2
Game 1086940 method call count for IClientUser::GetAuthSessionTicket : 1
Game 1086940 method call count for IClientUser::GetSteamID : 2
Game 1086940 method call count for IClientUser::BLoggedOn : 2
Uploaded AppInterfaceStats to Steam

(process:18386): GLib-GObject-CRITICAL **: 17:40:54.008: g_object_unref: assertion 'G_IS_OBJECT (object)' failed
reaping pid: 9563 -- gameoverlayui
reaping pid: 18386 -- gameoverlayui
Fossilize INFO: Setting autogroup scheduling.
Fossilize INFO: Setting autogroup scheduling.
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
WARNING: discarding _NET_WM_PID 2 as invalid for X11 window - use specialized XCB_X11_TO_PID function!
BRefreshApplicationsInLibrary 1: 1ms
BuildCompleteAppOverviewChange: 179 apps
BRefreshApplicationsInLibrary 1: 0ms
BuildCompleteAppOverviewChange: 179 apps
[2024-03-29 20:29:21] Shutdown

steamwebhelper.sh[51584]: === sáb 30 mar 2024 01:35:59 -03 ===
steamwebhelper.sh[51584]: Starting steamwebhelper under bootstrap sniper steam runtime at /home/wesley/.local/share/Steam/ubuntu12_64/steam-runtime-sniper
exec ./steamwebhelper --no-sandbox -lang=en_US -cachedir=/home/wesley/.local/share/Steam/config/htmlcache -steampid=51543 -buildid=1709846872 -steamid=0 -logdir=/home/wesley/.local/share/Steam/logs -uimode=7 -startcount=0 -steamuniverse=Public -realm=Global -clientui=/home/wesley/.local/share/Steam/clientui -steampath=/home/wesley/.local/share/Steam/ubuntu12_32/steam -launcher=0 -no-restart-on-ui-mode-change --enable-smooth-scrolling --password-store=basic --log-file=/home/wesley/.local/share/Steam/logs/cef_log.txt --disable-quick-menu --disable-features=DcheckIsFatal
[0330/013601.602913:ERROR:context.cc(100)] The browser_subprocess_path directory (./steamwebhelper) is not an absolute path. Defaulting to empty.
[0330/013601.639338:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 0
[0330/013601.639566:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1709756822
[0330/013601.639568:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0330/013601.639570:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0330/013601.639571:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0330/013601.642611:INFO:crash_reporting.cc(239)] Crash reporting enabled for process: browser
[0330/013601.650472:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0330/013601.652341:WARNING:task_impl.cc(32)] No task runner for threadId 0
[0330/013601.666691:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 0
[0330/013601.666740:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1709846872
[0330/013601.666742:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0330/013601.666744:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0330/013601.666746:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0330/013601.667517:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 0
[0330/013601.667564:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1709846872
[0330/013601.667566:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0330/013601.667568:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0330/013601.667569:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
Gtk-Message: 01:36:02.401: Failed to load module "window-decorations-gtk-module"
Gtk-Message: 01:36:02.401: Failed to load module "colorreload-gtk-module"
[0330/013602.441360:INFO:crash_reporting.cc(262)] Crash reporting enabled for process: gpu-process
[0330/013602.502752:WARNING:sandbox_linux.cc(385)] InitializeSandbox() called with multiple threads in process gpu-process.
[0330/013602.676023:WARNING:crash_reporting.cc(278)] Failed to set crash key: UserID with value: 0
[0330/013602.676077:WARNING:crash_reporting.cc(278)] Failed to set crash key: BuildID with value: 1709846872
[0330/013602.676079:WARNING:crash_reporting.cc(278)] Failed to set crash key: SteamUniverse with value: Public
[0330/013602.676081:WARNING:crash_reporting.cc(278)] Failed to set crash key: Vendor with value: Valve
[0330/013602.676083:WARNING:crash_reporting.cc(278)] Failed to set crash key: Platform with value: Linux
[0330/013602.676975:INFO:crash_reporting.cc(239)] Crash reporting enabled for process: utility
[0330/013602.691324:INFO:crash_reporting.cc(262)] Crash reporting enabled for process: utility
[0330/013602.951492:INFO:crash_reporting.cc(262)] Crash reporting enabled for process: renderer
Invalid browser dimensions: 0 x 0
src/webhelper/html_chrome.cpp (3435) : CefCurrentlyOn( TID_UI )
[0330/013606.079377:WARNING:connection.cc(41)] X error received.  Request: DestroyWindowRequest, Error: WindowError{.sequence = 465, .bad_value = 74448903, .minor_opcode = 0, .major_opcode = 4}
failed to create drawable

I just posted before looking to help fellows in the same situation.

Valve what are you doing with the steam client and that bloaded sandboxing stuff? I'm suffering from "Steamwebhelper is not responding" too. I read that symlinks can be a problem with .config, but thats not the case here. What can i do? Also -vgui cef/gpu switches won't help.

Is there any workaround to this particular issue at the moment?

I can't tell what specific issue you are having (the "Steamwebhelper is not responding" dialog is a symptom, not a root cause), and you haven't attached any logs that I've seen. steamwebhelper.log is the most important. Until I see a log, I can't tell what point it's failing at - one possibility is that the Steam Linux Runtime container framework is failing to start, and another possibility is that the container is starting up OK but then the steamwebhelper is crashing.

If whatever issue you are having is triggered by having Steam installed on NFS, the workaround would be to try having Steam not be installed on NFS, as above.

I'm trying to get steam working on a multi-user system where the home directories are mounted through NFS, and moving the installation outside of the home directory is not really an option.

Before the January update it used to work fine, even on the NFS mount; has the root cause of the issue with $HOME being on NFS been identified at all?

I attach below the steamwebhelper.log file from the one of the crashes in question, as well as the CLI output:

cli.log

steamwebhelper.log

Replying to https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-1915301643

I'm using flatpak Steam in fedora 39. I got the same issue 2 month ago. I still can't find a fix after 2 updates. Anyone have any ideas for this error?

Use vmtouch, I run a few diskless clients, the race condition can't seem to be bypassed by any tmpfs or even using rdma or any other fancy things I've tried. For anyone on gentoo just emerge dev-util/vmtouch
cd into the ubuntu12_64 folder, then do vmtouch -L ./
it uses about 1G of ram to lock it all, then you can start steam the normal way you launch it.
As soon as it signs in and you get your standard library or store page you can control-C in the terminal to kill the vmtouch process and let the kernel go back to doing it's awesome job. Hope that helps at least a few others in this thread.

I signed up to github just to post this, Stay strong fellow Diskless Gamers!

I don't feel this is correct place. However.

Steam client. Tue May 7 01:02:46 PM EDT 2024 (steamwebhelper.sh)
Ubuntu 22.04 LTS
Not beta client.
System is up to date. Steam was running early this week, not updates since then.
GPU AMD RTX6600

After latest update, steam went into update loop. Removed steam and reinstalled and same problem. By manually running steam-runtime-sniper.sh manual and waiting for it to update. I managed to get it to install. However it crashes when I run steam command. Please see steam ticket HT-DTJG-PKXJ-8T4G for install log and crash file from steamwebclient. If you need other logs I can reinstall and proved them. I have looked through the logs and can find nothing that looks wrong. Note I am running my home directories over NFS. 1GB NIC, Intel Celeron N3160 (4GB). Primary desktop Intel I7-9700K @ 3.60Ghz, 32GB memory. If you need more information let me know.

I did some testing. I installed steam on local account, so it is something with NFS. Not sure why this would be happening.

More information. I was able to get steam to load and play via a secondary user. However while I can get graphics to display, I can only get sound on either primary user or steam user at one time. I also created symbolic links between .steam and .local/share/Steam to directories on my local disk. I was able to install steam on there. However I had to create a link from .local/share/steamapp to .local/share/Steam/steamapp. I could now get steam client to run. However I got errors with trying to run a game. Also before games where installed into .local/share before they where now uninstalled in .local/share/Steam/steamapp. It is unclear why steam would care where the files where. Symbolic links should be transparent. I also created a second storage area on my NFS directory, but steam would not install to it.

i have what seems to be the same issue on the zfs filesystem as of about 3 days ago (6.6.30-gentoo)
it persists across 1 reinstall and 1 distro hop

Time for an information dump to see if any of this can help narrow down the issue.

Server side, ZFS Mirrors hosting NFS server
Clients use NFS4.1 for rootonnfs, after booting they run through some scripting and...

1. Thin clients mount their internal drives at /home (steam does work fine there)
2. Diskless clients mount their own unique home folders from the server (Been using RDMA for a few years via Roce)
3. Basic deduction from 1+2=Same OS so no possible differences in libs and other programs

Steam has been breaking for years on the diskless clients, have been thru all the normal "fixes/hacks" to get it working over time like the fakeflock.c workaround. But when it works, performance is pretty epic, between ZFS arc caching and RDMA removing loads of latency, anything that would be effected by small latency spikes you would assume simply launching it again would bypass the issue as it should already be cached on the clients ram, or worst case cached in the ARC on the server.

TMPFS for the steam-runtime-sniper/var used to work quite well, something has changed since JAN this year that that's no longer the case.

vmtouch on the whole of ubuntu12_64 worked for a few weeks, once in awhile the first start on booting would have the crash, then it would be fine restarting steam.

Currently there's a very very small chance of steam starting correctly, if I let it run instead of closing the webhelper crash dialog I get steam in a partially usable state, I can play some games, but no friends or store page or anything like that. But if it does start correctly everything is how it should be, and I never close it, instead I have the machines go into suspend mode instead of just shutting down, and they resume from suspend just fine with steam still running perfectly. vmtouch does seem like it's increasing the odds of steam starting correctly still, but something has changed in the updates to steam since I first posted about it.

I'm up for anything regarding making changes to configs to anything that you'd like to try out, so feel free to throw ideas out, I've always been a huge fan of steam and I've been using Gentoo specifically for around 15 years.

Played some 7 Days to Die online with friends last night, while steamwebhelper was crashed because I got tired of restarting it. Here's the steamwebhelper.log

steamwebhelper.log

client mount info for the share used by diskless steam clients (each has their own home)
serveraddr:/home on /home type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,soft,softreval,nocto,proto=rdma,port=20049,timeo=50,retrans=2,sec=sys,clientaddr=x.x.x.x,local_lock=none,addr=x.x.x.x)

Not sure if it helps, but used vmtouch on ubuntu12_64, after 3 attempts steam started correctly, so I thought I'd share the log of what it looks likes when things run like normal.

steamwebhelper.log

I'm sensing a pattern here. Steam was working fine under NFS for over a year until it decided to update last week. I am wondering if they are trying to implement some sort of anti-piracy system or messed up something. Creating symbolic links between ~/.steam and ~/.local/share/Steam to local disk based directories should work just fine. However it requires adding a link ~/.local/share/steamapp to ~/.local/share/Steam/steamapp. This will allow the steam client to run, however when you attempt to run any games it gets errors trying to open some shared libraries. One issue with NFS is that you can only access it from the userid you are using. If you are a different userid you can't write or modify any files on the NFS share. Perhaps the latest Steam client is doing something with a setuid executable as root. This would not explain why symlinking to a local disk would break. Also when doing the symlink the games are installed into ~/.local/share/Steam/steamapps rather then ~/.local/share.

If Value is interested I am a currently available as Linux developer. Been developing on Linux since very beginning. My direct email is in my profile. If you are interested, drop me a private email and I will send you my resume.

would be nice if valve just let us rollback steam a week

@rcornwell
You can try using all_squash in /etc/exports
I use that on a different mount for sharing mods or whatever needs shared

I can try, however steam should NOT be doing anything as root! Also I am the only user on this NFS share. I keep my home directory on a NAS so I can use it on any of my systems.

I created a directory on local disk, /mnt/username installed steam it worked correctly. However when I run 7 Days to Die I get "Anti Cheat Launcher Error". I tried symlinking ~/.steam to /mnt/username/.steam and ~/.local/share/Steam to /mnt/username. I set $HOME to /mnt/username, I also tried updating /etc/passwd for my account to point to /mnt/username. I know I probably will not get an answer, but what do I need to add to get the game to run. If you would like to send me (or post here) what files I need to edit to get my game to play. I have legitimately purchased the game and have about 400 hours into the game. It does not make sense that I can't have steam installed in a directory other then my home directory. Since I have to log into Steam to run game.

Personally after installing, I verify it's NOT launching with proton as it's a linux native game thanks to unity. Then I use the launcher that pops up on first run to switch to using Vulkan for graphics, even though it says experimental, it's been working better then opengl since before amdgpu came out for me. If both of those are true, my guess would be that having steam outside of home is messing with Pressure-Vessel.

Pressure-Vessel is a modified flatpak that tries to load whatever libraries are newer between the various steam-runtime versions and your actual OS files. I'm guessing it's having issues with the symlinking not always being followed like they should.

I would try mounting to ~/.steam, it's not a good solution for multiple users, but you can make the steam library outside of that mount just fine.

if you still have issues after trying that, run steam from the terminal, then after the game crashes post whatever you think is relevant.

And lastly a big thank you @rcornwell since you also play 7 days to die and posted your hours, I peeked at mine and fate smiled at me.

I got steam running several games with local directory, however I can't seem to get the anti-cheap to run correctly. I am the only user on my machine. I do have test user account, however I can only play sound on one user at a time. Note I am trying to disable AntiCheat software, just make it run so I can run the game.

Anticheat's all work fine over here, you can disable loading it in 7dtd with the start popup. Go in Properties on the game in the steam library side panel, and under launch options on the right, pick show launcher. I don't run any client sided mods, so maybe that makes a difference.

This issue is about steamwebhelper specifically. If you are having problems with any game (7 Days to Die or otherwise) then that is not this issue: if Steam gets far enough into launching that you are able to start games, then the steamwebhelper must be working correctly.

It still is an issue. I got steam to work by messing around. I now have steam installed on a local disk with symbolic links in my home directory from .steam and .local/share/Steam to the local drive. I am still unable to install steam on my NFS mounted home directory like I have been able to do for over a year prior. These are work arounds not fixes. As a Unix/Linux developer who has been using Linux since basically day one, and Unix a long time before. I see no reason I can't install steam on a NFS filesystem, I also see no reason things should break when I install it on a local disk. Something in the update that was pushed out at beginning of May caused things to break.

@smcv steamwebhelper crashes don't lead to steam not loading for me, if I let it run with the crash dialog up, steam does indeed load in a broken state that does allow at least some gameplay, but there's no networking features like the friends list refuses to go online. I did previously state that. If there's anything I can do to help troubleshoot the issues please let me know.

It still is an issue. I got steam to work by messing around. I now have steam installed on a local disk with symbolic links in my home directory from .steam and .local/share/Steam to the local drive. I am still unable to install steam on my NFS mounted home directory like I have been able to do for over a year prior. These are work arounds not fixes. As a Unix/Linux developer who has been using Linux since basically day one, and Unix a long time before. I see no reason I can't install steam on a NFS filesystem, I also see no reason things should break when I install it on a local disk. Something in the update that was pushed out at beginning of May caused things to break.

Do "steam -no-browser"

The problem is it keeps looping during the install. It never brings up the browser. Installed on local disk on same machine it works fine. Until it updated early this month it worked fine on NFS system. This is installation issue, not a run time issue.

@MrFrog222 adding -no-browser has no effect here, vmtouch is the only work around with any real impact for me. Keep the suggestions coming please.

@tsukasa1234567 for me this was the only thing that worked in combination with installing steam-native-runtime but i dont know if thats important

@smcv steamwebhelper crashes don't lead to steam not loading for me, if I let it run with the crash dialog up, steam does indeed load in a broken state that does allow at least some gameplay, but there's no networking features like the friends list refuses to go online. I did previously state that. If there's anything I can do to help troubleshoot the issues please let me know.

I am seeing the same issue. Ubuntu 20.04, suddenly cannot see family-shared games, pull up the friends list, or do anything else. I do get the steamwebhelper crash notification and then the UI loads anyways.

-no-browser doesnt fix anything for me
also for some reason now whenever i try to open some of the games that were not working instead of just crashing it freezes and then i have to restart xorg to get opengl or vulkan working again

i just copied ~/.local/share/Steam to another ssd using the xfs filesystem and i changed the stuff in ~/.steam to link to the stuff in /mnt/steam
it is actually using that directory but it still doesnt work even though using another distro without zfs does

From what I've seen trying to debug and workaround various issues with steam, the pressure-vessel bubble-wrap flatpak containerization breaks pretty easily when when symlinks are added. If you instead mounted that ssd onto ~/.local/share/Steam it's got a pretty decent chance of working.

mounting /dev/nvme0n1 to ~/.local/share/Steam does seem to work

EDIT:
a reboot fixed most of the games on there when using this workaround but some native games still dont work

I was able to run just symlinking ~/.local/share/Steam and others in ~/.steam to a local ext4 directory (Steam v021-20240524-1716584667 NVidia 550.78 Fedora 40 xfce4).

The 3GB of local space for Steam app is a small price to pay for allowing game storage to be from nfs share: Settings -> Storage, click current local drive, Add Drive, choose nfs mount path where games are stored, click ... button, Make Default.

I tried this also, it would not let me select the NFS store to save games in. Also is saved games under ~/.local/share/Steam rather then ~/.local/share where it did before. So we are looking at a large amount of storage on my local disk. My local disk is not much bigger then I need for the system. And I can flush it if my system gets messed up. With Steam on it I would have to back it up before trashing it.

I have a few machines with no hard drives at all, so the only alternative that I tried was using zram/ext4 mounted at ~/.local/share/Steam, it works for using steam without the steamwebhelper crashes, but it won't simply let me pick a folder for using as the steam library, and I'm not about to install games to RAM lol.

So, steam had an update in the last few days, luckily I got to grab some good games from the steam sale before rebooting. Now even using vmtouch doesn't seem to be helping the chances of getting into steam without steamwebhelper crashing. I used to be able to just leave the error window alone, and it would load into steam with friends and some other things not working, but I could play games still.

But now I get this

But now I get this

Please try to copy/paste text, instead of pictures of text: pictures of text aren't searchable or available to accessibility tools.

The error message shown in the screenshot is:

Something went wrong while displaying this content. _Refresh_

Error Reference: Shared SteamUI_9015621_3b4c674e2a32af39

Cannot read properties of undefined (reading 'GetPlayer')

This is a different issue, which is not the same thing discussed here. It's probably best if you report it separately.

(The issue described in the initial report here is the "Steamwebhelper is not responding" crash dialog. In the error message shown in your screenshot, the steamwebhelper must be communicating with Steam successfully - otherwise the window content would be blank - but the web content that it's displaying is not working as intended. The fact that it mentions properties of undefined suggests that something has gone wrong at the Javascript level.)

Sorry, I'll try to copy the text next time, also, I still had the error window with steamwebhelper has crashed, like I said before usually I can still play games by ignoring that error window. So steamwebhelper was not communicating correctly. Don't worry about it too much, it seems to be a kernel bug, I tried updating mesa, reboot, test, then kernel, reboot, test. I didn't see anything in the shortlog's that seemed fitting. I'll try to stay more on topic.

Are there any links you can share about pressure-vessel's general execution flow? I'd like to mess around with it since not much has been discovered on what happened to cause this issue way back in Jan. I'll do some digging, but tips from the pro would be helpful

CrashID=bp-dfd0d27a-89ab-43fb-965a-9a24e2240713
reinstalled steam, it wouldn't work for me to even sign-in without the steamwebhelper crash dialog.
Then I made a zram/ext4 drive, installed steam to it, everything worked fine, used about 3.5 gigs of ram.
couldn't add a library folder anywhere, after I pick a location it just instantly crashes.
copied the same install back to nfs, getting the steamwebhelper crashes again, but now I'm seeing this in the terminal repeated quite a bit
src/clientdll/compatmanager.cpp (1104) : GetCompatibilityToolCommandLineInternal: inconsistent state! do not call while cache off job is running.

Looks like there are multiple crash ID's so I ran it again to paste all of them.
CrashID=bp-ccb122bf-1578-4ee0-a8ab-6f19d2240713
CrashID=bp-d4d9bc45-8a55-42db-a3bf-b78ef2240713
CrashID=bp-bbb68945-13e1-4339-9c4d-e0c122240713
CrashID=bp-9d461bb6-4285-489c-98c1-ba8d62240713

That's in 1 start up attempt, yes it opens a steamwebhelper crash dialog, and yes it open the black screen saying
Something went wrong while displaying this content. Refresh
Error Reference: Shared SteamUI_9015621_3b4c674e2a32af39
Cannot read properties of undefined (reading 'GetPlayer')

What other information can I provide that'll help get this resolved?

I've made lots of progress, I can get steam to start without error's now on the 2nd attempt. The issue is with the tmp-RANDOM folders being created, I haven't figured out the why part yet at all. Forcing only those to tmpfs wastes about 3.5 gigs of ram, so it's not ideal, but sure beats looping restarts at around 40 seconds a piece with closing the errors and all.

@smcv do you have any ideas how I can narrow down things any further? The 2 tmp folders have the issue.

So the 1st tmp-blah folder is created trying to load steam. After the crash, reloading steam usually works. If it doesn't I have to run an "env-update && . /etc/profile" to regenerate the ld.so cache, then do it again and it works.

The 2nd tmp-blah folder seems to be generated by proton, I made a file name listing of both tmp-blah folders and compared them, they are identical. So it might be worth looking into to have proton reuse the already generated tmp folder of which libraries to load, that would be 1.7 GB wasted instead of 3.5 GB

I believe all that's needed is to check that the fallback copying of the libs is complete before moving on, I get the "For best results" ... "should both be on the same fully-featured Linux filesystem" For quite awhile after the steamwebhelper crash.

FWIW: I now have 2.5GbE to my desktop with a 10GbE backbone to the NFS server. NFS performance is now better than some of my older machines' hard drives and they run steam with no problem. I don't see how the performance of NFS can be blamed for this, especially since my workaround has put Steam into a local drive with all game data (everything under the "steamapps" subdir) still on NFS via a symbolic link, and all games launch and run just fine. So the problem is exclusively with the Steam launcher. Why would something like a game launcher have more stringent requirements than any/all games?

Unfortunately, my workaround is still pretty painful as it locks all my family members to one machine each, where previously we could log into our own accounts on any machine to play any of our games. This is like going back 15 years in time. It's especially painful since I lost a local (NVMe) drive recently which meant a manual recovery that wouldn't be necessary with everything in NFS (it should have been an inconsequential event).

Is there any way we can narrow this down further?
Do tsukasa1234567's comments and suggestions lead in a good direction for testing? (s)he seems to have done quite a bit of solid debugging and the suggestions sound solid.

Nice setup glabifrons, and I totally agree, I've tweaked NFS/ZFS/Linux in general to work far better than spinning disks would locally for sure. In the source code for pressure-vessel it calls to make a ref link between binaries and libs in the root of the linux machine and steam's steamruntime folder. When they aren't on the same filesystem or that file system doesn't support ref links, it gives out tons of warning as it falls back to "slow" copying of what it needs to run to a tmp folder in steam. The issue I see directly from that is in the log at the near beginning that says launching steamwebhelperalongsidesteam. So I get the standard crash dialog window while my terminal is still copying files. if I wait the extra couple seconds for the copying to finish, then restart steam, it'll show that it's doing the copying all over again, but since the files are already there, it works and signs in without issues while I still see the copying activity.

For anyone that wants to see that in action and use the clunky work around.

PRESSURE_VESSEL_VARIABLE_DIR=/any/tmpfs/location steam
wait for copying to finish, then pick exit on the crash dialog
PRESSURE_VESSEL_VARIABLE_DIR=/any/tmpfs/location steam
should be working, leave the terminal up

You will lose around 3gigs of ram doing it that way, which isn't ideal lol

steam needs to either wait for the copying to complete before launching steamwebhelper, or give us a command line switch so we can individually add a delay ie. steam -delay-start=15

Thanks, yeah, it was shockingly cheap for all the parts on Aliexpress. Total was cheaper than the cost of a 2.5GbE switch from only a year ago (even including the single-mode fiber and SFP+ modules).

As to reflinks, are you sure? The standard filesystems (eg: ext4) don't support reflinks, so that should be breaking the vast majority of installations out there. From what I can tell, they're only supported on XFS & BTRFS. Google says the Steam Deck uses ext4, so I wonder what's different about its build.

Back when I first opened this issue, I spent a lot of time digging and found absolute gobs of symbolic links being created. Did they change from that approach?

I'll have to try your workaround, as it sounds much more resilient than mine. Thank you very much for the pointer!

As to your command line switch suggestion, I hope someone in Steam is still following this.

Edit:

I get the standard crash dialog window while my terminal is still copying files.

I forgot to mention... I have a screenshot of a dialog from Steam showing it's in the middle of unpacking with the crash dialog on the screen at the same time.

It's been awhile since I looked through the source code, had reflink in my head at the time. It's making HARD links, so as my machines have a shared nfs root, and each there own home folder also mounted, it's counted as a different filesystem, hardlinks can't cross filesystems, not sure if it would be the same if it was somehow a single nfs share. I can try to move just 1 user and try to bypass some startup entries to test it. Maybe a winter project.

FWIW: I now have 2.5GbE to my desktop with a 10GbE backbone to the NFS server. NFS performance is now better than some of my older machines' hard drives and they run steam with no problem.

How do you measure NFS performance BTW? Accessing tons of small files is an entirely different story than streaming data from a large file, and I suspect Steam may be more dependent on the former.

I don't see how the performance of NFS can be blamed for this, especially since my workaround has put Steam into a local drive with all game data (everything under the "steamapps" subdir) still on NFS via a symbolic link, and all games launch and run just fine. So the problem is exclusively with the Steam launcher. Why would something like a game launcher have more stringent requirements than any/all games?

In my experience it's more dependent on the performance of the client machine. My main desktop with a Ryzen 5950X and 128 GB of DDR4 can usually run Steam from NFS. For the first run of the day it may fail once but works on subsequent tries. My home theater PC with an i7-950 (yes a 1st gen i7) and 12 GB of DDR3 can hardly get Steam started at all, and it takes a lot longer to even show the "not responding" popup. Both of the machines are connected to the same NFS server through the same network. The server is a Xeon E3-1230v2 with 32 GB of DDR3, so not that much newer or faster than the home theater PC.

Perhaps they are doing something under root, my NFS share does not allow root access to files. Have not tried latest installer. So maybe they fixed it.

There's plenty to tune on every layer.
sysctl entries, you'll have to test for your setup, trade offs between single/multi client performance or latency/bandwidth, these have a large impact.

fs.nfs.nfs_congestion_kb *based on un-acked packets I believe, raising is good for my setup
net.core.max_skb_frags *newer kernels can (if you roll your own kernel config) use more addresses in the scatter gather to build packets, helping smaller files

net.core.netdev_max_backlog *this is a backlog of packets, not data, tiny files fill this quickly leading to start/stop behavior

net.core.rmem_default *auto config does well, but basically heavy network users/servers can safely increase by double or so since we are more network intensive than usual users
net.core.rmem_max
net.core.wmem_default
net.core.wmem_max

net.ipv4.tcp_autocorking *adds microsecond delays at times to have fuller packets, helps small files
net.ipv4.tcp_ecn *I use MTU 9000 network wide and 65k packets with GRO
net.ipv4.tcp_ecn_fallback

net.ipv4.tcp_mem * NFS uses tcp, very similar to the net.core section
net.ipv4.tcp_mtu_probing *part of using higher MTU
net.ipv4.tcp_rmem
net.ipv4.tcp_slow_start_after_idle *tcp forgets bandwidth per path after time passes, this start sending faster than the traditional slow ramp up, not recommended for networks that are never idle
net.ipv4.tcp_wmem

now back to topic :)

so steam creates usually under ~/.steam/steam/ubuntu12_64/steam-runtime-sniper/var
a tmp-6Chars folder when starting up, containing.
bin -> usr/bin
etc -> usr/etc
lib -> usr/lib
lib32 -> usr/lib32
lib64 -> usr/lib64
overrides -> usr/lib/pressure-vessel/overrides
sbin -> usr/sbin
usr

For standard users, the host files are quickly hard linked into these locations while steamwebhelper is launching.
For NFS users, the Hard links fail, pressure-vessel falls back to copy mode and copies everything itself into those locations.
The race condition depends entirely on how far the copy process gets to completion when steamwebhelper tries to access the folders as that is the runtime that's used.
So that's why either steamwebhelper needs to wait and check for the process to finish before launching, or give a command line switch to let those with issues set their own delay in seconds or whatever to have a reliable experience.

This thread has been open almost a year, so I assume I won't be too harshly reprimanded for going off topic to help fellow performance enthusiasts.

I've been trying the workaround here https://github.com/ValveSoftware/steam-for-linux/issues/10431#issuecomment-2560090669 but each time the temporary directory in $PRESSURE_VESSEL_VARIABLE_DIR gets wiped and the copy starts all over again.

I think the underlying issue might be related to mount namespaces. If I do this then an error message "Steam now requires user namespaces to be enabled." is reported in a popup and "steam-runtime-check-requirements[1400964]: W: Child process exited with code 1: bwrap: Creating new namespace failed: nesting depth or /proc/sys/user/max_*_namespaces exceeded (ENOSPC)" is reported in the terminal.

$ sudo sysctl -w user.max_mnt_namespaces=0

Possibly something about NFS is limited when creating the mnt or user namespaces. These seem to be the only ones in use comparing the steamwebhelper process against the running ls.

$ ls -la /proc/1397521/ns/
total 0
dr-x--x--x 2 user @user 0 Jan  9 17:38 .
dr-xr-xr-x 9 user @user 0 Jan  9 17:37 ..
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 mnt -> mnt:[4026539302]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 net -> net:[4026531840]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 pid -> pid:[4026531836]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 time -> time:[4026531834]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 time_for_children -> time:[4026531834]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 user -> user:[4026539301]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 uts -> uts:[4026531838]


$ ls -l /proc/self/ns/
total 0
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 cgroup -> cgroup:[4026531835]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 ipc -> ipc:[4026531839]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 mnt -> mnt:[4026531841]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 net -> net:[4026531840]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 pid -> pid:[4026531836]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 pid_for_children -> pid:[4026531836]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 time -> time:[4026531834]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 time_for_children -> time:[4026531834]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 user -> user:[4026531837]
lrwxrwxrwx 1 user @user 0 Jan  9 17:38 uts -> uts:[4026531838]

The "tmpfs" location is on tmpfs, not a temporary folder mounted over NFS.

The "tmpfs" location is on tmpfs, not a temporary folder mounted over NFS.

I set the tmpfs mount up:

# mount -t tmpfs tmpfs /tmp/.steam-user
# chown user:@user /tmp/.steam-user

During the first copy PRESSURE_VESSEL_VARIABLE_DIR=/tmp/.steam-user steam:

$ ls -la /tmp/.steam-user/
total 0
-rw-r--r--  1 user @user   0 Jan 10 05:53 .ref
drwx------ 3 user @user 60 Jan 10 05:53 tmp-A9F7Z2

At the end of the first execution when exec ./steamwebhelper .... is logged in the terminal I select "Exit Steam" from the "Steamwebhelper is not responding" dialog.

If I repeat the start command the original tmp-xxxxxx is removed and a new one is created with the copy starting all over again seems to take just as long and still results in the crash dialog.

$ ls -la /tmp/.steam-user/
total 1025
-rw-r--r--  1 user @user  0 Jan 10 05:53 .ref
drwx------  3 user @user 60 Jan 10 06:01 tmp-3FEM02

Strange, I usually leave the terminal up and watch to copy process while I have the steamwebhelper error dialog and steam itself is loaded with the black error screen, I don't see an exec ./steamwebhelper line, I'll have to do some testing to see what could be different in our setups.

I'm running Gentoo, I update my kernel to the newest mainline roughly every week (none of the RC's) I have the namespaces configs in the kernel as =Y for at least the last 30 releases.

I'm not sure why your tmp folder is being removed and mine isn't at this point. As a possible test, once steam is stuck in it's non-working state, in another terminal do a chown or chmod and remove your permissions to the tmp folder, so you can't delete it when the process stops, you can then change it back and see if it works when you start it up again.

I get the same problem as @JKDingwall, steam always creates a temporary subfolder with a unique name in the tmpfs parent dir at startup, so steam can never reuse any previous copy. Also it is very slow and takes hours to copy things.

So I just did a test, started steam like 20+ times without tmpfs, and big surprize it never worked. Went back and tried it with the tmpfs, when the transfers finish in the terminal, I pick to exit steam in the popup. when the terminal returns to the normal user prompt, I usually just hit up arrow and enter, and it starts fine after showing that it's transferring again. So what I did notice is that it's still trying to send the last crash report while it's starting up the 2nd time, I think that's whats holding the lock and allowing me to start it with the tmp-blah folder from the 1st attempt.
So next I deleted the tmpfs and tried it again, after the 1st crash I let it sit after using exit from the popup. After maybe 3-5 seconds I got the crash upload report in the terminal, so then I tried it again after that and it still crashed. I don't know if that helps, but that's what I'm seeing here. Sure would be nice for someone from valve to get involved.

After significant upgrades and questionable (unsafe) tuning to my NFS server, I now have Steam consistently launching.
However, it still displays this failure dialog every time, even though Steam is actually launching.
The timing is consistent:
25 seconds after launch, the error message is displayed.
32 seconds after launch, Steam's full UI appears with no problems, but the failure dialog remains.

So at this point it looks like the failure dialog is bogus, which is not encouraging.

I will repeat my previous statement that a game launcher should definitely not have higher performance requirements than any game that launcher will ever launch. I've never had a game fail to launch due to performance reasons after 15 years of running Steam on Linux, so this 15 month long issue is baffling.

Having been frustrated by this bug not yet being resolved I have devised a rather dumb workaround that allows users (aka the kids) to switch machines and have their Steam profile/installation follow them. A block file on NFS with the ext3 filesystem for Steam to use (and then installing games direct to NFS). This has been working fine for us for a few months now.

For my particular setup the steam library data goes on an NFS mount at /nas/steam-libraries. Under which there is a directory for each user.

In /etc/fstab for my setup the mount point is configured like this:

nas-hostname.local:/mnt/data2/network-shares/games/steam-libraries  /nas/steam-libraries     nfs4 sec=krb5,rw,auto        0       0

Then in each /nas/steam-libraries user directory goes the following.

Create a directory for mounting an alternate filesystem:

mkdir steam-share

Create block file for mounting (I picked 10GB because for one user the current ~/.local/share/Steam was 5.2GB):

mkfs -t ext3 steam-share-fs 10000M

A quirk of my set up is that root on client machines don't get access as root on the nas, so this change may not be required in other environments depending on permissions, but here I chown the file to nobody:nogroup so that 'root' on client machines can mount the file as read/write:

chown nobody:nogroup steam-share-fs

I haven't worked out an automount solution that doesn't result in a cycle breaking the boot process, so have a bit of additional jank mounting wise from this point (an entry per user in both /etc/fstab and /etc/sudoers). Replace <username> with relevant username.

Adding to /etc/fstab:

/nas/steam-libraries/<username>/steam-share-fs /nas/steam-libraries/<username>/steam-share ext3 rw,loop,noauto

(reminder to reboot or systemctl daemon-reload here for updated fstab)

Mount for the first time and set user ownership inside the block file (set group as appropriate):

sudo mount /nas/steam-libraries/<username>/steam-share
sudo chown <username>:<group> /nas/steam-libraries/<username>/steam-share

visudo and add to allow users to mount their own steam-share:

<username> ALL=(root) NOPASSWD:/usr/bin/mount /nas/steam-libraries/<username>/steam-share

As the user:

Mount the steam-share (if it isn't aready mounted from earlier):

sudo mount /nas/steam-libraries/$USER/steam-share

WARNING: backup of ~/.steam and ~/.local/share/Steam might be prudent here since I'm about to suggest removing them if they already exist.
Remove or rename ~/.steam and ~/.local/share/Steam.

Link to our new mounted steam-share location for Steam to use:

ln -s /nas/steam-libraries/$USER/steam-share ~/.local/share/Steam

At this point the user should be able to run Steam (and have it install the required files on the block mounted file via the symlink from ~/.local/share/Steam to /nas/steam-libraries/$USER/steam-share). The default install location would be inside the 10GB steam-share filesystem that was created, so in Settings -> Storage a 'Steam Library' should be created that points to the regular NFS mount (and set as the default), in my case for example: /nas/steam-libraries/$USER/library.

From here I added a script to the kids desktop profiles to run sudo mount /nas/steam-libraries/$USER/steam-share when they login so they don't need to be operating a terminal and Steam can then be started normally at will and it is all coming from NFS. No Steam specific local storage required.

Interesting! I'm thinking about turning some machines into diskless ones, so the trick may prove useful.

This bug not being resolved is very frustrating indeed!

I've got steam working on an nfs home again using the beta client and/or SteamRT3. I didn't check them separately to see which was the significant change. In order to use these without changing settings via the user interface:

$ echo "publicbeta" ~/.steam/root/package/beta
$ touch ~/.steam/root/.steam-enable-steamrt64-client

Yeah, I gave up on them fixing it as they seem to have abandoned it (look how much later my last reply is vs theirs.

Steam (employees and/or contractors):
The Steam UI should not have more stringent performance requirements than any/all games it will ever launch!

When I had an ugly series of links to stuff Steam into a local SSD with all games and compat files still on the NFS server, not one person in my family ever had a single game complain. Yet if Steam is loaded from NFS, it cries that it's not loading instantly and commits hara-kiri!

My current workaround is to set up ZFS (behind NFS) for the Steam filesystem only as follows:

• atime disabled (recommended)
• compression enabled (recommended, decompression is faster than drive transfer)
• sync=disabled (not recommended)

I also have a crazy over-powered NFS server now (massive upgrade from previous) with 64GB of RAM available for ARC (caching). It only sees very mild spikes in load during game updates/patching... nothing else we do stresses it the slightest bit.

...
In case it was missed:

Steam (employees and/or contractors):
The Steam UI should not have more stringent performance requirements than any/all games it will ever launch!

I basically got tired of waiting on this and moved my games to a local disk with a bit of symbolic links. Unfortunately steam client still can't be located on a NFS disk, with games being stored on a local disk. I have since added a second hard drive to my system to store my games since I don't have large drive on machine that I only have the OS installed on. With a bit of symbolic links I moved steam out of home directory to directory on second disk.

Warning to anyone moving Steam to a local drive and using links to keep your games and saves on your NFS server:
If you want your saves on the NFS server for backups/resiliency, you'll need multiple directories relocated/linked:

• ~/.local/share/Steam/steamapps/ (doing it at this level will include both "common" and "compatdata" which are your game installations and your emulated windows environments where saves are usually stored for Windows platform games).
• ~/.local/share/Steam/userdata/ (a small handful of games store saves here... we found out the hard way recently)
• ~/.local/share/Steam/compatibilitytools.d/ ( less critical, but makes it easier to manage, this is where your GE-Proton versions are unpacked)