Test with the beta client on the previous scout runtime. Maybe with scout SLR branches?

If you want to try the beta client with the previous scout runtime in order to rule out this being triggered by runtime changes, you can run with:

STEAM_RUNTIME=/path/to/steam-runtime \
STEAM_RUNTIME_SCOUT=/path/to/steam-runtime \
steam

where /path/to/steam-runtime is the result of unpacking https://repo.steampowered.com/steamrt-images-scout/snapshots/0.20211207.0/steam-runtime.tar.xz (or older).

This should work for both the runtime configurations that matter:

• the LD_LIBRARY_PATH runtime used by Steam itself and non-SLR games uses $STEAM_RUNTIME;
• the container runtime used by SLR games uses $STEAM_RUNTIME_SCOUT

It seems the Steam update causing this has been rolled out now, at least I'm also having crahes now since today's update of the Steam client.

Noteworthy differences to above description:

1. The crash does happen also when the Steam overlay is disabled (tried both just for the game and globally in the client), so it is not related to the overlay but some other component.
2. The crash does not happen when creating a steam_appid.txt with the appid in it and starting from command line. This works even when the Steam client is running and the game started from CLI attaches to the running client (I see the " to start overlay" message, which kind of proves that the enabled overlay is not the problem).
3. While the crash sometimes happens before the main menu, it mostly crashes immediately after loading a savegame for me (main interface is appearing for a fraction of a second, then crash).

I also tried to run it with above snapshot of the Steam runtime, but it seems Steam tries to be clever and updates the runtime and the game is crashing anyway.

steam.sh[3287]: STEAM_RUNTIME has been set by the user to: /home/ubuntu/snapshots_0.20211207.0/steam-runtime
setup.sh[3376]: Updating Steam runtime environment...

Is there a way (e.g. environment variable) to prevent that auto-update?

Edit: Maybe I'm also misinterpreting that message and it meant that it applied the change to the other runtime? In that case: Since the crash was still happing, the problem is either not the Steam runtime or that snapshot was already affected.

Updating Steam runtime environment...

This message is a bit misleading, but it's the closest we could get while keeping it reasonably concise. Technically it should say something more like: updating the Steam Runtime's idea of whether each library it includes is newer or older than the one on your host system, so that it can use the system version if it's at least as new as the Steam Runtime version.

does happen also when the Steam overlay is disabled

If you disable the Steam overlay through Steam's UI, I think that doesn't completely disable the LD_PRELOAD module, just the part that alters what's displayed.

Setting the launch options to LD_PRELOAD="" %command% will fully disable it, at the cost of breaking some Steam features that rely on it (in particular Steam Input and Remote Play).

Hi,

I am one of the developers of Railway Empire and had a look into this after our QA and users had the same issue. I basically came to the same conclusion as described above, although I do not think the crash itself originates in FMOD.
Sure, there is almost always a thread with a stack frame in FMOD, but the game is also almost never really silent. The crashes I could observe were always threads somewhere at a memory address between shared libs, but none of the loaded libs' address space actually matched that address. Not sure how or where that thread is spawned, but this behavior only occurs if gameoverlayrenderer.so is loaded, as stated above. I am not sure how to proceed, let me know if I can provide you with any more information.

I do not think the crash itself originates in FMOD. Sure, there is almost always a thread with a stack frame in FMOD, but the game is also almost never really silent.

What sort of a crash is this? SIGSEGV, SIGABRT, SIGILL, other?

If you can get the game to crash with a debugger like gdb or gdbserver attached, the debugger should be able to identify which thread the fatal signal was delivered to. It is not necessarily that thread's fault (if there's memory corruption, then it can cause "action at a distance"), but it's reasonably good evidence at least.

Yeah, this is probably a memory corruption somewhere else. At least the trace does not make much sense to me:

Thread 35 "RailwayEmpire" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f269723a640 (LWP 3519)]
0x00007f2752f02a81 in ?? ()
(gdb) thread apply 35 bt full

Thread 35 (Thread 0x7f269723a640 (LWP 3519) "RailwayEmpire"):
#0  0x00007f2752f02a81 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007f27535f8700  0x00007f2753665bea  Yes (*)     /home/gminds/.local/share/Steam/ubuntu12_64/gameoverlayrenderer.so
0x00007f27535c6020  0x00007f27535d5e46  Yes (*)     /usr/lib/libspeex.so.1
0x00007f2753473020  0x00007f27534fcd79  Yes (*)     /usr/lib/libopenal.so.1
                                        No          libsteam_api.so
                                        No          libfmodex64-4.44.42.so
                                        No          libfmodevent64-4.44.42.so
                                        No          libfmodeventnet64-4.44.42.so
0x00007f2752cda020  0x00007f2752d6377e  Yes (*)     /usr/lib/libX11.so.6
0x00007f2752204800  0x00007f2752256de8  Yes (*)     /home/gminds/.local/share/Steam/ubuntu12_32/steam-runtime/pinned_libs_64/libGLU.so.1
0x00007f2752928020  0x00007f27529a1e18  Yes (*)     /usr/lib/libm.so.6
0x00007f27531fc020  0x00007f27531fc0f5  Yes (*)     /usr/lib/libdl.so.2
0x00007f27525bc020  0x00007f27525bea1d  Yes (*)     /usr/lib/libGL.so.1
0x00007f2752073020  0x00007f275217b0e3  Yes         /usr/lib/libstdc++.so.6
0x00007f27531e3020  0x00007f27531f4455  Yes         /usr/lib/libgcc_s.so.1
0x00007f27531dc020  0x00007f27531dc0f5  Yes (*)     /usr/lib/libpthread.so.0
0x00007f2751dfc440  0x00007f2751f7069d  Yes (*)     /usr/lib/libc.so.6
0x00007f27531d7020  0x00007f27531d7227  Yes (*)     /usr/lib/librt.so.1
0x00007f275368c000  0x00007f27536b2535  Yes (*)     /lib64/ld-linux-x86-64.so.2
0x00007f2752ca0020  0x00007f2752cb2b86  Yes (*)     /usr/lib/libxcb.so.1
0x00007f2752503020  0x00007f2752504f57  Yes (*)     /usr/lib/libGLdispatch.so.0
0x00007f27528e8020  0x00007f2752901aa1  Yes (*)     /usr/lib/libGLX.so.0
0x00007f27531d2040  0x00007f27531d2d52  Yes (*)     /usr/lib/libXau.so.6
0x00007f27531cb020  0x00007f27531cc882  Yes (*)     /usr/lib/libXdmcp.so.6
0x00007f26e2578000  0x00007f26e38f28ff  Yes (*)     /home/gminds/.local/share/Steam/linux64/steamclient.so
0x00007f27528a3020  0x00007f27528ad33e  Yes (*)     /usr/lib/libXext.so.6
0x00007f26e2112020  0x00007f26e2156e33  Yes (*)     /usr/lib/libGLX_mesa.so.0
0x00007f26e20cb020  0x00007f26e20d8921  Yes (*)     /usr/lib/libglapi.so.0
0x00007f275247c020  0x00007f27524864ca  Yes (*)     /usr/lib/libdrm.so.2
0x00007f26e20ac020  0x00007f26e20b46a6  Yes (*)     /usr/lib/libxcb-glx.so.0
0x00007f27531be020  0x00007f27531be10b  Yes (*)     /usr/lib/libX11-xcb.so.1
0x00007f27531b8020  0x00007f27531b91a6  Yes (*)     /usr/lib/libxcb-dri2.so.0
0x00007f2752c8d020  0x00007f2752c8f5e6  Yes (*)     /usr/lib/libXfixes.so.3
0x00007f2752899020  0x00007f275289b82b  Yes (*)     /usr/lib/libXxf86vm.so.1
0x00007f27531b2020  0x00007f27531b2b44  Yes (*)     /usr/lib/libxcb-shm.so.0
0x00007f26e2073020  0x00007f26e2091357  Yes (*)     /usr/lib/libexpat.so.1
0x00007f2752473020  0x00007f2752473e34  Yes (*)     /usr/lib/libxcb-dri3.so.0
0x00007f26e206b020  0x00007f26e206b8a6  Yes (*)     /usr/lib/libxcb-present.so.0
0x00007f26e2063020  0x00007f26e20651b2  Yes (*)     /usr/lib/libxcb-sync.so.1
0x00007f26e205c020  0x00007f26e205c2ec  Yes (*)     /usr/lib/libxshmfence.so.1
0x00007f26e2054020  0x00007f26e20566cc  Yes (*)     /usr/lib/libxcb-xfixes.so.0
0x00007f26e0528020  0x00007f26e170b373  Yes (*)     /usr/lib/dri/radeonsi_dri.so
0x00007f26da38c040  0x00007f26ddba0d02  Yes (*)     /usr/lib/libLLVM-13.so
0x00007f26d9a52020  0x00007f26d9a5f19c  Yes (*)     /usr/lib/libz.so.1
0x00007f26d99a5040  0x00007f26d9a3c332  Yes (*)     /usr/lib/libzstd.so.1
0x00007f26d9992020  0x00007f26d9998776  Yes (*)     /usr/lib/libsensors.so.5
0x00007f26d9982020  0x00007f26d9988045  Yes (*)     /usr/lib/libdrm_radeon.so.1
0x00007f26d9968020  0x00007f26d9978885  Yes (*)     /usr/lib/libelf.so.1
0x00007f26d995c020  0x00007f26d9960974  Yes (*)     /usr/lib/libdrm_amdgpu.so.1
0x00007f26d9951020  0x00007f26d9954dab  Yes (*)     /usr/lib/libdrm_nouveau.so.2
0x00007f26d98e9020  0x00007f26d9929ecd  Yes (*)     /usr/lib/libvulkan.so.1
0x00007f26d98c9040  0x00007f26d98ce636  Yes (*)     /usr/lib/libffi.so.8
0x00007f26d9895020  0x00007f26d98b19d8  Yes (*)     /usr/lib/libedit.so.0
0x00007f26d982e020  0x00007f26d986edd0  Yes (*)     /usr/lib/libncursesw.so.6
0x00007f26d96bd020  0x00007f26d97c1975  Yes (*)     /usr/lib/libxml2.so.2
0x00007f26d94fa020  0x00007f26d95eb348  Yes (*)     /usr/lib/libicuuc.so.70
0x00007f26d946c020  0x00007f26d94832d3  Yes (*)     /usr/lib/liblzma.so.5
                                        Yes (*)     /usr/lib/libicudata.so.70
0x00007f26c47aa020  0x00007f26c47dfe01  Yes (*)     /usr/lib/libnss_mymachines.so.2
0x00007f26d401f020  0x00007f26d40237bb  Yes (*)     /usr/lib/libcap.so.2
0x00007f26d4017020  0x00007f26d4018353  Yes (*)     /usr/lib/libnss_mdns4_minimal.so.2
0x00007f26d4006020  0x00007f26d400e1da  Yes (*)     /usr/lib/libresolv.so.2
0x00007f26c4782020  0x00007f26c4798321  Yes (*)     /usr/lib/libnss_resolve.so.2
0x00007f26c475d020  0x00007f26c476b501  Yes (*)     /usr/lib/libnss_myhostname.so.2
0x00007f26c4352020  0x00007f26c43532e1  Yes (*)     /usr/lib/libpulse-simple.so.0
0x00007f26c4308020  0x00007f26c4336f20  Yes (*)     /usr/lib/libpulse.so.0
0x00007f26c4286020  0x00007f26c42ce593  Yes (*)     /usr/lib/pulseaudio/libpulsecommon-15.0.so
0x00007f26c422c020  0x00007f26c425a56a  Yes (*)     /usr/lib/libdbus-1.so.3
0x00007f26c41a3020  0x00007f26c41f8e5f  Yes (*)     /usr/lib/libsndfile.so.1
0x00007f26c40d6020  0x00007f26c4159cd1  Yes (*)     /usr/lib/libsystemd.so.0
0x00007f26c40ba020  0x00007f26c40bc67a  Yes (*)     /usr/lib/libasyncns.so.0
0x00007f26c4021020  0x00007f26c402363f  Yes (*)     /usr/lib/libvorbisenc.so.2
0x00007f269bfcc020  0x00007f269bff4c36  Yes (*)     /usr/lib/libFLAC.so.8
0x00007f269bf6c020  0x00007f269bfaf412  Yes (*)     /usr/lib/libopus.so.0
0x00007f269bf3f020  0x00007f269bf557e8  Yes (*)     /usr/lib/libvorbis.so.0
0x00007f26c4004020  0x00007f26c40063be  Yes (*)     /usr/lib/libogg.so.0
0x00007f269bf1b020  0x00007f269bf35c97  Yes (*)     /usr/lib/liblz4.so.1
0x00007f269bdeb040  0x00007f269becbf88  Yes (*)     /usr/lib/libgcrypt.so.20
0x00007f269bdba050  0x00007f269bdcf4f2  Yes (*)     /usr/lib/libgpg-error.so.0
0x00007f26966d0020  0x00007f26966fc50d  Yes (*)     /usr/lib/libjack.so.0
0x00007f2696530020  0x00007f26966731b1  Yes (*)     /usr/lib/libdb-5.3.so

Obviously the exact addresses change from crash to crash, but I noticed that the crashing address always ends with 0x2a81

Edit: This is the latest Manjaro with the preinstalled Steam runtime if that changes anything

This could be a bad function pointer getting saved in a vtable, or some part of a vtable getting overwritten with a non-pointer?

0x2a81 is odd, so it can't possibly be a valid pointer to a function (or anything else with non-trivial alignment).

We do not deliberately manipulate any vtables, but it could of course still be a side effect of memory corruption.
I'm not really sure how to proceed. I could try to create a build with FMOD disabled and see if we get the same behavior, but that could just shift the problem somewhere else without us noticing.
I guess the overlay library only injects into the renderer, I also could have a look if I can verify the OpenGL calls or at least see any notable differences. Otherwise I am out of ideas, the game ran fine up to a point and still does when the library is not loaded. But so far I did not see any other game having the same problem. It must be some interaction between our code and the library then, right?

Just to give an update: we could relate the crashes to certain music files being played, so the culprit seems indeed to be FMOD. For some reason coredumps analyzed after the crash occurred give another trace than observing the crash live

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f88a3502a81 in ?? () from libfmodex64-4.44.42.so
[Current thread is 1 (Thread 0x7f880458e640 (LWP 5400))]
(gdb) bt full
#0  0x00007f88a3502a81 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#1  0x00007f88a34fe239 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#2  0x00007f88a34ff147 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#3  0x00007f88a34ff6a6 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#4  0x00007f88a345960b in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#5  0x00007f88a34b9632 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#6  0x00007f88a3460cca in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#7  0x00007f88a34b9684 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#8  0x00007f88a344671e in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
#9  0x00007f88a34472e6 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
[#10](/issue/ValveSoftware/steam-for-linux/10) 0x00007f88a344391f in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
[#11](/issue/ValveSoftware/steam-for-linux/11) 0x00007f88a3443aff in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
[#12](/issue/ValveSoftware/steam-for-linux/12) 0x00007f88a344e7b3 in FMOD::SystemI::createSoundInternal(char const*, unsigned int, unsigned int, unsigned int, FMOD_CREATESOUNDEXINFO*, FMOD::File**, bool, FMOD::SoundI**) () from libfmodex64-4.44.42.so
No symbol table info available.
[#13](/issue/ValveSoftware/steam-for-linux/13) 0x00007f88a305daf0 in ?? () from libfmodevent64-4.44.42.so
No symbol table info available.
[#14](/issue/ValveSoftware/steam-for-linux/14) 0x00007f88a305ea10 in ?? () from libfmodevent64-4.44.42.so
No symbol table info available.
[#15](/issue/ValveSoftware/steam-for-linux/15) 0x00007f88a3422006 in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
[#16](/issue/ValveSoftware/steam-for-linux/16) 0x00007f88a34c6c0c in ?? () from libfmodex64-4.44.42.so
No symbol table info available.
[#17](/issue/ValveSoftware/steam-for-linux/17) 0x00007f88a285d5c2 in start_thread () from /usr/lib/libc.so.6
No symbol table info available.
[#18](/issue/ValveSoftware/steam-for-linux/18) 0x00007f88a28e2584 in clone () from /usr/lib/libc.so.6
No symbol table info available.

There still is this ominous 0x2a81 I have no idea about, and I also do not understand how Steam's game overlay lib changes anything in how FMOD behaves.
We are currently trying to find out what the differences between the music tracks that work and those wo don't are. So far there seems to be nothing of note, I will report if we find anything

There still is this ominous 0x2a81 I have no idea about, and I also do not understand how Steam's game overlay lib changes anything in how FMOD behaves.

@smcv some dynamic linker black magic happening maybe? With the overlay preloaded some symbol resolution ends up unfolding in a different way?

And there's also the fact that the stable steam client and runtime was still running this fine, and the steam beta (at the time, I think it shipped now) was crashing, on the same machine.

The value of 0x2a81 is just part of the instruction address, so no need for alignment on x64 (I think). The instruction at this address tries to store a value in something that looks like an array, but the address of that array turns out to be 0. The crash is understandable then, but I do not know why the address is wrong. Being a shared library I do not know how to set breakpoints at that specific instruction to see what the actual behavior is, nor can I follow what FMOD does exactly. We call their API, this spawns a thread which then crashes further down the line.
I'll contact FMOD to see if they can find out anything more, but I do not get my hopes up - the game runs fine without the overlay lib loaded and did with previous versions as well. I would be surprised if this is not something related to the overlay lib hooking some API and clashing with FMOD.

Update: I contacted FMOD support and they were actually able to help me with this issue. The problem is the stack overflowing when loading certain sound files inside FMOD's decompression thread. We have a fix for this, the patch is currently in review at our QA.
I cannot give a date when the patch goes live, but it will happen at some point. Until then the workaround starting the without the lib in LD_PRELOAD will have to do.