protonscr

static OpenSSL can cause crashes on Debian/Ubuntu

steamclosed Steam clientDistro Family: Debian
ValveSoftware/steam-for-linux#7223 · opened 2020-06-26 by TamasSzerb · updated 2022-08-26 · 40 comments · github
TTamasSzerb 2020-06-26 github

Your system information

  • Steam client version (build number or date): 1:1.0.0.62
  • Distribution (e.g. Ubuntu): Debian
  • Opted into Steam client beta?: [Yes/No] No
  • Have you checked for system updates?: [Yes/No] Yes

Please describe your issue in as much detail as possible:

Describe what you expected should happen and what did happen. Please link any large code pastes as a Github Gist

Steps for reproducing this issue:

  1. starting steam

toma@inky  ~  steam  ✔  23043  12:27:08
/home/toma/.steam/debian-installation/steam.sh: line 114: VERSION_ID: unbound variable
/home/toma/.steam/debian-installation/steam.sh: line 114: VERSION_ID: unbound variable
Running Steam on debian 64-bit
/home/toma/.steam/debian-installation/steam.sh: line 114: VERSION_ID: unbound variable
STEAM_RUNTIME is enabled automatically
Pins up-to-date!
Steam client's requirements are satisfied
/home/toma/.steam/debian-installation/ubuntu12_32/steam
[2020-06-26 12:27:16] Startup - updater built Jun 4 2020 05:50:42
Installing breakpad exception handler for appid(steam)/version(1591251555)
Looks like steam didn't shutdown cleanly, scheduling immediate update check
Installing breakpad exception handler for appid(steam)/version(1591251555)
[2020-06-26 12:27:16] Checking for update on startup
[2020-06-26 12:27:16] Checking for available updates...
[2020-06-26 12:27:16] Downloading manifest: client-download.steampowered.com/client/steam_client_ubuntu12
Installing breakpad exception handler for appid(steam)/version(1591251555)
[2020-06-26 12:27:17] Download skipped: /client/steam_client_ubuntu12 version 1591251555, installed version 1591251555
[2020-06-26 12:27:17] Nothing to do
[2020-06-26 12:27:17] Verifying installation...
[2020-06-26 12:27:17] Performing checksum verification of executable files
[2020-06-26 12:27:17] Verification complete
Loaded SDL version 2.0.13-5893924
Gtk-Message: 12:27:18.960: Failed to load module "gail"
Gtk-Message: 12:27:18.960: Failed to load module "atk-bridge"

(steam:15717): Gtk-WARNING **: 12:27:18.963: Unable to locate theme engine in module_path: "adwaita",
/usr/share/themes/Adwaita/gtk-2.0/main.rc:733: error: unexpected identifier 'direction', expected character '}'

(steam:15717): Gtk-WARNING **: 12:27:18.964: Unable to locate theme engine in module_path: "adwaita",
/usr/share/themes/Adwaita/gtk-2.0/hacks.rc:28: error: invalid string constant "normal_entry", expected valid string constant
Installing breakpad exception handler for appid(steam)/version(1591251555)
STEAM_RUNTIME_HEAVY: ./steam-runtime-heavy
[0626/122719.314953:INFO:crash_reporting.cc(247)] Crash reporting enabled for process: browser
[0626/122719.333359:WARNING:crash_reporting.cc(286)] Failed to set crash key: UserID with value: 0
[0626/122719.333431:WARNING:crash_reporting.cc(286)] Failed to set crash key: BuildID with value: 1591249846
[0626/122719.333440:WARNING:crash_reporting.cc(286)] Failed to set crash key: SteamUniverse with value: Public
[0626/122719.333449:WARNING:crash_reporting.cc(286)] Failed to set crash key: Vendor with value: Valve
/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so: undefined symbol: g_log_structured_standard
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so
GLib-GIO-Message: Using the 'memory' GSettings backend. Your settings will not be saved or shared with other applications.
[0626/122719.386232:WARNING:crash_reporting.cc(286)] Failed to set crash key: UserID with value: 0
[0626/122719.386291:WARNING:crash_reporting.cc(286)] Failed to set crash key: BuildID with value: 1591249846
[0626/122719.386297:WARNING:crash_reporting.cc(286)] Failed to set crash key: SteamUniverse with value: Public
[0626/122719.386315:WARNING:crash_reporting.cc(286)] Failed to set crash key: Vendor with value: Valve
[0626/122719.386810:INFO:crash_reporting.cc(247)] Crash reporting enabled for process: gpu-process
[0626/122719.491158:WARNING:crash_reporting.cc(286)] Failed to set crash key: UserID with value: 0
[0626/122719.491214:WARNING:crash_reporting.cc(286)] Failed to set crash key: BuildID with value: 1591249846
[0626/122719.491220:WARNING:crash_reporting.cc(286)] Failed to set crash key: SteamUniverse with value: Public
[0626/122719.491225:WARNING:crash_reporting.cc(286)] Failed to set crash key: Vendor with value: Valve
[0626/122719.491767:INFO:crash_reporting.cc(247)] Crash reporting enabled for process: utility
Installing breakpad exception handler for appid(steam)/version(1591251555)
Installing breakpad exception handler for appid(steam)/version(1591251555)
Installing breakpad exception handler for appid(steam)/version(1591251555)
Installing breakpad exception handler for appid(steam)/version(1591251555)
Installing breakpad exception handler for appid(steam)/version(1591251555)
Installing breakpad exception handler for appid(steam)/version(1591251555)
assert_20200626122716_1.dmp[15773]: Uploading dump (out-of-process)
/tmp/dumps/assert_20200626122716_1.dmp
/home/toma/.steam/debian-installation/steam.sh: line 750: 15717 Segmentation fault $STEAM_DEBUGGER "$STEAMROOT/$STEAMEXEPATH" "$@"
toma@inky  ~  assert_20200626122716_1.dmp[15773]: Finished uploading minidump (out-of-process): success = yes  ✔  23044  12:27:20
assert_20200626122716_1.dmp[15773]: response: CrashID=bp-44c308c2-2647-4ba7-b347-903172200626
assert_20200626122716_1.dmp[15773]: file ''/tmp/dumps/assert_20200626122716_1.dmp'', upload yes: ''CrashID=bp-44c308c2-2647-4ba7-b347-903172200626''

assert_20200626122716_1.dmp.zip

Kkisak-valve maintainer 2020-06-26 github

For reference, the attached minidump is a SIGSEGV in libcrypto.so.1.1.

TTamasSzerb 2020-06-26 github

I can confirm this by installing https://packages.debian.org/stretch/amd64/libssl1.0.2/download and starting steam by:

LD_PRELOAD=/usr/lib/i386-linux-gnu/libcrypto.so.1.0.2 steam

working fine. Thanks! How did you analyse the dump, @kisak-valve ?

Ffolknor 2020-06-26 github

This might - and might not - be related, but I upgraded these packages today on my ubuntu groovy machine

bsdmainutils (12.1.2) to 12.1.2build1
bsdutils (1:2.35.1-5ubuntu2) to 1:2.35.2-5ubuntu2
code-insiders (1.47.0-1592905677) to 1.47.0-1593150187
eject (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
fdisk (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
gir1.2-cogl-1.0 (1.22.6-1) to 1.22.8-1
gir1.2-coglpango-1.0 (1.22.6-1) to 1.22.8-1
google-chrome-unstable (85.0.4173.0-1) to 85.0.4181.8-1
libblkid-dev (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libblkid1 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libblkid1:i386 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libcogl-common (1.22.6-1) to 1.22.8-1
libcogl-pango20 (1.22.6-1) to 1.22.8-1
libcogl-path20 (1.22.6-1) to 1.22.8-1
libcogl20 (1.22.6-1) to 1.22.8-1
libefiboot1 (37-2ubuntu3) to 37-2ubuntu4
libefivar1 (37-2ubuntu3) to 37-2ubuntu4
libfdisk1 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libgpgme11 (1.13.1-7ubuntu2) to 1.13.1-8ubuntu1
libgpgmepp6 (1.13.1-7ubuntu2) to 1.13.1-8ubuntu1
libgssdp-1.2-0 (1.2.2-1) to 1.2.3-2
libgupnp-1.2-0 (1.2.2-2) to 1.2.3-1
libmount-dev (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libmount1 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libmount1:i386 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libsmartcols1 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libssl-dev (1.1.1f-1ubuntu2) to 1.1.1f-1ubuntu3
libssl-doc (1.1.1f-1ubuntu2) to 1.1.1f-1ubuntu3
libssl1.1 (1.1.1f-1ubuntu2) to 1.1.1f-1ubuntu3
libssl1.1:i386 (1.1.1f-1ubuntu2) to 1.1.1f-1ubuntu3
libuuid1 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libuuid1:i386 (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
libwww-perl (6.45-2) to 6.46-1
linux-base (4.5ubuntu3) to 4.5ubuntu4
linux-firmware (1.187) to 1.188
man-db (2.9.2-1) to 2.9.3-1
meson (0.54.3-1ubuntu1) to 0.54.3-1ubuntu2
mount (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
openssl (1.1.1f-1ubuntu2) to 1.1.1f-1ubuntu3
python3-gpg (1.13.1-7ubuntu2) to 1.13.1-8ubuntu1
rfkill (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
util-linux (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
uuid-dev (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2
uuid-runtime (2.35.1-5ubuntu2) to 2.35.2-5ubuntu2

and it causes steam to segfault

juni 26 16:47:30 plantasjen crash_20200626164726_1.dmp[2135]: Uploading dump (out-of-process)
                                                              /tmp/dumps/crash_20200626164726_1.dmp
juni 26 16:47:30 plantasjen kernel: show_signal_msg: 13 callbacks suppressed
juni 26 16:47:30 plantasjen kernel: CIPCServer::Thr[2122]: segfault at 6cd820e7 ip 00000000f798d6ed sp 00000000eb8fd940 error 4 in libc-2.31.so[f7923000+15b000]
juni 26 16:47:30 plantasjen kernel: Code: fb 57 56 e8 ae 05 0c 00 81 c6 35 39 16 00 53 83 ec 10 8b 5c 24 20 8b 86 4c ff ff ff 8b 00 85 c0 75 7b 85 db 0f 84 80 00 00 00 <8b>>
juni 26 16:47:31 plantasjen crash_20200626164726_1.dmp[2135]: Finished uploading minidump (out-of-process): success = yes
juni 26 16:47:31 plantasjen crash_20200626164726_1.dmp[2135]: response: CrashID=bp-7d8c8cec-0956-4856-b402-386bc2200626
juni 26 16:47:31 plantasjen crash_20200626164726_1.dmp[2135]: file ''/tmp/dumps/crash_20200626164726_1.dmp'', upload yes: ''CrashID=bp-7d8c8cec-0956-4856-b402-386bc2200626''

Downgrading the packages again and steam works fine even without rebooting. So it's probably the libssl upgrade that did it, since that includes /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1.

Ssmcv 2020-06-26 github

Let me guess, this is Debian testing/unstable? (If so, please specify that in bug reports - the version matters, and I don't think this crash will happen in Debian 10 or older.)

I think what's happening here is a conflict between the mystery version of OpenSSL that is statically linked into the Steam binary (perhaps 1.0.x?), and the shared-library version that is pulled in via dependencies.

The context here is that until recently, none of the libraries that Steam uses depended on OpenSSL (libcrypto/libssl), so the only copy of OpenSSL in the process was the one that is statically linked into the Steam binary.

However, Steam depends on GTK and GLib, and recent versions of GLib depend on libmount. The version of libmount in Debian unstable recently (as in this week) added a dependency on libcryptsetup, to enable a dm-verity feature - but libcryptsetup depends on OpenSSL, so we now have two copies of OpenSSL in the process space, the one in Steam and the one from Debian. They are incompatible, but export symbols with the same names; some users of those symbols will not get the one they were expecting, so they'll crash.

With dynamically-linked OpenSSL, versioned symbols would save us from this, but static linking doesn't have versioned symbols.

If my theory is correct, some things that might help to solve this in the Steam client:

  • link to OpenSSL dynamically, and make sure the bundled copies of libcrypto*.so.* and libssl*.so.* have versioned symbols
  • link to OpenSSL statically, but make sure any symbols that are re-exported by parts of the Steam client get versioned symbols
  • link to OpenSSL statically, but with -Wl,-Bsymbolic
Ssmcv 2020-06-26 github

So it's probably the libssl upgrade that did it

No, I think it's the libmount1 upgrade.

Ssmcv 2020-06-26 github

One workaround for this that seems to work for some people is: don't have libglib2.0-0:i386 installed (because that's what pulls libmount1 into Steam's address space; if it isn't installed system-wide, the much older version in the Steam Runtime will be used; and that's too old to have the libmount1 dependency).

However, that might not be an option if you use recent versions of Wine, which depend on GLib via GStreamer, among others.

Ssmcv 2020-06-26 github

presumably using a built-in copy of GTK

It's bundled as part of the Steam Runtime in ~/.steam/steam/ubuntu12_32/steam-runtime.

If you try this workaround, please carefully check what apt is going to remove before pressing "y".

Yes, always do this! :-)

Bbellini666 2020-06-26 github

I'm on debian testing/unstable and can confirm that removing libglib2.0-0:i386 as suggested by @smcv works! Just, as mentioned before, be sure that you don't need any of the packages that depends on it.

Tttelford 2020-06-26 github

However, that might not be an option if you use recent versions of Wine, which depend on GLib via GStreamer, among others.

Yes... if only I had the time to find docs on how I can just use Proton instead of Wine for some apps... Ah well, I imagine it won't hurt to uninstall Wine for a while.

Ddpanter 2020-06-26 github

Ran into this problem today on Debian sid.
Downgrading libmount1 from sid 2.35.2-6 to testing 2.35.2-4 solves the issue.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963525

Iimaami 2020-06-26 github

Ran into this problem today on Debian sid.
Downgrading libmount1 from sid 2.35.2-6 to testing 2.35.2-4 solves the issue.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=963525

This worked for me.

PPlagman 2020-06-26 github

Which Steam module exports OpenSSL symbols? It probably just shouldn't do that at all.

Ddebianmain1 2020-06-27 github

One workaround for this that seems to work for some people is: don't have libglib2.0-0:i386 installed (because that's what pulls libmount1 into Steam's address space; if it isn't installed system-wide, the much older version in the Steam Runtime will be used; and that's too old to have the libmount1 dependency).

However, that might not be an option if you use recent versions of Wine, which depend on GLib via GStreamer, among others.

Verified that this works with my Sid install.....Thanks!

Ddpanter 2020-06-27 github

One workaround ... don't have libglib2.0-0:i386 installed ...

However, that might not be an option if you use recent versions of Wine...

Confirmed. I use Wine staging, removing libglib2.0-0:i386 will also nuke Wine.

Mmlassnig 2020-06-27 github

One workaround for this that seems to work for some people is: don't have libglib2.0-0:i386 installed (because that's what pulls libmount1 into Steam's address space; if it isn't installed system-wide, the much older version in the Steam Runtime will be used; and that's too old to have the libmount1 dependency).
However, that might not be an option if you use recent versions of Wine, which depend on GLib via GStreamer, among others.

Verified that this works with my Sid install.....Thanks!

Confirmed for my Sid install as well.

Ssmcv 2020-06-27 github

Which Steam module exports OpenSSL symbols? It probably just shouldn't do that at all.

@Plagman: If none of them re-export symbols, then my suggestion to "make sure any symbols that are re-exported by parts of the Steam client get versioned symbols" becomes irrelevant, and the other two suggestions are (mutually exclusive) possible ways to resolve this. Loading all Steam's modules with RTLD_LOCAL is another thing that might help.

I think the problem here is that unlike Windows libraries (which form a tree), ELF binaries normally form a flat global namespace, and ELF toolchains support symbol interposition. That's the same mechanism that lets you LD_PRELOAD a module that intercepts calls to a function and substitutes its own implementation of that function. If Steam dlopens a module that loads the dependency chain GTK -> GLib -> libmount -> libcryptsetup -> (Debian's) OpenSSL, particularly if it uses RTLD_GLOBAL, then that might have almost the same effect as an LD_PRELOAD - symbols of the same name from Debian's OpenSSL will be interposed into calls that were intended to go from Steam into its statically linked copy of OpenSSL. Because Debian's OpenSSL appears to be a newer version with a different ABI (or possibly an older version with a different ABI, it could go either way), that ends badly.

Explaining the possible solutions that I suggested in more detail:

  • Linking -Wl,-Bsymbolic changes internal references within the module being compiled/linked (including statically linked libraries, I think?) so that they don't support interposition, which might solve this (and is also meant to make them a bit more efficient).

  • Dynamically linking Steam's OpenSSL, and making sure it had versioned symbols, would mean that Steam's OpenSSL 1.0.2 (or whatever version it really is) was looking for AES_decrypt@OPENSSL_1_0_2 or similar, whereas Debian's OpenSSL only provides AES_decrypt@OPENSSL_1_1_0, and they can coexist within one process space. (I don't know which symbols Steam actually uses, AES_decrypt() is just a random example.)

  • Loading the module that depends on GTK with RTLD_LOCAL means that "Symbols defined in this shared object are not made available to resolve references in subsequently loaded shared objects" (see dlopen(3)) which is a bit more like the Windows linker behaviour: each shared object would be (at least to some extent) own little world. If the GTK dependency and the OpenSSL dependency are in the same module, this wouldn't help at all, but if they're in different modules, it might insulate them from each other.

Ssherpya 2020-06-27 github

the simplest workaround I've found:

Ssmcv 2020-06-27 github

@sherpya: Downloading an older version of libmount into the Steam Runtime is at best a temporary workaround. As soon as GLib starts using a symbol from newer libmount versions, your older version of libmount will become actively harmful (and will potentially break Steam).

If you want to work around this by using an older libmount, install the version of libmount1 from testing (2.35.2-4) and put it on apt hold.

Tterroreek 2020-06-27 github

I followed @sherpya instructions, but I used libmount1_2.35-4_i386.deb found in my apt cache, /var/cache/apt/archives, but my steam runtime was found ~/.steam/debian-installation/ubuntu12_32

PPlagman 2020-06-28 github

If there are no symbols accidentally exposed from statically linking OpenSSL code (dynamic or otherwise), then neither LD_PRELOAD nor any sort of dynamic loading of other libraries should have the potential to cause the statically linked copy of that code to pick up external code, and dynamically loaded copies of OpenSSL, or code that depends on it, should not be able to be aware of our statically linked copy.

Ssmcv 2020-06-28 github

dynamically loaded copies of OpenSSL, or code that depends on it, should not be able to be aware of our statically linked copy

@Plagman: That's a reasonable thing to expect, but on ELF platforms like Linux it's actually only true if you link with -Wl,-Bsymbolic (or -Wl,-Bsymbolic-functions, which is like -Wl,-Bsymbolic for function calls but keeps the default behaviour for extern global variables).

This is the default partly for historical reasons, and partly because it's how interposition works: if you use something like electric-fence and define your own versions of malloc() and free() in an LD_PRELOAD module, they need to be used all the time, even when other glibc functions call malloc() and free() internally. However, it's a problem when you want to have internal implementation details that stay internal, like you do with Steam's statically linked bits of OpenSSL.

You can see this happening with DEBUGGER="env LD_DEBUG=all" steam (warning: this produces a lot of output, 1.4G on my Debian unstable system). Search the output for binding file.*ubuntu12_32\/steam.*libcrypto and you'll see things like this:

     25567:     file=/home/player/.steam/debian-installation/ubuntu12_32/steamui.so [0];  needed by /home/player/.steam/debian-installation/ubuntu12_32/libvideo.so [0] (relocation dependency)
...
     25567:     symbol=SHA1_Init;  lookup in file=/home/player/.steam/debian-installation/ubuntu12_32/steam [0]
... it tries lots more libraries ...
     25567:     symbol=SHA1_Init;  lookup in file=/usr/lib/i386-linux-gnu/libcrypto.so.1.1 [0]
     25567:     binding file /home/player/.steam/debian-installation/ubuntu12_32/steamui.so [0] to /usr/lib/i386-linux-gnu/libcrypto.so.1.1 [0]: normal symbol `SHA1_Init'

What is happening in this bit of the log is that steamui.so contains a reference (i.e. a call) to SHA1_Init, the dynamic linker is looking for it in various places, and the dynamic linker eventually finds it in libcrypto.so.1.1, and the binding file line says it resolves the reference by using libcrypto.so.1.1's copy of SHA1_Init() for calls from steamui.so.

Looking at the LD_DEBUG=all output more closely, this also seems to be happening the other way around, which I had not expected: as well as Steam looking for OpenSSL symbols and finding the system OpenSSL instead of its own, there seem to be some instances of the system OpenSSL looking for its own symbols and finding them (possibly with an incompatible ABI) in Steam's statically linked copy. I'm not sure why this can happen, because OpenSSL should be looking for the function with a versioned name and I've confirmed with objdump -Tx that Steam's copy isn't decorated with a version, but somehow it seems to find Steam's version anyway. If you search for binding file.*libcrypto\.so\.1\.1.*ubuntu12_32\/steam the results include things like this:

     25567:     relocation processing: /usr/lib/i386-linux-gnu/libcrypto.so.1.1
     25567:     symbol=ACCESS_DESCRIPTION_it;  lookup in file=/home/player/.steam/debian-installation/ubuntu12_32/steam [0]
     25567:     binding file /usr/lib/i386-linux-gnu/libcrypto.so.1.1 [0] to /home/player/.steam/debian-installation/ubuntu12_32/steam [0]: normal symbol `ACCESS_DESCRIPTION_it' [OPENSSL_1_1_0]

The symbol seen here, ACCESS_DESCRIPTION_it, seems to be something internal rather than part of the ABI. What is happening in this bit of the log is that libcrypto.so.1.1 contains a reference to ACCESS_DESCRIPTION_it, the dynamic linker finds that symbol in the main Steam executable, and the binding file line says it resolves the reference by using the copy of ACCESS_DESCRIPTION_it in the Steam executable.

Linking Steam modules with -fvisibility=hidden, and explicitly exporting only the symbols that need exporting to other parts of Steam with __attribute__((visibility("default"))), is another thing that might help here.

Ssmcv 2020-06-28 github

Inspecting steamui.so with objdump -Tx, its reference to SHA1_Init is an undefined symbol with no version:

00000000       F *UND*  00000000              SHA1_Init
...
00000000      DF *UND*  00000000              SHA1_Init

so it will be resolved by finding any implementation of SHA1_Init(), regardless of version. (There's nothing particularly special about SHA1_Init() here, it's just the first one I found - the backtrace found on the Debian bug seems to involve EC crypto, which presumably uses a more complex data structure that has had incompatible changes between OpenSSL versions.)

If Steam is going to embed a copy of OpenSSL, then it needs to either be in a normal OpenSSL shared library, or statically linked into a larger shared library (perhaps a new libsteamshared.so?) that is DT_NEEDED by all the Steam modules that need it. Having undefined references to OpenSSL functions in loadable modules, which expect to get their undefined references resolved by a statically-linked OpenSSL in the main executable, is not going to work reliably.

MMte90 2020-06-28 github

the simplest workaround I've found:

* download http://ftp.it.debian.org/debian/pool/main/u/util-linux/libmount1_2.33.1-0.1_i386.deb

* extract it with `dpkg -X libmount1_2.33.1-0.1_i386.deb out`

* copy `out/lib/i386-linux-gnu/libmount.so.1.1.0` to `~/.local/share/Steam/ubuntu12_32/libmount.so.1 (not 1.1.0)`

In my case I had to to copy the file in a different path (debian sid): cp out/lib/i386-linux-gnu/libmount.so.1.1.0 ~/.steam/ubuntu12_32/libmount.so.1

Ssmcv 2020-06-28 github

@Mte90, @terroreek, @sherpya: Please do not work around this issue by copying random versions of libmount into the ubuntu12_32 or steam-runtime directory. This will cause binary compatibility problems for you later, when you have forgotten that you have modified Steam in this way.

Some other options for workarounds include:

  • Temporarily downgrade the libmount1 package to a version that does not depend on libcryptsetup12, and put it on hold in apt, until this issue has been resolved in some other way
    • If you don't know how to do this, reconsider whether a development distribution like Debian testing/unstable or Ubuntu devel is the right choice for you; you might get better results with a stable release like Debian 10 or Ubuntu 20.04, which do not trigger this issue
  • Temporarily remove libglib2.0-0:i386, if you do not need any of the packages that depend on it
  • Install Steam via Flatpak
Ssherpya 2020-06-28 github

@smcv I'll remove the stream directory and recreate as I did if I get problems

Ssmcv 2020-06-29 github

This seems to be basically the same thing as #6861, but for Debian derivatives rather than OpenMandriva.

Ssmcv 2020-07-02 github

libglib2.0-0 version 2.64.3-2 in Debian works around this by disabling its use of libmount and using a fallback code path.

Tterroreek 2020-07-03 github

I can confirm the newest version works after downgrading and holding the an older libmount as per @smcv suggestion the work around was not the right answer. After unholding and upgrading, steam works great.

Ssherpya 2020-07-06 github

I've removed old libmount workaround, with latest pins and latest packages from debian unstable the problem is fixed

Eedomato 2020-07-06 github

I've removed old libmount workaround, with latest pins and latest packages from debian unstable the problem is fixed

This is correct since today, 6 of July, libmount1 has a new version that doesn't pull libcryptsetup12 package which generates conflict with other packages on Debian too.

You can see libmount1 changelog (https://tracker.debian.org/news/1158200/accepted-util-linux-2352-7-source-into-unstable/) for more information.

Anyway, I think that this issue shouldn't be closed since the main problem (Steam Client statically linked with an older or modified version of libssl) should have a long term solution since, at some point, the same package or any other package could pull newer version of libssl library and the problem is going to appear again.

TTTimo 2020-07-21 github

Changes made in Steam client beta >= Jul 20 should address this

Ddpanter 2020-07-28 github

Debian sid has updated libmount1/util-linux a few times since the intial bug report, the problem has been worked around and Steam works since July 6.
Retesting Steam now with older versions of these packages isn't a trivial task if you have an updated system. Both sid and testing are now on the same versions, and stable is way behind.

Kkisak-valve maintainer 2020-07-29 github

Hello @mikaelhg, you issue is unrelated to the issue reported here. Odds are your system has a broken 32 bit OpenGL render stack, most likely from missing nVidia libraries while using the nVidia proprietary driver. I'm going to take a blind guess that you're using nVidia 440.100, and do not have libnvidia-gl-440:i386 installed. The nVidia proprietary driver requires that you have the exact same version for the running kernel module, 64 bit userspace, and 32 bit userspace to work as intended with Steam.

If this guess is right, something like sudo apt install libnvidia-gl-440:i386 should fix your issue. Please verify the guess before blindly installing the package.

Mmikaelhg 2020-07-29 github

@kisak-valve, yes, I discovered that independently, and deleted the comment, but not before troubling you. I installed the NVidia CUDA 450 driver stack from their repo, which doesn't include i386 packages. Just now installing the graphics-drivers PPA to get the 450 drivers from a source which includes the i386 packages. That will likely fix the issue I was having, but which wasn't relevant to OP's report. Edit: Steam + NVidia 450 drivers + CUDA 11 all work together through the graphics-drivers PPA installation.

BROKEN with Steam:

https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/

WORKING with Steam:

sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt-get install libnvidia-gl-450:i386
Ssmcv 2020-07-29 github

Retesting Steam now with older versions of these packages isn't a trivial task if you have an updated system

It's possible to do by upgrading from Debian 10 using snapshot.debian.org on a test machine, to get the state that testing or unstable would have been in on a suitable date in the past (2020-07-07 looks like it would reproduce this bug). I need a similar version for some other testing anyway, so I'll see whether I can reproduce the bug like that and confirm whether the beta client fixes it.

Eedomato 2020-07-29 github

Retesting Steam now with older versions of these packages isn't a trivial task if you have an updated system

It's possible to do by upgrading from Debian 10 using snapshot.debian.org on a test machine, to get the state that testing or unstable would have been in on a suitable date in the past (2020-07-07 looks like it would reproduce this bug). I need a similar version for some other testing anyway, so I'll see whether I can reproduce the bug like that and confirm whether the beta client fixes it.

I was going to say something similar but didn't have the time to do it :disappointed: , using snapshot.d.o as a time machine is great for this kind of things.

Anyway, we should be able to reproduce the bug with the older version of the Steam Client which I don't know if that's possible at all to be sure that the new one fixes the problem.

Just my two cents :grin:

Ssmcv 2020-07-29 github

@TTimo: sorry, the changes in beta build 1595990357 do not seem to be sufficient to address this.

Because it has been worked around in libmount, this issue shouldn't be a practical problem in Debian testing/unstable or Ubuntu 20.04 right now. However, as @edomato said, similar crashes might come back in future if some other library required by Steam starts to pull in OpenSSL.

I would suggest retitling this issue report to "static OpenSSL can cause crashes on Debian/Ubuntu" or something like that, to avoid attracting follow-up comments from people who get the same high-level symptom (Steam does not start) for an unrelated reason.

Steps to reproduce this:

  • Opt out from the Steam client beta, if you are using it
  • Have Debian 10 or an outdated (pre-July) Debian testing/unstable installation (I started from Debian 10 with GNOME)
  • Use snapshot.debian.org to upgrade to the state it was in as of 2020-07-01 (some retries will be needed, because snapshot.debian.org has rate-limiting and apt doesn't support rate-limiting its downloads)
  • Use dpkg-query -W to make sure you have:
    • libglib2.0-0:i386 version 2.64.3-1 or older (version 2.64.3-2 has a workaround for this issue and cannot be used to reproduce it)
    • libmount1:i386 version 2.35.2-6 exactly (newer versions have a workaround for this issue and cannot be used to reproduce it)
  • Run steam (I used the .deb from repo.steampowered.com for this test)

(For what it's worth, I actually got the date wrong and upgraded to the state Debian was in as of 2020-07-07, and had to add a 2020-07-01 apt source and downgrade GLib from 2.64.3-2 to 2.64.3-1 to reproduce the crash.)

With the GA (non-beta) build 1595977781 (2020-07-28) I get a crash, with this backtrace visible in coredumpctl gdb:

#0  0xf2bcb2e3 in EC_KEY_new_by_curve_name () from /usr/lib/i386-linux-gnu/libcrypto.so.1.1
#1  0xeb6e6635 in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#2  0xeb6e6b2a in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#3  0xeb674276 in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#4  0xead58cd8 in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#5  0xead7412a in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#6  0xeaec8318 in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#7  0xeaec9936 in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/steamclient.so
#8  0xf2a732c0 in SteamThreadTools::CThread::ThreadExceptionWrapper(void*) ()
   from /home/desktop/.local/share/Steam-alt/ubuntu12_32/libtier0_s.so
#9  0xf2a7136b in ?? () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/libtier0_s.so
[#10](/issue/ValveSoftware/steam-for-linux/10) 0xf2a7166f in CatchAndWriteMiniDumpExForVoidPtrFn () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/libtier0_s.so
[#11](/issue/ValveSoftware/steam-for-linux/11) 0xf2a716c1 in CatchAndWriteMiniDumpForVoidPtrFn () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/libtier0_s.so
[#12](/issue/ValveSoftware/steam-for-linux/12) 0xf2a75ffc in SteamThreadTools::CThread::ThreadProc(void*) () from /home/desktop/.local/share/Steam-alt/ubuntu12_32/libtier0_s.so
[#13](/issue/ValveSoftware/steam-for-linux/13) 0xf7be9082 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
[#14](/issue/ValveSoftware/steam-for-linux/14) 0xf79faf86 in clone () from /lib/i386-linux-gnu/libc.so.6

Workaround: by installing libssl1.0.2 from Debian 9 'stretch', I was able to run LD_PRELOAD=/usr/${LIB}/libcrypto.so.1.0.2 steam and it was successful enough to be able to opt-in to the beta.

Next I tried opting in to the beta, exiting, and re-running Steam.

With beta build 1595990357 (2020-07-29) and without the LD_PRELOAD workaround, Steam does not crash immediately, but it fails to log in to the Steam network and suggests starting in offline mode, with warnings that look SSL-related. This suggests that some of the Steam libraries are still picking up OpenSSL symbols from the OS copy.

/data/src/common/opensslconnection.cpp (2276) : Assertion Failed: COpenSSLConnection(c0xe704a800) made no progress: SSLv2/v3 write client hello B (0 in 218 out)
/data/src/common/opensslconnection.cpp (2276) : Assertion Failed: COpenSSLConnection(c0xe704a800) made no progress: SSLv2/v3 write client hello B (0 in 218 out)
Installing breakpad exception handler for appid(steam)/version(1595990357)
Opted-in Controller Mask for AppId 0: 0
Installing breakpad exception handler for appid(steam)/version(1595990357)
crash_20200729153253_21.dmp[31241]: Uploading dump (out-of-process)
/tmp/dumps/crash_20200729153253_21.dmp
/data/src/common/opensslconnection.cpp (2276) : Assertion Failed: COpenSSLConnection(c0xe704a800) made no progress: SSLv2/v3 write client hello B (3746 in 0 out)
/data/src/common/opensslconnection.cpp (2276) : Assertion Failed: COpenSSLConnection(c0xe704a800) made no progress: SSLv2/v3 write client hello B (3746 in 0 out)
/data/src/common/opensslconnection.cpp (2301) : Assertion Failed: bForceRequeue || m_unThreadBytesToRead == 0 || m_eResultRunSSL != k_EResultOK
/data/src/common/opensslconnection.cpp (2301) : Assertion Failed: bForceRequeue || m_unThreadBytesToRead == 0 || m_eResultRunSSL != k_EResultOK

It also crashes on exit:

#0  0xf2c05ea3 in ?? () from /usr/lib/i386-linux-gnu/libcrypto.so.1.1
#1  0xf2bdaf92 in OPENSSL_cleanup () from /usr/lib/i386-linux-gnu/libcrypto.so.1.1
#2  0xf78df20e in ?? () from /lib/i386-linux-gnu/libc.so.6
#3  0xf78df3f1 in exit () from /lib/i386-linux-gnu/libc.so.6
#4  0xf78c6efd in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6
#5  0x5665e5d9 in _start ()

In the beta build, steamui.so no longer imports symbols like EC_KEY_new_by_curve_name from other Steam modules like steamclient.so: it now has its own copy (presumably part of OpenSSL was statically linked into it). However, these OpenSSL-derived symbols are exported dynamic symbols with no symbol-version specified, so the dynamic linker ld.so is allowed to resolve references to that symbol by using the copy from libcrypto.so.1.1 instead, and that can cause crashes if incompatible data structures end up being used.

For this to be robust, I think it would be necessary for the dynamic symbol tables of steam, steamui.so, steamclient.so, etc. (as shown by (objdump -T) to not mention any OpenSSL-related symbols. Or, it might be sufficient to make sure all the symbols are versioned, for example by linking all Steam libraries with the -Wl,--default-symver linker argument.

MMte90 2022-08-26 github

it is still a bug on steam or can be closed?

TTamasSzerb 2022-08-26 github

can be closed, thanks

VWOL
Tamas SZERB @.***>

On Fri, Aug 26, 2022 at 3:18 PM Daniele Scasciafratte <
@.***> wrote:

it is still a bug on steam or can be closed?


Reply to this email directly, view it on GitHub
https://github.com/ValveSoftware/steam-for-linux/issues/7223#issuecomment-1228477767,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACILVUNAGPGE3DST64ALO3V3C725ANCNFSM4OJGOG5A
.
You are receiving this because you authored the thread.Message ID:
@.***>

Kkisak-valve maintainer 2022-08-26 github

Closing per the last couple comments.