It seems that there are actually multiple versions of libstdc++6 on Tumbleweed and effectively, if I replace libstdc++6 (v. 13.0.0+git197351-1.1) with libstdc++6-gcc12 (v. 12.2.1+git537-1.2 with all dependencies), the issue disappear.

I reported also on openSUSE issue tracker https://bugzilla.opensuse.org/show_bug.cgi?id=1206503

Thanks.

For me the issue is gone on latest Steam Beta Update, it just happens on Stable Steam update.

I can confirm we already shipped libstdc++ from GCC 13 into Tumbleweed and that's why we see the crash.

Anyway, the issues started with https://github.com/gcc-mirror/gcc/commit/b3ac43a3c05744d62a963d656bed782fc867ad79.
@jwakely: Is the revision supposed to introduce an ABI change and thus we face the aforementioned issue where _M_data == nullptr?

Is the revision supposed to introduce an ABI change

No, and I don't see how it can introduce one. It only changes how we access values that are already present (using static_cast instead of dynamic_cast for facets guaranteed to be present, so the RTTI check is unnecessary). It shouldn't change what is accessed (or whether it's null), only how it's accessed. Are you 100% confident in the bisection result?

and thus we face the aforementioned issue where _M_data == nullptr?

Are you sure it's null, and not just garbage? The stack trace above doesn't actually show the value of _M_data where it faults. Edit: it's null

There's not enough info here to debug anything. How is the binary linked to libstdc++? Dynamically? Statically? If dynamically, is the new libstdc++.so.6 being used at runtime?

I need a reproducer really.

I need a reproducer really.

OK, I can reproduce a crash on Fedora using LD_LIBRARY_PATH=$HOME/gcc/13/lib steam so I should be able to debug this now.

But LD_LIBRARY_PATH=$HOME/gcc/13/lib DEBUGGER=gdb steam doesn't crash :confused:

Doh, that's because my client updated itself to the latest beta. Reverting to stable gets me back to crashing, including under gdb. I probably can't debug this for a few days though, I have a higher priority libstdc++ ABI issue to fix first.

Why is std::num_put even calling a member of std::moneypunct ?! Something is corrupted on the stack here.

(gdb) p this
$3 = (const std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > > * const) 0xede36894 <(anonymous namespace)::moneypunct_cf>
(gdb) info vtbl this
vtable for 'std::num_put<char, std::ostreambuf_iterator<char, std::char_traits<char> > >' @ 0xede31888 (subobject @ 0xede36894):
[0]: 0xedcb9ae0 <std::__cxx11::moneypunct<char, false>::~moneypunct()>
[1]: 0xedcb9be0 <std::__cxx11::moneypunct<char, false>::~moneypunct()>
[2]: 0xedce8530 <std::__cxx11::moneypunct<char, false>::do_decimal_point() const>
[3]: 0xedce8540 <std::__cxx11::moneypunct<char, false>::do_thousands_sep() const>
[4]: 0xedce8c40 <std::__cxx11::moneypunct<char, false>::do_grouping() const>
[5]: 0xedce8f40 <std::__cxx11::moneypunct<char, false>::do_curr_symbol() const>
[6]: 0xedce8fa0 <std::__cxx11::moneypunct<char, false>::do_positive_sign() const>
[7]: 0xedce9000 <std::__cxx11::moneypunct<char, false>::do_negative_sign() const>
[8]: 0xedce8550 <std::__cxx11::moneypunct<char, false>::do_frac_digits() const>
[9]: 0xedce8560 <std::__cxx11::moneypunct<char, false>::do_pos_format() const>

it's called here:
│     2398        iter_type                                                                                                                                                                                                                                                    │
│     2399        put(iter_type __s, ios_base& __io, char_type __fill, long __v) const                                                                                                                                                                                         │
│  >  2400        { return this->do_put(__s, __io, __fill, __v); }                                                                                                                                                                                                             │

Are you 100% confident in the bisection result?

Yes, I'm pretty sure.

I wonder if we're seeing https://gcc.gnu.org/bugzilla/show_bug.cgi?id=91057 here. It looks like some of the steam code is statically linked to libstdc++, maybe an older version that doesn't have that fix.

If I change the static_cast in std::__try_use_facet to be dynamic_cast then the segfault doesn't happen, but the dynamic_cast fails and returns a null pointer. This means that when an iostream object is constructed, it gets std::ios::_M_num_put == nullptr and writing numeric types to such a stream will throw std::bad_cast exceptions. This is what was happening with GCC 12 and previous GCC versions.

With GCC 13 the corrupted facet IDs result in a crash when the stream is constructed, instead of unusable streams. I think the bug with corrupted facet IDs was always present in steam, but the new GCC code is unable to cope with that corruption and crashes. IIUC this is only a problem when a multithreaded application is statically linked to the libstdc++.a from GCC 9 or older, but there's no way to detect that and make the new code more resilient. To support that case I think we need to revert the very significant performance improvements from https://github.com/gcc-mirror/gcc/commit/b3ac43a3c05744d62a963d656bed782fc867ad79 :cry:

Switch to Big Picture mode

To be clear, is this the old Big Picture (the blue one, as shown in https://help.steampowered.com/en/faqs/view/3725-76D3-3F31-FB63), or the new Big Picture (basically the Steam Deck UI, as shown in https://store.steampowered.com/news/app/593110/view/3394051164709183116)?

In the beta client, I believe both are available as steam -bigpicture -oldbigpicture and steam -bigpicture -newbigpicture respectively (and steam -gamepadui is the same as -bigpicture -newbigpicture).

Switch to Big Picture mode

To be clear, is this the old Big Picture (the blue one, as shown in https://help.steampowered.com/en/faqs/view/3725-76D3-3F31-FB63), or the new Big Picture (basically the Steam Deck UI, as shown in https://store.steampowered.com/news/app/593110/view/3394051164709183116)?

In the beta client, I believe both are available as steam -bigpicture -oldbigpicture and steam -bigpicture -newbigpicture respectively (and steam -gamepadui is the same as -bigpicture -newbigpicture).

The old one.

IIUC this is only a problem when a multithreaded application is statically linked to the libstdc++.a from GCC 9 or older

Parts of Steam are linked to the libstdc++.a from gcc 9 (as provided by the Steam Runtime). This is not something that is avoidable any time soon. Steam can't rely on a dynamically-linked libstdc++ from the OS, because it still supports being run on OSs like Debian 10 (and openSUSE 15.4, I think) where the OS libstdc++ version is older than ours. It also can't ship its own dynamically-linked libstdc++, because that would break the dependencies of the graphics stack on OSs like Debian 11 (and openSUSE Tumbleweed) where the OS libstdc++ version is newer.

If there's a fix that can be backported into the Steam Runtime's gcc 9, or a symbol that can be removed from the dynamic symbol table as a workaround, then those might be good short-term solutions.

For me the issue is gone on latest Steam Beta Update, it just happens on Stable Steam update.

Is this still true for people who are seeing this? I think the beta has some linker-script fixes so that fewer symbols originating from the statically-linked libstdc++ will "leak" outside each module, so it would not be surprising if it avoided this crash.

If you're using Steam on a development distribution like openSUSE Tumbleweed or Debian unstable, consider using the beta, which usually reflects how the stable version will behave in a few weeks' or months' time. That would let you provide early feedback on new crashes and other issues before they hit stable, and then get the fixes for those issues as soon as they become available. It would also give you the ability to fall back to stable if there is a regression in the beta, whereas if a regression reaches the stable release, there's nowhere further to fall back to.

the old Big Picture

In a version that is affected, does the new Steam-Deck-style Big Picture (steam -bigpicture -newbigpicture or steam -gamepadui) have the same failure mode?

I don't think the old blue Big Picture (which seems to be called "panorama" internally) is necessarily going to remain available forever, so if this issue is specific to old Big Picture, it might solve itself eventually.

In a version that is affected, does the new Steam-Deck-style Big Picture (steam -bigpicture -newbigpicture

This one also crashes.

or steam -gamepadui) have the same failure mode?

While this is fine!

If there's a fix that can be backported into the Steam Runtime's gcc 9

Yes, that is possible, and I think that would be a good fix to backport (even if I end up reverting the dynamic_cast -> static_cast change in GCC 13). I can prepare that backport and give you a patch.

In a version that is affected, does the new Steam-Deck-style Big Picture (steam -bigpicture -newbigpicture

This one also crashes.

or steam -gamepadui) have the same failure mode?

While this is fine!

This confuses me: I would have expected those two to be equivalent. But perhaps the stable release is too old to understand the -newbigpicture argument?

This confuses me: I would have expected those two to be equivalent. But perhaps the stable release is too old to understand the -newbigpicture argument?

Dunno. I'm using 1.0.0.75-2.1 and if I pass an unknown argument there's no warning displayed. So hard to guess if the argument is supported or not.

If there's a fix that can be backported into the Steam Runtime's gcc 9

Yes, that is possible, and I think that would be a good fix to backport (even if I end up reverting the dynamic_cast -> static_cast change in GCC 13). I can prepare that backport and give you a patch.

Here is the backport to the releases/gcc-9 branch:
https://github.com/jwakely/gcc/commit/11da50077c91b6dec1793b9c4d6ccfdcf6b8ea32

Or as a pull req comparison thingy:
https://github.com/jwakely/gcc/compare/releases/gcc-9...jwakely:gcc:gcc-9-steam-facet-id-backport?expand=1

The _GLIBCXX_LONG_DOUBLE_COMPAT stuff is only needed for powerpc64 (and legacy targets like DEC alpha) so you could rip that out for x86 if you prefer.

Thanks, I'll look into whether we can get that into the Steam Runtime.

@TTimo, FYI, relevant branches here would be scout, heavy and soldier (all of which have gcc older than 9 as default, and a non-default backport of gcc 9 which has been forced to always link libstdc++ statically). I think everything in Steam is compiled with the g++-9 from scout, except for steamwebhelper which uses heavy. For fixes in our libstdc++ to take effect, potentially all C++ code in Steam would need recompiling, so we'd need to choose a suitable time in Steam's release cycle to make that happen.

Here is the backport to the releases/gcc-9 branch: jwakely/gcc@11da500

@jwakely: This is a backport of both cc386cf2 (r276762) and the subsequent bug fix 9cfc400f (r276840), am I correct?

Yes, that's correct, I combined them into one commit.

The _GLIBCXX_LONG_DOUBLE_COMPAT stuff is only needed for powerpc64 (and legacy targets like DEC alpha) so you could rip that out for x86 if you prefer.

I'd prefer to use the most straightforward possible backport of the same code that's in gcc 10, even if some of it isn't relevant on x86 - that seems lower-risk. It seems that what you provided is exactly that backport (but with the two changes squashed into one commit), so that's ideal.

Today's beta SDK for soldier (version 0.20230117.0) includes the backported change, but this will not yet have any practical effect for Steam users.

A future release of Steam will hopefully be recompiled with a g++-9 that includes the same backported change.

Yesterday's Steam beta 1677103459 (2023-02-22) is the first to ship with scout and heavy Steam Runtime SDKs that include the backported change in their g++-9. I can't confirm whether Steam was recompiled with this version of g++-9, but it hopefully was, which should fix the segfault for users of the Steam client beta branch.

This change should get into the general-availability branch of Steam with the next big update.

This change should get into the general-availability branch of Steam with the next big update.

I believe this happened a while ago.