Steam overlay helper (gameoverlayui) stops accepting connections; Proton games freeze permanently on alt-tab (KDE Plasma Wayland)

I reproduced the same full-listener-queue hang in native Linux Dota 2 (Vulkan) and investigated it with strace, GDB, and static comparisons of the shipped binaries. I found a candidate change in the October 1 beta that may help narrow this down, but have not established a first-bad build by runtime testing.

Confirmed failure and recovery

• Dota’s render thread blocks in gameoverlayrenderer.so → blocking Unix-domain connect() to /tmp/steam_chrome_overlay_uid1000_spid<gamepid>.
• The corresponding gameoverlayui helper is alive, owns the listener, and stops accepting connections. Its queue reaches 11 with backlog 10. Its regular frame callback continues running.
• The helper logs the same diagnostic reported above:

  SerializedSurface.cpp (1437): OnChromePaintBufferClientDisconnect m_mapShMemFD has 1 entries, m_mapChromePaintBufferHandleToData 0 entries.
  

• Calling the helper’s existing acceptance routine once through GDB briefly restored rendering; the game subsequently froze again as the queue refilled.
• A temporary, memory-only hook that calls that acceptance routine once from CSerializedSurface’s regular frame callback drained the queue to zero. Repeated rapid Alt-Tab, opening/closing the overlay, and spectating subsequently stopped reproducing the freeze. This is an experimental workaround, not a proposed production patch; it changes timing. No installed Steam files were changed by the hook.

My reproductions involved Alt-Tab between Dota and another application, including Steam or a terminal; moving the game between workspaces was not required.

Candidate regression: renderer teardown added in the October 1 beta

I downloaded both complete bins_ubuntu12 archives from Valve’s CDN and verified their SHA-256 hashes against the archived manifests:

In the 64-bit gameoverlayrenderer.so, the latter build adds a block to the CBaseOverlayRenderer destructors. Approximate pseudocode inferred from disassembly; global/mutex names are placeholders:

lock(mutex);
if (active_renderer == this) {
    existing_connection_teardown(this);
    active_renderer = nullptr;
}
unlock(mutex);
// Continue pre-existing member/socket destruction.

The teardown routine sends IPC message 0x19 when the connection is valid and the IPC sender is available, closes socket member this + 0x154, and resets it to -1. The predecessor already closed that socket in the destructor, but did not call this disconnect-notification routine from that destruction path.

The static call chain is:

VulkanSteamOverlayDestroySwapChain
  → erase the swapchain's renderer from the map
  → CVulkanRenderer destructor
  → CBaseOverlayRenderer destructor
  → newly added conditional connection teardown / IPC 0x19

Useful ELF virtual addresses (before ASLR), for the two archived 64-bit renderer libraries:

In 1790904859, the added destructor blocks compare against global pointer 0xcb7f0; calls to teardown occur at 0x58e43 and 0x58f93. Class identities were checked against RTTI. The teardown routine also gains a null check around its IPC sender in this release.

This corresponds in time to the official October 1 beta notes, which mention a fix for Vulkan games recreating their swapchain. That correspondence is a lead, not proof that the advertised fix caused this hang.

Why the helper may fail to service reconnects

In the helper path identified in the current build, accepting a connection is nested inside sending a shared Chrome paint-buffer FD to the game:

shared paint-buffer drawing
  → if buffer is not already marked cached/sent: send its FD
    → if client socket lookup misses: accept a connection

Consequently, drawing an already-cached buffer skips this acceptance path; without that drawing path, it is not reached either. The regular frame callback does not independently service the listener in the inspected code.

With the overlay held open after the recovery hook was installed, a five-second GDB capture counted 357 shared-buffer draw calls and zero descriptor-send calls. The sampled cache entry for handle 0x22 had its cached/sent flag set to 1.

For current Steam build 1791918634, 32-bit vgui2_s.so addresses are: acceptance routine 0x2137b0; FD-send routine 0x215c40 (calls acceptance on socket lookup miss); cache branch 0x218958 skipping toward 0x218dfa; regular frame callback 0x20c020. Dota’s blocking renderer connect() call is at gameoverlayrenderer.so + 0x57a64.

Between the two adjacent beta packages above, vgui2_s.so’s entire .text section differs in only five bytes, all diagnostic immediate arguments apparently representing assertion line numbers. The accept/cache/frame-callback instruction bytes are unchanged. chromehtml.so is byte-identical. gameoverlayui.so also changes and has not been ruled out.

Static link from IPC 0x19 to the helper diagnostic

I also decoded the helper's IPC dispatch jump table, rather than inferring the message meaning from its number:

• In both adjacent betas, gameoverlayui.so jump table 0x20c834, entry 0x19, selects handler 0x2bb270. It obtains the overlay surface through virtual slot +0x2a8, then calls its slot +0x58 at 0x2bb288. The two case blocks have identical bytes.
• In current build 1791918634, the equivalent table is 0x20ca34, handler 0x2bb490, virtual call 0x2bb4a8.
• Current vgui2_s.so's IOverlaySurface vtable starts at 0x600824. Slot +0x58 points to thunk 0x20c300, which adjusts this by -0x198 and enters 0x20c150: the routine containing the OnChromePaintBufferClientDisconnect diagnostic.
• That inspected routine checks/prints remaining map entries but contains no map-clearing or cached-flag-reset operations. This does not exclude cleanup in another part of the system.

Thus there is a concrete static link between the newly added renderer-side notification and the diagnostic observed during the hang. Runtime evidence that the failing Alt-Tab actually takes the new destructor branch is still missing.

What remains unproven

The static comparison establishes that the destructor-side disconnect notification was added in this package transition. It does not establish that this is the first release with the hang, or that the new conditional path executes during the failing Alt-Tab transitions. The cache observations were made after recovery and do not establish the initial invalidation failure.

The most useful next checks appear to be:

1. Capture the teardown caller during swapchain recreation, together with the active-renderer pointer comparison, then follow IPC 0x19 and the next connection attempt. This would distinguish the candidate destructor change from other callers of the existing teardown routine.
2. Check what invalidates per-client shared-buffer sent/cache state on disconnect, and whether recreating a renderer for the same game PID can reuse a handle still marked sent to the previous connection.
3. Exercise repeated swapchain destruction/recreation with the overlay closed and with an already-cached visible buffer. Listener servicing should remain independent of whether a new texture FD needs to be sent, and the render thread should not wait indefinitely for a full listener queue.

The frame-callback hook only demonstrates the listener-service dependency. It is not a production fix: the existing acceptance routine also performs a blocking accepted-client handshake, so simply moving that whole routine into every frame callback would need additional care.

Suggested fix direction

Based on the recovery experiment, the primary fix appears to be servicing the listener independently of shared-buffer drawing and descriptor-cache misses. The following are proposed changes, not source-level findings:

1. Service incoming connections through the helper's event loop, including while the overlay is hidden or all buffers are cached. Accept queued connections without blocking, and process their handshakes asynchronously with bounded resource use. Simply calling the existing acceptance routine every frame is not sufficient as a production fix: it also performs a blocking accepted-client handshake. On Linux, accepted sockets do not inherit O_NONBLOCK; that needs to be set explicitly, e.g. with accept4(..., SOCK_NONBLOCK | SOCK_CLOEXEC) (accept(2)).
2. Audit descriptor-cache ownership across reconnects. A buffer marked sent must have been sent successfully to the current connection, not merely an earlier renderer with the same game PID. If that invariant is not already maintained elsewhere, invalidate the affected per-client state on disconnect or associate it with a connection generation, and resend the required descriptors. Our inspected disconnect callback does not perform that cleanup, but we have not excluded cleanup elsewhere.
3. Keep overlay connection establishment and descriptor reception from indefinitely blocking game rendering. Use asynchronous handling with deadlines/retries so rendering can continue while the overlay connection is unavailable or being re-established. This would contain the impact even if the helper encounters another failure.

// Helper event loop; runs regardless of overlay visibility or texture work.
// The listener itself is nonblocking and registered for level-triggered reads.
on_listener_readable() {
    while (accept_budget_remaining() && pending_peer_capacity_available()) {
        fd = accept4(listener, SOCK_NONBLOCK | SOCK_CLOEXEC);
        if (would_block(fd)) break;
        if (interrupted(fd)) continue;
        if (failed(fd)) { handle_accept_error(); break; }
        watch_pending_peer(fd, handshake_deadline());
    }
    // If capacity is exhausted, pause readiness notifications and re-arm
    // when capacity returns; also expire incomplete handshakes.
}

on_pending_peer_readable(peer) {
    result = peer.handshake.read_available_nonblocking();
    if (result == incomplete) return; // Preserve partial input; keep deadline.
    if (result == eof_or_error || !validate_handshake(peer)) {
        close_and_release(peer);
        return;
    }
    register_ready_connection(peer, fresh_connection_id());
}

ensure_buffer_available(peer, buffer) {
    key = (buffer.id, buffer.generation);
    if (!peer.ready || peer.sent.contains(key) || peer.in_flight.contains(key))
        return;

    peer.in_flight.insert(key);
    queue_nonblocking_fd_delivery(peer.connection_id, buffer, completion);
    // Completion removes in_flight and marks sent only at the protocol's
    // success point. On failure, do not mark sent; retry or retire the peer.
}

on_connection_closed(connection_id) {
    retire_that_connection_and_its_pending_io(connection_id);
    // Its sent/in_flight state dies with it. A replacement peer starts empty.
    // A late completion/close from the old connection must not mutate the new
    // connection's state merely because both identify the same game PID.
}

A useful regression test would repeatedly destroy/recreate Vulkan swapchains for the same game process, with the overlay both hidden and displaying already-cached buffers. Verify that fresh connections are accepted, required FDs are resent to each connection, stale sockets/FDs are released, and rendering continues. A client that connects but delays its handshake should not stall the helper either.

I would not recommend reverting the destructor change solely on this evidence: it may be necessary for the advertised swapchain crash fix. The static comparison identifies a candidate trigger; the listener-starvation failure and the benefit of independent listener servicing are supported by the runtime observations above.

Environment and unsuccessful alternatives

Runtime investigation: Steam 1791918634, native Dota build 25894101, NixOS, GNOME Wayland, private xwayland-satellite 0.8.3, RX 7900 XTX / Mesa 26.2.4.

Disabling Steam web-view GPU acceleration, testing [xwayland-satellite PR #499](https://github.com/Supreeeme/xwayland-satellite/pull/499) (ICCCM focus changes), and suppressing the startup overlay toast did not prevent the hang. No GPU reset or OOM was found. A sampled browser GPU presentation wait was not a persistent GPU stall. These did not explain the reproducible full socket queue or the recovery from servicing it.

1790721607 bins_ubuntu12 archive
ffda075bc03991f565c10298e8756c5c2c76a4c6fdb998c31a81d8ca584e027f
1790904859 bins_ubuntu12 archive
590d35091f87ced96517a69c0ddcf9a13c7b8554a472fa69c669af7ce8a87687

1790721607 ubuntu12_64/gameoverlayrenderer.so
ede84aed04c4e52c4e651856c95eb37c7a0892f4607712831d048d47f7d373c0
1790904859 ubuntu12_64/gameoverlayrenderer.so
e0248c176ad8cb55deb19433fde62f3335abfd179be9dcc637975adbff2616d8

1790721607 ubuntu12_32/vgui2_s.so
e354d4b8f4ffedf5250acdb63bbd50653250b0ea9df7ed2f1fdef8a5d4b656f3
1790904859 ubuntu12_32/vgui2_s.so
84a125ed71ce9cdd4394c299acf5ba6a5e7f8f3628cb526660d09ef1f6d433ce

System:

• Steam client: 1791918634 (Stable, build date Tue Oct 13 2026), not opted into beta
• OS: Debian unstable (sid), GNOME Shell 50.5 on Wayland
• GPU: NVIDIA GeForce RTX 3070 Ti Laptop GPU, proprietary driver 550.163.01 (hybrid laptop, Intel iGPU + NVIDIA dGPU; games run with __NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia)
• Affected: Valheim (892970, native Linux client) and Last Epoch (899770, Proton) — both freeze the same way

Same assert as #13740 (closed as a duplicate of this):

src/SerializedSurface.cpp (1437) : Assertion Failed: OnChromePaintBufferClientDisconnect m_mapShMemFD has 1 entries, m_mapChromePaintBufferHandleToData 0 entries.

Symptom:
The game window hangs completely after a short time (often on alt-tab or workspace switch; one freeze ~2 minutes after launch). Only fix is killing the process. While a game runs, gameoverlayui enters a crash/restart loop: repeated breakpad init + "reaping pid: ... -- gameoverlayui" every ~10 s, and minidumps are written to /tmp/dumps (auto-uploaded). The Steam console log also shows from gameoverlayui:

../common/pipes.cpp (900) : Fatal assert; application exiting

Regression boundary (in case it helps bisect):

• Previous stable build (overlay stamp 20260903013152, Sep 3): clean for ~2 weeks, including a 5h42m Valheim session with the same alt-tab / workspace-switch usage — zero SerializedSurface assertions.
• Current stable build (overlay stamp 20261013184430, Oct 13; rolled out here Oct 15, 13:49): freezes from the very first session; 170+ overlay assertions across multiple boots and hundreds of gameoverlayui restarts since.

Workaround confirms it's the overlay:
Disabling "Enable the Steam Overlay while in-game" globally (Steam → Settings → In Game) fully fixes it: 20+ minutes of alt-tab and workspace switching, no freeze, zero new assertions. With the overlay enabled the freeze reproduces every session.

Happy to attach Steam logs or minidumps if useful.