Component / version

• Repo: ValveSoftware/wine
• File: dlls/ntdll/unix/signal_x86_64.c, function sigsys_handler (under #ifdef HAVE_SECCOMP)
• Observed Wine version: wine-11.0 (Proton CachyOS 11.0-20260506-slr / 11.0-20260519-slr — both wine-11.0 base, pre-SUD)
• Also present in: Proton GE 10-x (carries the same patch), upstream Proton-Wine current
• Probably not affected: any Wine ≥ 11.5 with Syscall User Dispatch enabled (SUD takes a different code path; verification needed)
• Architecture: x86_64
• Kernel: Linux ≥ 5.x with CONFIG_SECCOMP=y, observed on 7.0.9 PREEMPT_RT and 7.0.9 SCHED_BORE (kernel-preempt-model-independent)

Summary

sigsys_handler advances %rip by +0xb (= 11 bytes) under the assumption that the trapping syscall instruction lives inside one of Wine's own 11-byte NT syscall trampolines. This holds when the trapping code is in Wine's ntdll.dll thunks, where the trampoline begins 9 bytes before the syscall and the next valid instruction starts 2 bytes after it (total trampoline length 11 bytes ⇒ rip + 0xb lands at the next instruction in the caller).

The assumption breaks for guest PE code that emits raw syscall instructions outside of Wine's thunks — notably Denuvo Anti-Cheat, which uses raw syscall instructions inside its VM-based anti-tamper logic to bypass any potential ntdll hooking. For those, rip + 0xb lands 9 bytes into the middle of the next Denuvo VM instruction (or, as observed below, onto a deliberately-placed jmp back to rip, forming a tight livelock).

What is observed

PROTON_LOG=1 trace from Arc Raiders (Steam App 1808500, Denuvo Anti-Cheat-protected UE5):

... ~750k legitimate NT syscall traps in the prior 18 s, with rax ∈
    {0x10..0x1e, 0x43, 0x50, 0xfd, 0x144e, 0x13e6, ...} and
    rip in Wine's ntdll syscall thunks (0x7000010fxxxx) ...

553.214:01b4:01b8:trace:seh:sigsys_handler SIGSYS, rax 0x50, rip 0x7000010fd988.
553.214:01b4:01b8:trace:seh:sigsys_handler SIGSYS, rax 0x43, rip 0x7000010fb06d.
553.214:01b4:01b8:trace:seh:sigsys_handler SIGSYS, rax 0xffffffff, rip 0x14a849ef0.   <-- onset
553.214:01b4:01b8:trace:seh:sigsys_handler SIGSYS, rax 0xffffffff, rip 0x14a849ef0.
553.214:01b4:01b8:trace:seh:sigsys_handler SIGSYS, rax 0xffffffff, rip 0x14a849ef0.
... 103,557,189 identical entries, ~27 wall-minutes, one Wine thread spinning,
    no game progress, 71 sibling threads correctly parked on
    NtWaitForSingleObject/NtWaitForMultipleObjects ...

PioneerGame-d.exe is loaded at PE base 0x140000000; rip = 0x14a849ef0 is at file offset 0xa849ef0 (≈ 177 MB into the Denuvo-wrapped binary). rax = 0xffffffff is not a valid NT syscall index — a classic Denuvo anti-tamper bogus-syscall pattern.

Root cause

The current sigsys_handler does (lightly elided):

static void sigsys_handler( int signal, siginfo_t *siginfo, void *sigcontext )
{
    extern const void *__wine_syscall_dispatcher_prolog_end_ptr;
    ucontext_t *ucontext = init_handler( sigcontext );
    struct syscall_frame *frame = get_syscall_frame();

    TRACE_(seh)("SIGSYS, rax %#llx, rip %#llx.\n",
                RAX_sig(ucontext), RIP_sig(ucontext));

    if (RAX_sig(ucontext) == 0xffff)
    {
        /* Test syscall from the Unix side (install_bpf). */
        RAX_sig(ucontext) = STATUS_INVALID_PARAMETER;
        return;
    }

    frame->rip = RIP_sig(ucontext) + 0xb;        /* <-- assumes Wine 11-byte thunk */
    frame->rcx = RIP_sig(ucontext);
    frame->eflags = EFL_sig(ucontext);
    /* ... eflags / restore_flags / rcx / r11 setup ... */
    RIP_sig(ucontext) = (ULONG64)__wine_syscall_dispatcher_prolog_end_ptr;
}

__wine_syscall_dispatcher performs the NT syscall translation in user-space, then returns to frame->rip (= original %rip + 0xb).

For Wine's own NT syscall thunks the +0xb is correct: the thunk has a fixed 11-byte form ending in the syscall instruction's 2 bytes; rip + 0xb lands at the first byte after the thunk.

For raw syscall instructions in non-Wine PE code (Denuvo, VMProtect, custom DRM, etc.) the +0xb is wrong — it skips 11 bytes into whatever happens to live there. In Denuvo's case (this report), the bytes at rip + 11 constitute (or eventually reach) a jmp back to the trapping rip, producing a livelock visible as 100M+ identical SIGSYS entries.

Three observed failure modes from the same root cause:

1. PREEMPT_RT kernel, no PROTON_LOG: livelock manifests as a hung Wine thread; sibling threads park on ntsync_char_ioctl+0x638 waiting for primitives the spinner is supposed to signal. Symptom = sustained microstutter until process kill. (Originally misdiagnosed as a kernel ntsync lost-wakeup — see ntsync_deadlock_report.md.)
2. SCHED_BORE kernel, no PROTON_LOG: same livelock, but +0xb lands on a slightly different decode and Denuvo's VM eventually executes an instruction reading from %rax (= bogus pointer like 0xfffffffffffff280), triggering SIGSEGV in user code. Symptom = hard game crash. (arc_segv_20260526_114343.txt.)
3. PREEMPT_RT, PROTON_LOG=1: same livelock, but +seh trace logging slows the trap rate and the game limps along while filling ~/steam-1808500.log at ~MB/s. Symptom = bimodal stutter (this report's primary trace).

Suggested fix

Validate rax against the syscall table before advancing %rip. For invalid rax, do not enter the dispatcher; instead set rax = STATUS_INVALID_PARAMETER (or similar) and advance %rip by only +2 (sizeof syscall instruction).

Sketch:

if (RAX_sig(ucontext) >= __wine_syscall_dispatcher_table_size)
{
    /* Not a Wine NT syscall — likely raw syscall from non-Wine PE code
     * (Denuvo Anti-Cheat, VMProtect, custom DRM, etc.). Don't assume the
     * 11-byte Wine thunk layout — advance past the 2-byte syscall
     * instruction only, return a defined error in rax. */
    RAX_sig(ucontext) = STATUS_INVALID_PARAMETER;
    RIP_sig(ucontext) += 2;
    return;
}

(__wine_syscall_dispatcher_table_size is illustrative — the actual check should mirror whatever bounds-check __wine_syscall_dispatcher does internally.)

This is consistent with the existing fast-path for rax == 0xffff (the install_bpf test syscall), which already short-circuits without entering the dispatcher.

Caveats

• Denuvo VM may rewrite the instruction stream. If Denuvo's anti-tamper actively detects skipped instructions and rewrites its VM dispatch to re-target the same rip, the proposed fix bounds the trap rate but does not prevent the livelock. In that case the only remaining defense is to refuse to dispatch the same (rip, rax) pair more than N times per second — an explicit rate-limit on sigsys_handler. This is uglier but salvageable: even an N=10 limit would change a hard livelock into "Denuvo VM gets one trap response per 100 ms".
• Wine 11.5+ with SUD likely takes a different code path and may already not exhibit this. Verification needed; if SUD makes the bug go away, the easiest user-facing remediation is "advance the Proton CachyOS / GE-Proton base to ≥ 11.5".

Why this matters

Affects Wine/Proton compatibility with all Denuvo Anti-Cheat-protected titles that exercise the VM tripwire mid-session. Currently manifests as either:

• "intermittent freeze / heavy stutter that won't go away until the game is restarted", or
• "the game just crashed" on certain kernel configurations.

Both have been widely reported under "Arc Raiders" specifically (ProtonDB, Proton issue tracker, r/linux_gaming), without prior identification of the root cause.

14_kernel_stacks.txt