@kisak-valve - not sure if this is related to a similar issue that I reported about Need For Speed Unbound (https://github.com/ValveSoftware/Proton/issues/6502) but I just got The Crew Motorfest, as it was now listed as Steam OS compatible, but I'm getting a crash on my Intel GPU when it tries to start the game

Ubisoft announced that Steam Deck verification will be available on November 5th.

@PratMavrick (or anyone else getting crashes) are you playing online matches? Could you try playing for the same amount of time but without doing the online live matches against other players to see if that crashes or not?

The November 5th update has been verified via Steam Deck, so it won't crash anymore.

@PratMavrick (or anyone else getting crashes) are you playing online matches? Could you try playing for the same amount of time but without doing the online live matches against other players to see if that crashes or not?

Hi So, I open the game, directly go to the open world map , put a way point to the race I want to do and 5-6 minutes after either flying or driving, it shows me that error .

It runs on steamOS with no problems but not bazzite, still same error on bazzite.

Updated diagnostic with the latest version of Proton Experimental on Bazzite OS and a new The Crew Motorfest update (Build ID: 20381042), my Gist upload for steam-2698940.log which is 75,000 lines wouldn't save as one gist, so I split it up. The game doesn't start - it seems like it crashes sometime after the Ubisoft launcher, but no longer throws up the Error that I first reported that the older Proton-10.0-2 beta version still shows, as does GE-Proton-10.0-25, the loader_thunks.c error. One thing, I noticed though is the older Protons seem to launch the Ubisoft Launcher a lot quicker than the current Proton Experimental does.

System Information

GPU: Intel Arc A770m
Video driver version: Mesa 25.2.6
Kernel Version: 6.17.7-ba01.fc43.x86_64
Link to full system information report as [Gist](https://gist.github.com/):

Steam System Information
https://gist.github.com/pboard/a2f935c7008e90aae3401d632bed97a4
Steam Runtime Diagnostics
https://gist.github.com/pboard/758df8cd75db032bd94afd14cc6d780b
Steam Runtime Diagnostics
https://gist.github.com/pboard/4556b2b4560b70db61a75d2466e09ed5 (part 1)
https://gist.github.com/pboard/5be6a414957857bc5f2e5f05deb95d6d (part 2)
https://gist.github.com/pboard/67a4f3f24750b7e95796665e399df2fc (part 3)
https://gist.github.com/pboard/d73db833de8d158cd514ac745b252f6f (part 4)

I got the same error with Bazzite. I tried several Proton versions, such as 10.0-2 (beta), Proton GE 10-25, and Proton Experimental.
The only version that doesn't show this error and makes it through is the latest CachyOS version of Proton, although it goes back and forth between two different frames, and the frame rate is stuck at 3fps.
Here's a video of the game running on the Legion Go 2 with CachyOS Proton: https://imgur.com/OYkKXSs

I got the same error with Bazzite. I tried several Proton versions, such as 10.0-2 (beta), Proton GE 10-25, and Proton Experimental. The only version that doesn't show this error and makes it through is the latest CachyOS version of Proton, although it goes back and forth between two different frames, and the frame rate is stuck at 3fps. Here's a video of the game running on the Legion Go 2 with CachyOS Proton: https://imgur.com/OYkKXSs

This is the error I get if I use Proton Experimental or any other Proton version that's not the one from cachyOS: https://imgur.com/IdHunP4

This is great finding, wonder what is different, and how can we make it any better. Thanks for sharing that

This is great finding, wonder what is different, and how can we make it any better. Thanks for sharing that

Honestly I don't know, but from the error I'm getting, it could be related to some type of Vulkan error.

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-3507146719

I tried the latest Catchos Proton using Proton Plus on Bazzite, but for my Intel Arc a770m, it behaves similar to Proton Experimental, it doesn't get the Vulkan Thunks crash in my first post, but it seems to crash out some point after the Ubisoft Launcher and eventually quits.

But great that its helping with Amd gpus, but I guess the performance isn't great, so not to playable yet. Hopefully over time compatibility will improve beyond the steam deck, which is now verified.

I am facing the same error on bazzite, tried with proton experimental, proton 9.04, ge-proton 10.25, 10.21. I can here game audio in background.

Hi guys, So the an error has occurred every 5 minutes has returned. I don't know if its proton or the game servers are the issue . Pls if anyone has an idea , pls do tell

This is in my steam deck running the steam os and tries the proton experimental and 9.0.4

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-3508590319

This error is still present with the latest release of Proton 10.0-3 on Bazzite.

Hi guys, any luck with this?

Hi guys, any luck with this?

No, still not working with the latest version.

/Just fired up the Crew Motorfest on Bazzite 43 with Mesa 25.3.0 and the Latest Proton Experimental on my Intel Arc a770m, and it's now gotten to a Crew Motorfest screen, but it seems to be sitting on a message now about Running to get Pipeline cache: 2877 caches.

It's been sitting at this message for about 5min and no change to the 2877 caches figure so far.... But this is the furthest it's gotten so far. Hard to say if it's doing anything... I will let it run for a bit longer.... To see if it gets past that and gers into the game...

It's a bit of progress...

Update: So i terminated it after waiting 10min, launched with the latest GE-Proton, it starts much faster than Proton Experimental, and also now gets up to the same screen, although at the moment, it says 2880 caches, and with performance overlay enabled it seems to be in a frozen state, the CPU counter has frozen in the performance overlay. But i guess this tells me that it was the Mesa 25.3.0 update that just came out that's helped my Intel GPU get a bit further

Update: Launching the game from Desktop mode, I can see that in the System monitor, its flat lining one of the CPU's, it seems to switch between CPU cores (so far 5,6,8), so I think it might be processing whatever this cache pipeline task is

Update: So I decided to install Catchy OS in a separate Partition on my Intel Nuc 12 SSD, to see if an Arch OS based distro (which is like Steam OS) would get past the same point, but it also hangs at the "Running to get Pipeline Cache" - so it might be a bug for Intel Arc GPU's still. But I now have Bazzite, Catchy OS and Windows booting - with a shared Steam OS NTFS partition that holds the games

Hi guys . Can anyone try it on a steam deck(steam os) . It's just not letting me stay on the game for more than 4 minutes. Tried almost everything changed proton versions and reinstalled the game. Tried even the ubisoft Discord help , but they don't seem to care at all.

Is the pipeline cache the same as the shader cache?

Is the pipeline cache the same as the shader cache?

I think it might be, on Windows it generates at first launch and then after seems to not need to be generated again. I think if it could get past this point, the game might run on my Intel Arc Bazzite or Catchy OS setup

I think it might be, on Windows it generates at first launch and then after seems to not need to be generated again. I think if it could get past this point, the game might run on my Intel Arc Bazzite or Catchy OS setup

Hey, it started working again for me on Bazzite with Proton GE 10-27.
Maybe it works for you too.

I think it might be, on Windows it generates at first launch and then after seems to not need to be generated again. I think if it could get past this point, the game might run on my Intel Arc Bazzite or Catchy OS setup

Hey, it started working again for me on Bazzite with Proton GE 10-27. Maybe it works for you too.

I just tried thr Proton GE 10-27 version on both Bazzite and Catchy Os, but with my Intel Arc a770m it still gets stuck at the Pipeline Cache step.

I just tried thr Proton GE 10-27 version on both Bazzite and Catchy Os, but with my Intel Arc a770m it still gets stuck at the Pipeline Cache step.

That's a bummer. But yeah, I tried on my Legion Go 2 and I get a vulkan error. Only works on my Desktop PC for some reason.

I think I found the reason for the game crashing when it is building shaders is that it's trying to load a shader that it does not support, and that shader is called "nir_intrinsic_trace_ray_intel". This happens in some Unreal Engine games, one being the finals, where at first it couldn't get past the shader loading, but after the game updated, I was able to load shaders just fine. Marvel's rivals, after the game update, I am still unable to fully load shaders. Other games, not built on Unreal Engine, are experiencing these issues that I was unaware of. I think that's the issue, but someone was able to get around it somehow on the Mesa GitLab website.

issue:
https://gitlab.freedesktop.org/mesa/mesa/-/issues/14337

Has anyone found a workaround? I also reported this issue on the Mesa GitLab, but haven’t received any response so far.

Keep getting an error occurred after 5-10min of play time, it is unplayable

@PhialsBasement

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-3881669086

No thats not the error i get, i get "An Error Occured. Returning you to menu", this happens extremely often, however sometimes it doesnt happen at all, there is no pattern to it that i could find, ive tried lookiing at the log it had nothing to do with why it happens. I have also tried isolating it to device, but my steam deck AND my linux desktop both get it. I have also checked with ubisoft support and apperently this isnt exactly a new issue https://www.ubisoft.com/en-us/game/the-crew/motorfest/bug-reporter/issues/TCM-6936 as other people have reported it too and it is in the bug tracker.

@PhialsBasement I tried on deck too, no issue yet. I see some people on windows also reporting the "same" issue (it's generic enough that it's hard to know it's actually the same issue). There's a chance network plays a role here, have you been able to check on windows with the same network to know whether it's affected?

@PhialsBasement I tried on deck too, no issue yet. I see some people on windows also reporting the "same" issue (it's generic enough that it's hard to know it's actually the same issue). There's a chance network plays a role here, have you been able to check on windows with the same network to know whether it's affected?

@simifor No i havent as i had moved to linux about 4 months ago, but i do know that for the first ~3 weeks of gameplay it worked flawlessly and didnt error at all, after that it became almost a guarantee. I tried wiping the pfx, tried different distro(used Cachy instead of Garuda) and tried turning off ipv6 but nothing had worked, and considering you tried it and it worked fine.. Im starting to suspect it has something to do with account data corruption or uplay persisted settings across installs/pfxs/devices.. Either that or for whatever reason i might be silently banned or something.

Once i get home from work ill try it on a friends account and see if it still does the same thing.

Compatibility Report

Name of the game with compatibility issues: The Crew Motorfest
Steam AppID of the game: 2698940

System Information

GPU: Intel Arc A770m
Video driver version: Mesa 26.1.0-dev
Kernel version: 6.16.4-116.bazzite.fc42.x86_64
Link to full system information report as [Gist](https://gist.github.com/):

Steam System Information
https://gist.github.com/pboard/26aef27c095308fdfec420010a9c0952

Steam Runtime Diagnostics (and PROTON_LOG=1 output)
https://gist.github.com/pboard/651a494f5e65568fe2ceaa6c4a5e6808

Proton version: Latest

Image

Proton Experimental (10.x branch)
I confirm:

that I haven't found an existing compatibility report for this game.
that I have checked whether there are updates for my system available.

PROTON_LOG=1 output (cut output at line number in 20,000 range when it started looping)
https://gist.github.com/pboard/651a494f5e65568fe2ceaa6c4a5e6808

Symptoms

When I launch the game, which is now Steam OS compatible in the latest version according to the announcements, the game crashes out just after the Pipeline Cache screen (which is possibly the furthest its gotten on Linux with my Intel Arc a770m) - this behavior occurs with both the latest Proton Experimental and the latest GE-Proton.

Hello @pboard, skimming through your Proton log, *** stack smashing detected ***: terminated is where the game fell over, but I'm not seeing a hint near that to help figure out what component was involved to continue troubleshooting.

Thanks for looking. I have found that Bazzite OS with the latest stable update, using Mesa 26.0, and CatchyOS with the latest Mesa 26.1 Dev build, when using the Latest Proton Experimental, or Proton-GE do the same crash / terminate at the Pipeline Cache screen. If I switch to Proton 10.0-4, it doesn't crash, but hangs like it did back on the Mesa 25.3 branch on the Pipeline Cache screen. Something in Mesa 26.x and the latest Proton Experimental seems to not be working - as under Windows the Pipeline Cache should generate the 2700 odd cache bits before finishing, and it doesn't seem to be doing that on Linux with the Intel GPU's.

Would anybody be able to capture a fossil of the game if you have the pipeline creation issue?

Unfortunately not able to run the demo locally (problem with the Ubisoft launcher) to investigate.

Would anybody be able to capture a fossil of the game if you have the pipeline creation issue?

Unfortunately not able to run the demo locally (problem with the Ubisoft launcher) to investigate.

Is there some instructions on how to do this? I can give it a try

I was having constant "gpu lost" crashes at random intervals, disabling steam's performance overlay seems to have fixed those crashes (arch linux 6.18.13.zen1 , 9070xt radv)

(as far as i remember, the crew motorfest is the only game where i have this issue)

[  652.902336] amdgpu 0000:03:00.0: amdgpu: [gfxhub] page fault (src_id:0 ring:24 vmid:7 pasid:32817)
[  652.902341] amdgpu 0000:03:00.0: amdgpu:  Process TheCrewMotorfes pid 13136 thread RenderThread pid 13390
[  652.902342] amdgpu 0000:03:00.0: amdgpu:   in page starting at address 0x0000000000000000 from client 10
[  652.902343] amdgpu 0000:03:00.0: amdgpu: GCVM_L2_PROTECTION_FAULT_STATUS:0x00701430
[  652.902344] amdgpu 0000:03:00.0: amdgpu:      Faulty UTCL2 client ID: SQC (data) (0xa)
[  652.902344] amdgpu 0000:03:00.0: amdgpu:      MORE_FAULTS: 0x0
[  652.902345] amdgpu 0000:03:00.0: amdgpu:      WALKER_ERROR: 0x0
[  652.902345] amdgpu 0000:03:00.0: amdgpu:      PERMISSION_FAULTS: 0x3
[  652.902345] amdgpu 0000:03:00.0: amdgpu:      MAPPING_ERROR: 0x0
[  652.902346] amdgpu 0000:03:00.0: amdgpu:      RW: 0x0
[  663.314826] amdgpu 0000:03:00.0: amdgpu: Dumping IP State
[  663.315724] amdgpu 0000:03:00.0: amdgpu: Dumping IP State Completed
[  663.315763] amdgpu 0000:03:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
[  663.315764] amdgpu 0000:03:00.0: amdgpu: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[  663.315765] amdgpu 0000:03:00.0: amdgpu: ring gfx_0.0.0 timeout, signaled seq=742722, emitted seq=742724
[  663.315766] amdgpu 0000:03:00.0: amdgpu:  Process TheCrewMotorfes pid 13136 thread RenderThread pid 13390
[  663.315767] amdgpu 0000:03:00.0: amdgpu: Starting gfx_0.0.0 ring reset
[  663.315920] amdgpu 0000:03:00.0: amdgpu: Ring gfx_0.0.0 reset succeeded
[  663.315922] amdgpu 0000:03:00.0: [drm] device wedged, but recovered through reset
[  673.555380] amdgpu 0000:03:00.0: amdgpu: Dumping IP State
[  673.556107] amdgpu 0000:03:00.0: amdgpu: Dumping IP State Completed
[  673.556119] amdgpu 0000:03:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
[  673.556121] amdgpu 0000:03:00.0: amdgpu: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[  673.556123] amdgpu 0000:03:00.0: amdgpu: ring gfx_0.0.0 timeout, signaled seq=742723, emitted seq=742726
[  673.556125] amdgpu 0000:03:00.0: amdgpu:  Process TheCrewMotorfes pid 13136 thread RenderThread pid 13390
[  673.556127] amdgpu 0000:03:00.0: amdgpu: Starting gfx_0.0.0 ring reset
[  673.556196] amdgpu 0000:03:00.0: amdgpu: Ring gfx_0.0.0 reset succeeded
[  673.556198] amdgpu 0000:03:00.0: [drm] device wedged, but recovered through reset
[  683.796988] amdgpu 0000:03:00.0: amdgpu: Dumping IP State
[  683.797573] amdgpu 0000:03:00.0: amdgpu: Dumping IP State Completed
[  683.797581] amdgpu 0000:03:00.0: amdgpu: [drm] AMDGPU device coredump file has been created
[  683.797582] amdgpu 0000:03:00.0: amdgpu: [drm] Check your /sys/class/drm/card1/device/devcoredump/data
[  683.797583] amdgpu 0000:03:00.0: amdgpu: ring gfx_0.0.0 timeout, signaled seq=742724, emitted seq=742728
[  683.797585] amdgpu 0000:03:00.0: amdgpu:  Process TheCrewMotorfes pid 13136 thread RenderThread pid 13390
[  683.797586] amdgpu 0000:03:00.0: amdgpu: Starting gfx_0.0.0 ring reset
[  683.797652] amdgpu 0000:03:00.0: amdgpu: Ring gfx_0.0.0 reset succeeded
[  683.797653] amdgpu 0000:03:00.0: [drm] device wedged, but recovered through reset

GPULostDevice 2026-02-28 21.50.52.txt

I am having a similar issue, when the game launch and compile the shaders it just freeze.

The game is launching when using the CPU as a graphic card but doesn't start anymore after changing it to my iGPU in the game settings

Steam log:

steam-2698940.zip

Original size of the logs was 1.2Gb
@kisak-valve

And why is the game labeled as "Game compatibility - Unofficial" when it's verified ?

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-3990710702

I have the same exact issue and sometimes it crashes my GPU drivers as well. It's a really weird issue and since it happens like every 20-30 minutes it's really annoying.

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-4079747180

I've dug more into this since then and disabling the perf overlay wasn't enough in the end.
What really worked was disabling the fpu underclock that I had set with LACT (I disabled overclocking and uninstalled LACT) and closing Firefox when launching the game (not sure if it must stay closed while running the game)
Deleting mesa shader cache might also help.
And the kernel version bump to 6.19 seems to have helped with stability

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-4130370808

Same issue for me !

same issue here. start the game, get to the "running to get pipeline cache: [...]" screen and then hangs, on mesa 25.3.6. additionally, if you alt-tab out of it, it becomes black, and then it becomes transparent after a few minutes. on mesa 26.0.3 it just closes the moment you get there but steam doesn't detect that so you have to quit from steam to try again. if you run from the terminal it also outputs *** stack smashing detected ***: terminated, not to mention a whole lot of fossilized errors, the moment it supposedly gives up. i've tried a whole lot of workarounds which even included adding arguments to not pre-compile shaders, but it seems this process seems to be hard-coded in the game.

I would very much like to investigate the problem on Intel but the launcher is gating that atm :

Anybody knows how to work around this issue?

I was able to get access to the game and the issue appears to be a shader going over the limit of descriptor sets that can be used.

Only 8 available on Anv. Will see if this is something that can be worked around in vkd3d-proton or whether we need to bump the limit (other drivers are in the same boat with a similar limit).

Here is a change that will the game start : https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/40744

Yeah, finally some progress !

Would this also fix other games that are also having issues, where, when preloading shaders, the game crashes?

Unlikely to fix other things, this is a title that uses the Vulkan API, most don't.

relating to The Crew Motorfest - 'An error has occured. Returning you to the login menu.' portion of this thread...

Regression report: working on Apr 1 bleeding-edge (proton experimental), but broken again on April 2 bleeding-edge.

System Information

• OS: Nobara Linux 43
• Kernel: 6.19.10-203.nobara.fc43.x86_64 
• GPU: NVIDIA GeForce RTX 5070 
• NVIDIA Driver: 595.58.03 
• Display server: Wayland 

Symptom:

Game launches fully, Ubisoft Connect authenticates successfully, game is playable online — then at consistently ~4 minutes, BattlEye kicks with "An error has occurred. Returning you to the login menu."
Log analysis with PROTON_LOG=1 confirmed BEClient_x64.dll loads and unloads exactly ~228 seconds after load, consistent with the QueryWorkingSetEx/ProcessCycleTime memory scan issue described in #9560.

Proton versions tested

• proton experimental stable (mar 2026)
• GE-Proton 10-34
• Proton Experimental bleeding-edge, Apr 1 AEST (Mar 31 US?)
• Proton Experimental bleeding-edge, Apr 2 AEST (Apr 1 US?) — experimental-bleeding-edge-10.0-337730-20260401-p89fa6e-wf865c4-d1676dc-vad62f1

Everything worked well on the April 1st bleeding-edge build with a 3-4hr session without being kicked. The 4-min kick has returned since the latest bleeding edge update (today). The working build auto-updated overnight to the Apr 2 build before the version string could be captured. The working build is identifiable as the bleeding-edge revision showing "Apr 1, 2026" in Steam's betas UI.

Launch options used:

PROTON_ENABLE_NVAPI=1 PROTON_BATTLEYE_RUNTIME="/path/to/SteamLibrary/steamapps/common/Proton BattlEye Runtime" %command%
Note: Proton BattlEye Runtime (appid 1161040) is installed to a different Steam library than the game. Explicitly pointing to it via PROTON_BATTLEYE_RUNTIME was required for BattlEye to initialise at all.

Here is a change that will the game start : https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/40744

So i applied the latest mesa-git on Cachyos and with the 26.1 dev mesa, it can now get past the pipeline cache, it launches the game, but then i get a loader_thunks.c error message (Tried latest Proton Experimental, Latest Proton-GE and Proton 10.0-4 all give the same error).

So i might need to generate a new error log to post what's the next bug to fix.

@kisak-valve @llandwerlin-intel - attached is the Proton Debug log for Proton 11.0-beta on Cachyos with Mesa 26.1-dev that gets past the pipeline cache error, but creates the crash above. If I click the cancel on the crash, I can hear the background game music playing, but I never seem to quite get to the game screen, it just stays on a Black screen, but the main game window for The Crew Motorfest has launched at that point. This is on my Intel Arc A770m GPU, with the latest build of Cachyos.

steam-2698940.log

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-4258861672

Assertion failed error is issue with default gpu choice - game picked software renderer as gpu and it "works" only with cachyos proton. So to mitigate this issue is to switch to cachyos proton and change ingame gpu to your current gpu, after that proton can be changed to another or force game to pick only the gpu. Stucking at "running to get pipeline cache" is intel gpu issue - on 1145g7 I have the same when I switched gpu.

Assertion failed error is issue with default gpu choice - game picked software renderer as gpu and it "works" only with cachyos proton. So to mitigate this issue is to switch to cachyos proton and change ingame gpu to your current gpu, after that proton can be changed to another or force game to pick only the gpu. Stucking at "running to get pipeline cache" is intel gpu issue - on 1145g7 I have the same when I switched gpu.

The Assertion Failed error happens on Bazzite with my Legion Go 2 Z2E too.
SteamOS is fine tho.

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-4288075870

Thanks for that tip, switching to Cachyos Proton, my Intel Arc a770m is now able to run the Crew Motorfest.

So i can confirm with Mesa 26.1 dev channel and Proton Cachyos latest build, it works on Intel Arc Gpu.

Btw, i exited the Crew Motorfest after, and tried Proton 11 beta, but for me it went back to the Crash with the Assertion failed. I verified before exiting with Cachyos Proton that the Intel Arc Gpu was selected. So for now, i guess i will stick to the Cachyos Proton for this game, as it's working

Works fine for me on B580 proton experimental

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-4288307842

I just tried Proton Experimental on my Intel Arc a770m, but got the assertion failed message. Not sure what's different in Cachyos Proton, but it's working. If the difference can be merged into Valves Proton Experimental, then that should fix up the A770m. I have an Intel Nuc 12 Enthusiast, that has the Intel Arc a770m gpu

I have the Ubisoft Connect version of the game and I run it on Steam as a "Non-Steam Game". The game opens, and when it is supposed to enter the game, it closes and sends me back to Ubisoft Connect. It seems like it is not recognizing the "Proton BattlEye Runtime." I haven’t played the game in about a month, so I don’t know since which Proton version the game stopped working. I have tried all Proton versions: 9, 10, 11 beta, experimental, and the hotfix, and nothing works. I tried uninstalling and reinstalling it, and I also tested it on two other computers I have, and nothing. The game used to work perfectly, and now I can’t play it in any way.

EDIT: I solved my problem with this:
PROTON_BATTLEYE_RUNTIME=~/.local/share/Steam/steamapps/common/Proton\ BattlEye\ Runtime/ %command%

Replying to [#9119 (comment)](https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-4288307842)

I just tried Proton Experimental on my Intel Arc a770m, but got the assertion failed message. Not sure what's different in Cachyos Proton, but it's working. If the difference can be merged into Valves Proton Experimental, then that should fix up the A770m. I have an Intel Nuc 12 Enthusiast, that has the Intel Arc a770m gpu

So I just switched to Bazzite-Deck:testing to get Mesa 26.1.4, to fix the Intel Arc issue that @llandwerlin-intel fixed to see if Bazzite could boot The Crew Motorfest. Once more I found that Proton 10.0-4, Proton 11.0, Proton Experimental all get past that issue now, but they then crash with the thunks loader error after that.

However if I use Catch OS Proton 10.0 (catchyos-10.0-20260425-slr) that the game runs without issues. So there must be something in that which fixes an issue that the Valve steam is missing on the Intel Arc A series cards (well atleast for my a770m gpu) - @kisak-valve

I can do a fresh PROTON_LOG=1 on Bazzite OS if anyone would like for debugging?

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-5009787717

https://gist.github.com/pboard/af86a94bd1ac7bb45c345f236e6586ff

@kisak-valve , @llandwerlin-intel this is the PROTON_LOG=1 output with the latest Valve Proton Experimental on Bazzite-Deck 44 testing that has Mesa 26.1.4 on my Intel Arc A770m, intel nuc 12 enthusiast PC.

I really dont think this game should be Steam Deck verified when 90% of the time you get kicked within 5min of gameplay with "An Error Has Occurred"

@PhialsBasement Are you seeing this error on a deck or somewhere else? (If somewhere else, please also copy your system information from Steam (Steam -> Help -> System Information and Steam -> Help -> Steam Runtime Diagnostics) and put each in a gist, then include a link to the gists in this issue report.)

We were unable to reproduce this failure on the deck. Could you give more details about exactly when/where you see this error? Are you on a specific map? Which mode? (etc)

Hey, i was seeing it on all my linux devices (Deck and PC) across all modes (even waiting around right after the main menu), this has been occurring since at least https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-3434625212 for other people, for me it started shortly after that report, heres my PC's runtime diag: https://gist.github.com/PhialsBasement/4d367ffb9f2f61469fb34e74a0574d75

I did make a short attempt at debugging it earlier this year to see if i could patch whatever was wrong with proton that exposed this issue but couldnt find anything specific to any one run and every winedebug output just seemed the same regardless of kick or no kick. So i am not even sure if it is a proton issue it might genuinely be an issue with Ubisoft.

hey @alasky17 i was able to replicate it just now, heres the log. FYI i did force quit the game at the end but the error itself never crashes the game.

steam-2698940.log.gz

Hey @alasky17, also tagging @madtommy101 (who found the 228s unload). After a fairly deep dive into how the game talks to its servers and to BattlEye, I think I have the mechanism behind the "An error has occurred" kick.

What the game is reacting to is a Query Timeout sent by the session server on the game's own relay channel. The sequence is: BattlEye attaches, the server sends a query carrying a cheat-signature scan list, the client runs that scan and produces a 20337-byte report, the report gets fragmented into 20 pieces, and only 14 of them are ever transmitted. The server cannot reassemble the answer, so it times the query out and kills the session.

Every fragment header declares total = 20, but the transmission count per fragment index tells a different story. Verified across three independent sessions (different session keys, different relay hosts, different dates), with a self-checking wire format that validated with zero failures:

idx      0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19
kick7   31 29  0 30 31  1 30 31  1 30 31  1 30 31  1 30 31  1 30 31
kick6   31 31  0 30 31  1 30 31  1 30 31  1 30 31  1 30 31  1 30 31
kick5   25 25  0 25 25  0 25 25  0 24 25  1 23 25  2 23 25  2 23 25

Fragment index 2 is transmitted zero times in every session. The starved set is idx ≡ 2 (mod 3): indices {2, 5, 8, 11, 14, 17}. Each burst sends exactly 14 fragments (the 13 x 1118B + 1 x 974B pattern, visible on the wire with no decryption needed), then waits ~5 seconds and repeats. That runs 31 times over 210 seconds until the server gives up.

The timing is exact rather than approximate: the kill lands at query + 209.987s in both kick6 and kick7, identical to the millisecond. That is a hard server-side timeout on an unanswered query, which also explains why the reported load-to-unload figures scatter between 228s and 260s depending on session type.

On where the fragments go missing, I can narrow the location down without pinning it exactly. Every fragment that does get sent goes through ws2_32.sendto and succeeds, 431 calls with zero failures and zero short writes, so the socket layer itself is delivering everything it is handed. Valve's BattlEye bridge looks clean too since its deferred callback queue was empty mid-burst in a memory snapshot, and there is no path in that code that discards a queued packet. BattlEye's own fragmenter (disassembled out of the unpacked .text) is a straightforward monotone counter from 0 to 19 that builds every fragment the same way, so fragment 2 is definitely constructed. The drop therefore happens somewhere between the fragmenter and the socket call, and isolating which component is responsible is where I have run out of road.

How to repro: the fragment census is verifiable from a packet capture alone, with no decryption and no game instrumentation. Capture all UDP on port 4000 during a kicked session with tcpdump -i any -n -s 0 'udp and port 4000' -w capture.pcap. Count outbound packets of length 1118 (fragment) and 974 (last fragment). A healthy session would need ceil(20337/1024) = 20 unique fragments per burst; a kicked session shows exactly 14, and the same six indices are missing every time.

no issues regarding being kicked on my end

@PhialsBasement Could you give some instructions on the easiest way you've found to reproduce the crash? We want to be 100% sure we are looking at the right thing, since we haven't been able to reproduce the failure at all locally yet.

@alasky17 I can't give you deterministic steps, because I don't know what decides whether a session kicks or survives. On my machines right now it's near 100%, every session I captured yesterday kicked. A few days earlier, most of my sessions survived on the same Proton builds. Something environmental flips it and I haven't identified it, which is likely why it won't reproduce on your end. What I can give you is a way to confirm we're looking at the same failure,

1. Launch, go online, and let it sit. BattlEye attaches when you connect to live servers, not at process start. The kick lands about 210 seconds after that, and it fires even when fully AFK, so no specific map or mode is needed.

2. Capture all game UDP for the whole session: tcpdump -i any -n -s 0 'udp and port 4000' -w cap.pcap

3. Look at outbound packet lengths to port 4000. The BattlEye report goes out as the client's only large upload, in bursts of 13 to 14 packets (twelve or thirteen of length 1118 plus one of 974), with about 5 seconds between bursts. I see this across four separate kicked captures. Each burst carries 14 fragments where the report header declares 20, so it goes out short every time, the server fails to reassemble it, and it times the query out at +209.99s.

Attached are two kicked sessions, each with its raw port-4000 capture and the redacted decrypted records(anonymized), showing the client's BattlEye report declaring total = 20 fragments and putting only 14 on the wire and then proceeding with the kick with message Query Timeout
motorfest-fragment-drop.zip

You can also use this to verify the 14 of 20 fragment count yourself which also gives you a way to test any candidate fix without needing to reproduce the kick.

PS since you can't reproduce the kick, could you run tcpdump -i any -n -s 0 'udp and port 4000' -w survivor.pcap over four minutes of a normal online session and tell me whether the outbound 1118-byte packets arrive in unbroken ~100ms runs or with ~200ms gaps every third one, because a capture from a session that doesn't kick is the one control I can't get on my own machines and it would confirm whether the missing fragments are actually the cause?

@simifor what hardware are you on?

Replying to https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-5421729257

Hi! Thanks for these efforts, the details described here make sense (while, as you mentioned, don't yet explain what is actually going wrong).

I am a Proton developer, and I tried to look at this issue (which I am absolutely not reproducing, not even once). However, I think I see the mechanics in game how the described issues may be happening (while don't know what makes it consistently reproducible on your setup).

What stands between BE callback and calling ws2_32.sendto for the data received from callback is game code. The game is supposed to process BE's request to send data to server by relaying it to the game's server in its own way and then interact with BE server part on the servers. The callback is rather convoluted, it doesn't just call send, it processes the data (prepares the own server packet which is bigger than BE payload) and queues that to network IO thread named NetStreamPoolReceiveThread (the thread id can be found from Proton log by, e. g., 'grep renamed ~/steam-2698940.log | grep NetStreamPoolReceiveThread'. The problem is, there are only 2 buffer objects which are used to store data while those are queued before the processing finished. If by the time BE callback is called both are not in '0' internal state the data will be just ignored and go nowhere. I am guessing it is what happens at repro. That depends on timing. Here, without reproducing the actual problem, the 3rd packet (at least with the logging and additional traces I use) is always lost (just like in your case). But then all the others are sent fine, and then 3 packets are sent again soon (maybe BE requests resends from the first one till the last missing one) and that always succeeds. I tracked when the buffer status is set to '3' in NetStreamPoolReceiveThread (after that the main thread which queues it can reset it to 0 and reuse the buffer object). The sequence in NetStreamPoolReceiveThread is:

1. send the requested packet;
2. loop select() with some timeout until the UDP socket (the same one used for interacting with server on that port 4000) is read ready;
3. read packet from it (which after sending the concerned BE data is 62 bytes);
4. Set the buffer state to '3' (which effectively indicates that the operation is complete and it can be reused for queuing packets from the main thread; yet if main thread gets to BE callback when both buffers are in non-zero state, even 3, it will still fail queuing send, it resets the status 3 -> 0 only after the attempt to queue the packet).

So to complete the send NetStreamPoolReceiveThread thread not only sends the UDP packet, it waits for reply before releasing the buffer for next send. So first two packets never fail with two buffers, while the 3rd is on the mercy of net thread completing the send / receive with the server before the second packet is sent. I am guessing that in repro state net send / receive is consistently behind, while the packets are still sent and received in principle. Upon retry send the 3rd packet is never there probably because the send / receive for the first two are always late.

This all looks like a game bug to me. While still I am not sure why the fatal condition consistently repro on your side while absolutely doesn't for me or our QA, not sure how I can figure that without reproducing. It still might be that we have some fixable bug or at least a possibility for workaround, or that is something in host or network setup which can be avoided. If you are curious (and since obviously know what are you looking at here), maybe you can figure something out what goes on with those send / receive of those first to packets (1118 bytes send / 62 bytes reply on remote port 4000) on your setup, why that conversation may be delayed? Something might be affecting that on host or router? Or is it just systematic slow roundtrip to the server?

... or maybe on the affected setups those reply UDP packets sent by server back to client don't arrive at all (get filtered out)?? that could maybe explain the observed facts.

@gofman That matches what I measure, though so far this is analysis of existing captures rather than a confirmed in-game result.

The round trip looks like the culprit. Timing the last 62-byte reply to drain after the final fragment of a burst, across four captures: minimum 201ms, median 223ms, and 120 of 120 samples above 200ms. I am in Australia and every relay I have captured is us-east-1, so around 200ms is close to the floor for that path.

Taking your two-buffer description literally, fragment k needs the buffer from fragment k-2, which is released one RTT after its send, and BE paces fragments 100ms apart. That gives a drop condition of RTT > 200ms. Simulating it at 228ms produces 14 sent with indices [2, 5, 8, 11, 14, 17] missing, which is exactly my census. At 328ms it predicts 10 per burst with a different set.

I have not tested this in game yet. Adding 100ms with tc netem should move me from 14 fragments per burst to 10 if the model is correct. Running that in about 15 minutes and I will report back either way, including if it fails to match.

If it does hold, netem may let you reproduce this on demand instead of hunting for an affected setup.

Artificially delaying to reproduce on Proton / Linux is easy, from inside Proton even easier for me than with external tools. But I am not sure what it gives me, I can’t fix this reason anyway. Perhaps more interesting would be to try to reproduce on Windows with traffic delayed this way to confirm that it is the only reason for the issue. Might try to find some way for that later.

@gofman Tested in game and its confirmed now.

Steady-state burst with netem delay 100ms:

gaps      102  299   99  301   99  298  101  299  104
ticks     0, 1, 4, 5, 8, 9, 12, 13, 16, 17
dropped   {2,3, 6,7, 10,11, 14,15, 18,19}

On the replies being filtered, no, they arrive. 432 fragments sent / 636 replies received in one capture, 434 / 625 in another. They are just late, 201ms min and 223ms median here against the 200ms threshold.

Agreed the Linux repro gives you nothing fixable. For Windows(on the ROG Ally X), nothing in the mechanism involves Wine, so the same delay should reproduce it there. Ubisoft has this as TCM-6936 and there are Windows reports of the same error in this thread, which would fit if the trigger is latency rather than platform. (One reporter (UTC+7, CachyOS, all Proton versions tested, fresh prefix) notes sessions dying in 3-7 minutes during peak hours and running 2+ hours off-peak on the same day.)

oof i just took a look at the ROG Ally X report's images and its a diff issue since that one is a game crash. I am guessing no one has tested it like this on Win yet then, so maybe it is in WINE's high ping network handling

oof i just took a look at the ROG Ally X report's images and its a diff issue since that one is a game crash. I am guessing no one has tested it like this on Win yet then, so maybe it is in WINE's high ping network handling

What is Rog Ally X report? I could look at Proton log (from official Proton, Experimental or any 11 version). There is no special handling for high ping networks in Wine, not sure what could it be. The crash is probably something else entirely (like many GPU related reports in this issue).

What is Rog Ally X report?

Looking at ubisofts thread for TCM-6936 https://www.ubisoft.com/en-us/game/the-crew/motorfest/bug-reporter/issues/TCM-6936

The crash is probably something else entirely

Not sure if this helps, but The Crew 2 has the identical error on Windows/console going back to 2018, and the fix people pass around there is port forwarding / DMZ / UPnP, which would stall the same two buffers by the reply never arriving instead of arriving late.

https://steamcommunity.com/app/646910/discussions/1/1679190184061508123/
https://steamcommunity.com/app/646910/discussions/0/3815166629926677851/
https://www.reddit.com/r/The_Crew/comments/8uioag/the_crew_2_constantly_getting_thrown_back_to_the/

Oh, looks like that Ubisoft tracker thread is collection of any possible issues with zero actionable info, like Proton log. I am not sure even if Rog Ally crash report relates to us anyhow, that is probably from Windows and that is crash at menu and not kick / disconnection??

@gofman Might have something, no promises yet. i ran a few sessions with artificial delay (both random sets and static set and variable) applied via tc netem. Two of six completed the report and stayed connected instead of being kicked. Neither reproduced when I repeated its configuration, however utilizing this method across 383 recorded sends, every round goes 0, 1, skip 2, and adding delay only ever changed which of 3..19 dropped. In both sessions that survived, the round that got fragment 2 out had fragment 1 fail instead, so the failure moved one slot earlier.

If that is a buffer left in the completed state from the previous round being consumed by a failed queue attempt, it would fit what you described. Easier for you to confirm in the code than for me to prove from the wire.

Still digging, will report in the next hours if i find a stable workaround using this method, mainly cause its the only one i have seen actually change anything.

I am now able to force every index except idx 2 to be sent through by varying delay with netem, idx 2 seems to require a coincidence and only shows up ~26% of the time

Big news, i got it deterministically working and reproduced it 4/4 times.

The report consists of 20 fragments offered on a 100 ms tick, with two send buffers available. Once RTT is above ~200 ms, both buffers are still in flight when fragment 2 is offered, so fragment 2 gets dropped. The server then eventually times out at 210s. In the 144 rounds I recorded before this, fragment 2 was only sent twice.

The client also sends a small control record on the same stream shortly before each round. That consumes one of the buffers, and its ACK releases it again. Normally the record goes out around 500 ms before the round, so the buffer is freed too early to affect fragment 2.

Delaying only that datagram by 330 ms moves its ACK into the gap between the fragment 1 and fragment 2 ticks. The released buffer is then available for fragment 2.

It’s easy to distinguish on the wire since the control record is the only 94-byte UDP payload on the port; report fragments are 1118 bytes and ACKs are 62 bytes.

With that change, I’m seeing the same result on each run: fragment 2 makes it into the first burst, and the report completes in two bursts instead of timing out across 31 rounds:

burst 1  [0, 2, 4, 5, 7, 8, 10, 11, 13, 14, 16, 17, 19]
burst 2  [0, 1, 3, 4, 6, 7, 9, 10, 12, 13, 15, 16, 18]

I prototyped this with tc, so it can be tested without any Proton changes. It also appears to leave general latency unaffected; I’m still seeing about 0.75 ms to my gateway while it’s active.

The underlying issue appears to be Ivory Tower’s fixed two-buffer pool: when both buffers are occupied, the fragment is dropped rather than backpressured. The delay is a workaround for that behavior rather than a fix for the underlying issue.

I’ll open a PR shortly so we can continue the discussion there.

Hey @gofman the PR is up, lets move discussion there.

Hello @PhialsBasement ! Thanks a lot for the debugging and fixing efforts. Unfortunately I am afraid we can't take such a big and complicated diff to workaround what is most likely a game bug. This is too hacky and is hard to maintain going forward. I think this should be left for the game to fix. The only thing I'd try to do at some moment is to try to replicate the issue on Windows, even if with artificial delays to fully confirm there is nothing else going different under Proton (not much likely but can't say we have a full proof of that).

For the issue itself, I did some checks on Windows. While results are a bit inconclusive I still don't see anything fixable so far.

The part about missing sends in the presence of delays is the same. Normally without artificial delays there is about the same ~20 packtes sent to port 4000 of 1118 length, with the next one a bit shorter and then 3x1118 packets sent again. Adding delay of 100ms (I am using 'clumsy' tool on Windows and controlling the effect with Wireshark) results in only 15x1118 packets being ever sent. So the part how they are lost (as described in https://github.com/ValveSoftware/Proton/issues/9119#issuecomment-5433825775) seems to be the same on Windows and there is probably no room for Proton bug here. However, the consequences of that are different. On Windows after sending 15 packets with relatively close timing (~5 sec, roughly corresponds to what happens in Proton) there are no more attempts to send those again. And sitting in game over 15 minutes doesn't result in the kick. However, filtering those outgoing 1118 length packets completely does result in the crash.

The difference is most likely attributed to different BattleEye behaviour or setup of that (would it be the client part, or server, or both, I've got no insights on those details). So far I still don't see anything we can do here.

UPDATE: While on Proton I confirm previous findings, same artificially added delay does result in repeated attempts to resend the packets and eventually end with a server kick.

Unfortunately I am afraid we can't take such a big and complicated diff to workaround what is most likely a game bug.

Hey @gofman, understandable, would a much smaller version be worth considering, roughly 40 lines holding that one 94 byte datagram by a fixed 330ms gated on the appid, in the same shape as fake_ethernet_adapter in nsiproxy, or is the objection to carrying any workaround for this at all regardless of size? Also on Windows with the same artificial delay, does the server still send the 1024 byte type 0x04 query that the 210s timeout is anchored on, or does that exchange never start there in the first place?

Unfortunately I am afraid we can't take such a big and complicated diff to workaround what is most likely a game bug.

Hey @gofman, understandable, would a much smaller version be worth considering, roughly 40 lines holding that one 94 byte datagram by a fixed 330ms gated on the appid, in the same shape as fake_ethernet_adapter in nsiproxy, or is the objection to carrying any workaround for this at all regardless of size?

I am afraid no. Not only it is still very hacky and hard to maintain, it is also not a reliable workaround depending on fragile timing deduced from the debugging (depends on BE behaviour, how fast it is going to suggest the messages, gory game internals details which may change any moment). Besides, it might be interfering with the behaviour on better latency connections (it is probably arguable how much the game is playable at all with persistent 100+ms server pings).

Also on Windows with the same artificial delay, does the server still send the 1024 byte type 0x04 query that the 210s timeout is anchored on, or does that exchange never start there in the first place?

I wasn't looking inside such details precisely and where the type of query is, there are a bunch of 1206 UDP payload length UDpackets coming from server preceding the 1118 size replies, probably others too... the game is definitely replying on some server request. And as I mentioned filtering out those 1118 sized replies completely does result in a kick, just when the game ends up skipping packets same way as in Proton there are no resends and no kick. Yet again, sadly I don't have 100% proof that there is no Proton difference here, but that looks very unlikely to be something other than BE client and / or server behaviour difference. In the core of that I also don't think what we observe on Proton can be BE bug, it is game which refuses to send packets while in principle could.

Fair enough, and the fragility point is well taken. The hold depends on a measured round trip, on BattlEye's pacing, and on game internals that can shift under it at any time, so I can see why that is a poor thing to carry long term.

Thanks for taking the time to test it on Windows, that was the one piece I had no way to check myself. I will keep looking for an actual behavioural difference between Proton and Windows here, since if one exists that is the fix that is actually accurate and would belong here. If I turn up anything concrete I will bring it back to the issue.

Yeah if there was some simple and solid way I probably wouldn't be opposed to a game specific workaround in this case, but so far there is no such way seen (nor a bug which we would prefer to fix properly of course).

To be fair i could get it down to like 1 line
if (tcm_report_delay() && total == 94 && port == 4000) Sleep( 330 );
Provided the record and the fragments aren't on the same thread 🤣

For anyone hitting this from a high latency region, the same effect can be had with a tc rule and no Proton changes. It delays only the single 94 byte control datagram and leaves all other traffic alone (measured 0.75 ms to my gateway while active).

Find your interface:

ip route get 1.1.1.1

Apply, as root, replacing wlan0 with your interface:

IF=wlan0
tc qdisc del dev $IF root || true
tc qdisc add dev $IF root handle 1: prio bands 3
tc qdisc add dev $IF parent 1:1 handle 10: netem delay 330ms
tc qdisc add dev $IF parent 1:2 handle 20: pfifo
tc qdisc add dev $IF parent 1:3 handle 30: pfifo
tc filter add dev $IF parent 1: protocol ip prio 1 u32 \
    match ip protocol 17 0xff match ip dport 4000 0xffff \
    match u16 122 0xffff at 2 flowid 1:1

Remove:

tc qdisc del dev $IF root

The filter matches UDP to port 4000 with an IP total length of 122, which is the 94 byte payload plus the 8 byte UDP and 20 byte IP headers. That length is unique to this record type; report fragments are 1118 bytes and acks are 62.

I have also reported the bug over at TCM's bug tracker, upvote it so ubisoft can see it https://www.ubisoft.com/en-us/game/the-crew/motorfest/bug-reporter/issues/TCM-10312

I have also reported the bug over at TCM's bug tracker, upvote it so ubisoft can see it https://www.ubisoft.com/en-us/game/the-crew/motorfest/bug-reporter/issues/TCM-10312

NVM these guys are not interested in fixing bugs they said this and then closed the issue, the only way to get this "fixed" is to do it at proton level 😭

I have also reported the bug over at TCM's bug tracker, upvote it so ubisoft can see it https://www.ubisoft.com/en-us/game/the-crew/motorfest/bug-reporter/issues/TCM-10312

NVM these guys are not interested in fixing bugs they said this and then closed the issue, the only way to get this "fixed" is to do it at proton level 😭

I read that differently, it might be such technical details are just out of scope of user support and they are suggesting a way to reach game developers in a more direct way.

I read that differently, it might be such technical details are just out of scope of user support and they are suggesting a way to reach game developers in a more direct way.

In January their support told me to take this exact issue to the bug reporter and locked the thread (screenshot), and now the bug reporter closes it and sends me to support. I'll keep trying on their side regardless, i'll try to open a new thread.

Obviously contributing to the existing bug report wont do anyone much good because its swarmed with everything BUT this bug.