split lock detection spamming dmesg
I'm getting the same on my Dell Latitude 5520 laptop (i7 iGPU + Nvidia MX450 dGPU) running Gentoo.
Same thing on Asus Vivobook (i5 1035G1 + nvidia mx350) on kubuntu 20.04, uname -r relevant parts:
5.11.0-37-generic [#41](/issue/ValveSoftware/steam-for-linux/41)~20.04.2-Ubuntu
I'm getting the same on my Thinkpad X1 Extreme Gen 4 laptop (i7-11850 + RTX 3070 max-q) running Gentoo. Thousands of these per minute.
dmesg log sample:
[36016.234614] x86/split lock detection: #AC: CJobMgr::m_Work/29551 took a split_lock trap at address: 0xea549163
[36062.856553] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36063.053937] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36063.250999] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36063.447947] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36063.646125] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36063.843318] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36064.040236] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36064.237212] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36064.434420] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36064.632189] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36071.445435] split_lock_warn: 1 callbacks suppressed
[36071.445437] x86/split lock detection: #AC: CJobMgr::m_Work/29551 took a split_lock trap at address: 0xea549163
[36109.158387] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36109.207848] x86/split lock detection: #AC: CJobMgr::m_Work/29550 took a split_lock trap at address: 0xea549163
[36109.306497] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36109.355958] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36109.405409] x86/split lock detection: #AC: CJobMgr::m_Work/29550 took a split_lock trap at address: 0xea549163
[36109.504461] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36109.553897] x86/split lock detection: #AC: CJobMgr::m_Work/29578 took a split_lock trap at address: 0xea549163
[36109.603366] x86/split lock detection: #AC: CJobMgr::m_Work/29550 took a split_lock trap at address: 0xea549163
[36109.702930] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36160.615124] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36160.812676] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36161.009782] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36161.207078] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36161.404459] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36161.602186] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36161.800162] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36161.998170] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36177.196524] x86/split lock detection: #AC: CJobMgr::m_Work/29551 took a split_lock trap at address: 0xea549163
[36182.346968] x86/split lock detection: #AC: CJobMgr::m_Work/29551 took a split_lock trap at address: 0xea549163
[36207.067028] x86/split lock detection: #AC: CJobMgr::m_Work/29551 took a split_lock trap at address: 0xea549163
[36207.315238] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36207.512238] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36207.710316] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36207.908317] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36208.105540] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36208.303372] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36208.500928] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36208.699036] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
[36208.896665] x86/split lock detection: #AC: CJobMgr::m_Work/34434 took a split_lock trap at address: 0xea549163
uname -a:
Linux x1e 5.14.10-gentoo-ligma #1 SMP Fri Oct 8 20:38:34 CDT 2021 x86_64 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz GenuineIntel GNU/Linux
Latest version of steam client:
Steam client version (build number or date): Oct 6th, 2021 at 18:51:29
Steam API: v020 Steam Package Versions: 1633666232
Distribution (e.g. Ubuntu): Gentoo
Opted into Steam client beta?: [Yes/No] No.
Have you checked for system updates?: [Yes/No] Yes.
CPU: model name : 11th Gen Intel(R) Core(TM) i7-11850H @ 2.50GHz
GPU: VGA compatible controller: NVIDIA Corporation GA104M [GeForce RTX 3070 Mobile / Max-Q] (rev a1)
Driver: 470.63.01
Same logs
[ 5656.175277] x86/split lock detection: #AC: CHTTPClientThre/21899 took a split_lock trap at address: 0x566a9e23
[ 5659.223628] x86/split lock detection: #AC: CHTTPClientThre/22014 took a split_lock trap at address: 0xea44b163
[ 5659.245241] x86/split lock detection: #AC: CJobMgr::m_Work/22000 took a split_lock trap at address: 0xea44b163
on my
Host: ZenBook UX325EA
OS: Ubuntu 21.10 x86_64
Kernel: 5.13.0-20-generic
CPU: 11th Gen Intel i7-1165G7 (8) @ 4.700GHz
GPU: Intel TigerLake-LP GT2 [Iris Xe Graphics]
Memory: 5427MiB / 15699MiB
Once it was hard freeze after that. Not good.
I'm seeing this, too. Steam Beta as of 2021-12-08:
[84130.796009] x86/split lock detection: #AC: CHTTPClientThre/3752 took a split_lock trap at address: 0xf20bb273
[84130.944128] x86/split lock detection: #AC: CJobMgr::m_Work/3261 took a split_lock trap at address: 0xf20bb273
[84133.751362] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84135.721442] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84135.967341] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84136.460236] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84136.706968] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84137.693282] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84138.186812] x86/split lock detection: #AC: CJobMgr::m_Work/3477 took a split_lock trap at address: 0xf20bb273
[84139.765611] x86/split lock detection: #AC: CJobMgr::m_Work/3352 took a split_lock trap at address: 0xf20bb273
[84141.196171] x86/split lock detection: #AC: CJobMgr::m_Work/3683 took a split_lock trap at address: 0xf20bb273
[84142.527785] x86/split lock detection: #AC: CHTTPClientThre/3270 took a split_lock trap at address: 0xf20bb273
[84144.648846] x86/split lock detection: #AC: CJobMgr::m_Work/3683 took a split_lock trap at address: 0xf20bb273
[84148.350382] x86/split lock detection: #AC: CJobMgr::m_Work/3683 took a split_lock trap at address: 0xf20bb273
[84149.483809] x86/split lock detection: #AC: CJobMgr::m_Work/3262 took a split_lock trap at address: 0xf20bb273
This problem wasn't present with my old Ivybridge system but since I upgraded to Alder Lake, I'm seeing this.
Split lock detected is thus probably a feature of modern CPUs and NOT a problem hitting ONLY modern CPUs, it's also present in older CPUs. According to kernel commits, this detection was added to find bad process behavior which negatively affects the performance of the whole system (even unrelated processes). I thus believe Valve should fix this, especially since Steam is about gaming, and gaming is about performance. The linked LWN aricle (https://github.com/ValveSoftware/steam-for-linux/issues/8003#issuecomment-965956455) indicates that fixing this may be as easy as recompiling with properly adjusting alignment.
However, I don't think processes are killed by the kernel as suggested in the previous comment: In my logs, I see repeating PID patterns which indicates that the same threads take the trap over and over again, the kernel would not recycle PIDs in a way that would explain this. The LWN article also says, killing offending processes can be one way to address the problem. Maybe it becomes default in the future, so it should be fixed sooner than later.
-/oyddmdhs+:. kakra@jupiter
-odNMMMMMMMMNNmhy+-` -------------
-yNMMMMMMMMMMMNNNmmdhy+- OS: Gentoo Base System release 2.7 x86_64
`omMMMMMMMMMMMMNmdmmmmddhhy/` Host: Z690 Pro RS
omMMMMMMMMMMMNhhyyyohmdddhhhdo` Kernel: 5.15.6-gentoo
.ydMMMMMMMMMMdhs++so/smdddhhhhdm+` Uptime: 23 hours, 43 mins
oyhdmNMMMMMMMNdyooydmddddhhhhyhNd. Packages: 2046 (emerge), 13 (flatpak)
:oyhhdNNMMMMMMMNNNmmdddhhhhhyymMh Shell: fish 3.1.2
.:+sydNMMMMMNNNmmmdddhhhhhhmMmy Resolution: 1920x1080, 3840x2160, 3840x2160
/mMMMMMMNNNmmmdddhhhhhmMNhs: DE: Plasma 5.23.4
`oNMMMMMMMNNNmmmddddhhdmMNhs+` WM: KWin
`sNMMMMMMMMNNNmmmdddddmNMmhs/. Theme: Breeze Light [Plasma], Breeze [GTK2/3]
/NMMMMMMMMNNNNmmmdddmNMNdso:` Icons: [Plasma], breeze [GTK2/3]
+MMMMMMMNNNNNmmmmdmNMNdso/- Terminal: konsole
yMMNNNNNNNmmmmmNNMmhs+/-` Terminal Font: Fantasque Sans Mono 14
/hMMNNNNNNNNMNdhs++/-` CPU: 12th Gen Intel i7-12700K (20) @ 6.300GHz
`/ohdmmddhys+++/:.` GPU: NVIDIA GeForce GTX 1660 Ti
`-//////:--. Memory: 8330MiB / 31885MiB
I'm seeing this on my Framework laptop as well, using Arch Linux.
-` animus@Xenon
.o+` ------------
`ooo/ OS: Arch Linux x86_64
`+oooo: Host: Framework FRANBMCP0C
`+oooooo: Kernel: 5.16.0-rc4-next-20211210-1-next-git-06579-gea922272cbe5
-+oooooo+: Uptime: 6 days, 13 hours, 11 mins
`/:-:++oooo+: Packages: 1065 (pacman)
`/++++/+++++++: Shell: bash 5.1.12
`/++++++++++++++: Resolution: 2256x1504
`/+++ooooooooooooo/` DE: GNOME 41.2 (Wayland)
./ooosssso++osssssso+` WM: Mutter
.oossssso-````/ossssss+` WM Theme: Arc-Dark
-osssssso. :ssssssso. Theme: WhiteSur-dark [GTK2/3]
:osssssss/ osssso+++. Icons: Papirus-Dark [GTK2/3]
/ossssssss/ +ssssooo/- Terminal: gnome-terminal
`/ossssso+/:- -:/+osssso+- CPU: 11th Gen Intel i7-1185G7 (8) @ 4.800GHz
`+sso+:-` `.-/+oso: GPU: Intel TigerLake-LP GT2 [Iris Xe Graphics]
`++:. `-/+/ Memory: 8931MiB / 64098MiB
.` `/
dmesg:
...
[29941.583864] x86/split lock detection: #AC: CHTTPClientThre/119074 took a split_lock trap at address: 0xe6d2f273
[29941.584627] x86/split lock detection: #AC: CHTTPClientThre/119074 took a split_lock trap at address: 0xe6d2f273
[29941.586852] x86/split lock detection: #AC: CHTTPClientThre/119074 took a split_lock trap at address: 0xe6d2f273
[29946.171847] split_lock_warn: 85 callbacks suppressed
[29946.171851] x86/split lock detection: #AC: CHTTPClientThre/119313 took a split_lock trap at address: 0xe6d2f273
[29946.575989] x86/split lock detection: #AC: CHTTPClientThre/119313 took a split_lock trap at address: 0xe6d2f273
[29946.628420] x86/split lock detection: #AC: CHTTPClientThre/119313 took a split_lock trap at address: 0xe6d2f273
...
I'm getting this on Arch with Intel Alder Lake CPU. Using Steam beta.
Also started happening for me quite recently... super annoying. Spams for almost the entire duration Steam is running.
@kisak-valve This affects all distros with a kernel that has split lock detection enabled, and with CPUs that can detect and report this situation to the kernel (although earlier CPUs might be affected as well). Future kernels will eventually kill such processes, currently it's a warning only. Most of these logs come from Steam client processes itself, please fix it. I'm also seeing this with some Uplay titles but it totally disappears in the log noise generated by the Steam client.
I'm currently using split_lock_detect=off as a kernel parameter to stop the log spamming but this isn't really helpful: The message is there to point to a situation degrading performance of the whole CPU.
Beginning with 5.19 the kernel will "make life miserable for split lockers".
Getting this on my machine too, and it seems to be periodically causing various other drivers to time out operations sometimes, including, but not limited to:
- SD cards
- Wi-Fi
- xHCI
This is bad enough that it could cause data corruption in various cases, potentially on the Steam Deck too.
It's easy to trigger this repeatedly just by doing Steam Remote Play. Which will sometimes even crash as a result of this.
In fact, it always crashes on exit (but it's silent outside of dmesg):
[197436.256887] streaming_clien[2629290]: segfault at 55b709e5c ip 000055b70825b744 sp 00007f92605f5d50 error 4 in streaming_client[55b707f38000+b86000]
Always in the same place too, modulo ASLR
Can you re-test on the Steam Client Beta?
https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/3387287522102609359
Can you re-test on the Steam Client Beta?
I can still see it but it seems much less noisy:
[ 112.572659] x86/split lock detection: #AC: CHTTPClientThre/5662 took a split_lock trap at address: 0xf21846d3
Only one occurence so far. Steam Client Beta 2022-07-21
Thanks, likely that some uses were missed. Will keep looking for them.
Some Assassins Creed games also throw that message but I'm not sure if wine or the Steam client could do something about it. In the light of future kernels killing such processes, how could that be handled? I'm not sure if Ubisoft would be interested in fixing such things, it's probably a non-issue under Windows?
The split lock can come from the game process even if it's caused by Steam (eg. overlay locking primitives), so I was hoping that reports of game instances would go away after this fix. If some pre-existing games indeed rely on split locks in their own code, I think we'll have to discuss the situation with upstream further and alert them to the fact there are pre-existing applications that are not under active maintenance that Linux desktop users still want to run. This might be a case of desktop-oriented distributions having to disable the mitigations by default.
Okay, so I'll retest the games I've seen logging this in the past - and report back here? Or per game?
Here are two other occurrences, one with different address, one with different thread name:
[272502.307107] x86/split lock detection: #AC: CIPCServer::Thr/5510 took a split_lock trap at address: 0xf218472d
[272502.330099] x86/split lock detection: #AC: CJobMgr::m_Work/5656 took a split_lock trap at address: 0xf21846d3
These seem the only messages left after a reboot, HTH:
[ 26.485020] x86/split lock detection: #AC: CHTTPClientThre/3024 took a split_lock trap at address: 0x565fc3e3
[ 49.462890] x86/split lock detection: #AC: CHTTPClientThre/3576 took a split_lock trap at address: 0xf1e496d3
[ 49.462923] x86/split lock detection: #AC: CHTTPClientThre/3575 took a split_lock trap at address: 0xf1e496d3
No game was started, the client just booted (and probably did its thing with fossilize and maybe spawning some prefix updates or whatever spawns wine processes after reboot).
In Linux 6.2 the kernel will actively punish split locks and would need kernel.split_lock_mitigate=0 set as kernel parameter to disable this behavior. Steam fixing this would be highly appreciated.
While the amount has gotten less, these still happen as of today:
x86/split lock detection: #AC: vulkandriverque/12320 took a split_lock trap at address: 0xf6cf9c47
x86/split lock detection: #AC: CHTTPClientThre/43404 took a split_lock trap at address: 0xe86df6d3
x86/split lock detection: #AC: CIPCServer::Thr/45705 took a split_lock trap at address: 0xe860b72d
x86/split lock detection: #AC: CJobMgr::m_Work/45708 took a split_lock trap at address: 0xe860b6d3
x86/split lock detection: #AC: ThreadedValidat/46251 took a split_lock trap at address: 0xe860b6d3
x86/split lock detection: #AC: CSystemManager:/11895 took a split_lock trap at address: 0xe872a19f
System information
- Steam client version (build number or date): 1668654564
- Distribution (e.g. Ubuntu): EndeavourOS
- Opted into Steam client beta?: No
- Have you checked for system updates?: Yes
System information
- Operating System: EndeavourOS
- KDE Plasma Version: 5.26.3
- KDE Frameworks Version: 5.99.0
- Qt Version: 5.15.7
- Kernel Version: 6.0.8-arch1-1 (64-bit)
- Graphics Platform: Wayland
- Processors: 20 × 12th Gen Intel® Core™ i7-12700K
- Memory: 31,1 GiB of RAM
- Graphics Processor: AMD Radeon RX 6800 XT
Actually, since the introduction of the new big picture mode into the client (although I don't use it), the spamming has increased again. But this is only one part of the problem, active punishing from the kernel will probably just throttle down the Steam client itself which shouldn't be such of a big issue (except the split locks are also in the API code games are using).
The bigger problem is games itself which Steam cannot do much about. Unless Microsoft introduces some similar way of punishing split locks in Windows, game devs won't fix it. And even if, what about the older/legacy games? Gamers and single user desktops probably just should use kernel.split_lock_mitigate=0.
The punishing is about preventing one user process from slowing down processes of other users. This is a non-issue in single user systems like when you are gaming on a desktop: The game mostly slows itself down by using split locks, and it's built around this performance characteristic. We should just ensure that other processes running on the system don't introduce additional performance costs - and that's why the Steam client should avoid these as much as possible.
And one more:
x86/split lock detection: #AC: CNet Encrypt:0/13152 took a split_lock trap at address: 0xe867319f
- Steam client version (build number or date): 1668654564
- Distribution (e.g. Ubuntu): EndeavourOS
- Opted into Steam client beta?: No
- Have you checked for system updates?: Yes
Got this one today. When witcher 3 crashed it showed this:
[ 1771.344713] x86/split lock detection: #AC: CSystemManager:/13367 took a split_lock trap at address: 0xf15c619f
And when I was playing Bendy 1 it showed this (But did not crash, I just did ALT+F4)
[ 2309.763555] x86/split lock detection: #AC: Bendy and the I/16798 took a split_lock trap at address: 0x3f40f64
And some more right after starting Steam, it downloaded some shaders I guess:
x86/split lock detection: #AC: CContentUpdateC/11219 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11220 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11258 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11259 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11261 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11262 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11263 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11264 took a split_lock trap at address: 0xee65019f
x86/split lock detection: #AC: CContentUpdateC/11266 took a split_lock trap at address: 0xee65019f
I'm currently seeing mostly this address (but a lot of them):
[499682.137546] x86/split lock detection: #AC: CJobMgr::m_Work/1429853 took a split_lock trap at address: 0xf1794f0f
[499682.186676] x86/split lock detection: #AC: CJobMgr::m_Work/1442500 took a split_lock trap at address: 0xf1794f0f
[499692.555101] x86/split lock detection: #AC: CJobMgr::m_Work/1442500 took a split_lock trap at address: 0xf1794f0f
[499692.703426] x86/split lock detection: #AC: CJobMgr::m_Work/1429612 took a split_lock trap at address: 0xf1794f0f
Steam package version: 1671133406
It looks like these are the only ones left currently, at least for an idle client:
# dmesg -t|grep "split lock"|sed 's#/[0-9]\+#/PID#'|sort -u
x86/split lock detection: #AC: CJobMgr::m_Work/PID took a split_lock trap at address: 0xf1794f0f
What worked for me (At least for now) was adding to /etc/default/grub the following:
GRUB_CMDLINE_LINUX_DEFAULT="split_lock_detect=off"
The split_lock_detect off solved the crashing or closing of the app in a rather abrupt way. I also read the following here for the 6.2 Kernel https://www.phoronix.com/news/Linux-Splitlock-Hurts-Gaming
What worked for me (At least for now) was adding to /etc/default/grub the following:
Yes and no: that successfully prevents the kernel from complaining, or adding any additional performance penalties or even kill a process. But it does also silently ignore the hardware-based performance hit that comes with that incident. This is not about silencing a kernel message, it's about removing the CPU-wide performance hit that comes with that situation by avoiding it in the code causing it (Steam in this case).
Silencing the message does not prevent the performance hit that this message actually tries to point at (although, kernel 6.2 actually adds an artificial performance penalty for the process causing the situation in favor of not slowing down other processes in the system, which is bad for games which usually cause this situation, and you can actually avoid that additional artificial performance cost by adding the parameter but you cannot avoid the hardware performance hit that comes with it in the first place).
You actually want Steam to not cause bus locks because a lock operation crosses a cache line, this will slow down your game or cause micro stutters if Steam does it in the background. It actually affects all processes running in parallel. If a game does it, this is acceptable (because the game is designed around that performance characteristic and it is the only foreground process you care about), but if background processes do that, it will hurt performance of processes potentially important to you.
I understand the silent part (not showing when checking dmesg) but how is it explained that only when O have that, I can play for example csgo, Witcher 3, cyberpunk for hours and without it O don't even last between 2 to 5 minutes before a crash happens. Only thing I changed was that. In regards to the penalty I would not know, all I was able to check was, if I have it, it does not crash or at least it does not crash for several continuous playing hours. If I remove it you can be sure I will never pass 5 minutes.
Could there be something else related to kernel 5.19 on Ubuntu 22.10?
Also for the steam part, I am 200% with you that either the game or steam should handle it and fix it.
There is one more lock that seems to not be mentioned in here:
x86/split lock detection: #AC: CNet Encrypt:0/17936 took a split_lock trap at address: 0xe732616f
This is right when starting Steam.
Not sure if this is related to Steam temporarily freezing my computer when starting.
Steam-Version: 1702079146
Steam-Client: Build-Datum: Fr., 8. Dez. 1:33 UTC -08:00
Steam: Webbuild-Datum: Sa., 9. Dez. 0:30 UTC -08:00
EDIT:
Actually I ran across more
x86/split lock detection: #AC: IPC:CSteamEngin/17834 took a split_lock trap at address: 0xe73261aa
Hi guys. I bought new laptop with i5-1135, tested it with Windoze10, all ran fine. Installed ArchLinux and Steam. Some Valve's Source engine games I play were working very very bad on ArchLinux. We are talking about more than 200fps on Windoze10, vs 30fps on GNU/Linux, for same game and same config. This crap happens because some geniustard Intel un-engineer Tony Luck decided to break userspace software, the kind of people who love to break other people's ficnished and working software just for fun. I even can not understand how things like this can get accepted into the kernel. What I found is that it affects at least Valve's Source engine based games, like Day Of Defeat Source, Counter-Strike Source, Portal1. Those run with 30 fps. I wasted the whole day trying to fix already working things. The other users maybe go back to Windows (faster painless fix, sadly).
@vitacell I fail to see how this fits here. This is an issue tracker for Steam, and split-locks should not be used since they slow the system down significantly.
Valve seems to understand this and has removed almost all their uses for split-locks in Steam as far as I can tell.
@vitacell I fail to see how this fits here. This is an issue tracker for Steam, and split-locks should not be used since they slow the system down significantly. Valve seems to understand this and has removed almost all their uses for split-locks in Steam as far as I can tell.
"as far as I can tell"? Valve's Source engine based games are still suffering from that, and they don't care very much about it. And users left on their own. It's not so hard to add "split_lock_detect=off", yeah. But it's not funny buying new hardware, and play old games at 30fps, then wasting your whole day trying ton of fixes and workarounds, trying to figure out what is happening. The quick fix is to install Windows, and this is what usually happens.
This throttling has actually been introduced because split locks are expensive for performance. This can be a real problem on cloud machines when someone accidentally or on purpose floods the system with split locks. The "fix" by the kernel devs was to throttle the processes causing the split locks, and it was also introduced so developers fix their bad behaving software. So this is actually a valid and proper fix and does not break user-space. But the problem is that especially Windows games are having this exact bad behavior. So I suggest desktop- or game-focused distributions should really ship with a kernel turning the throttling off by default.
You shouldn't point to the kernel devs, actually the kernel never tries to break user-space, such commits are usually considered bugs or regressions then. But in this case, it prevents a real problem and penalizes the causing processes, so it is a fix for a performance regression caused by bad behaving processes.
You should rather ask distributions to turn off split-lock detection by default, at least for game-focused kernels because old games won't be fixed, and current games won't be fixed either because Windows does no penalization. With such a kernel, this stops penalizing the games although the CPU still performs bad in split-lock situations. Just the extra penalty to the causing process would be prevented.
And yes, Steam has removed most if not all split-lock uses in the Steam client itself but that doesn't magically fix games that use split-locks.
We can only hope that game devs consider Linux performance (through the Steam Deck probably) and fix their games to not use split-locks. But for this to happen, it's probably better to not ship a kernel with split-lock detection disabled by default. So this is a double-edged sword.
Is this being worked on? It takes just opening Steam and letting games update. No need to start any game and dmesg already reports following:
[ 1353.685982] x86/split lock detection: #AC: CHTTPClientThre/7356 took a split_lock trap at address: 0xe988b1ef
[ 1359.668839] warning: `ThreadPoolForeg' uses wireless extensions which will stop working for Wi-Fi 7 hardware; use nl80211
[ 1375.266097] x86/split lock detection: #AC: CHTTPClientThre/7673 took a split_lock trap at address: 0xe988b1ef
[ 1379.850865] x86/split lock detection: #AC: CHTTPClientThre/7772 took a split_lock trap at address: 0xe988b1ef
[ 1473.812157] x86/split lock detection: #AC: CHTTPClientThre/8044 took a split_lock trap at address: 0x56646d1f
[ 1621.113060] x86/split lock detection: #AC: IPC:CSteamEngin/7341 took a split_lock trap at address: 0xe988b22a
This is on Steam in flatpak, but it does not matter how steam is installed (deb,rpm,flatpak).
Is this the cause for really slow "Validating" process for my games? I have been trying to debug what Steam is doing while it shows "Validating" because it doesn't seem to be using CPU nor causing IO load but the process is still really slow. Some kind of lock contention would definitely explain the slowness.
There are split locks in components of the Steam client itself. Younger kernels "punish" processes that cross CPU cache line boundaries which causes a split lock by pausing those processes for a few ms. The background is that on a server or VM host, non-privileged or isolated processes could cause a major performance slowdown by spamming the CPU with split locks maliciously. If the kernel forces such processes to pause, the performance penalty is mostly gone for other processes just the the causing process will make slow progress.
That said, you don't need that on a single user desktop system, or an otherwise trustworthy environment.
Open /etc/kernel/cmdline and add split_lock_detect=off to the end. If the file does not exist yet, copy the contents of /proc/cmdline first, remove any systemd.machine_id values and add the split lock value instead. This change will make the change persist updates.
Then recreate your boot files, e.g. for grub, systemd-boot, or initramfs. Consult your distro documentation on how to do that.
If even /proc/cmdline does not exist, you can also edit your boot menu entry directly. Or your distro has different ways of modifying the kernel cmdline. Ask your distro on how to do that.
With this change, the artificial split lock penalty will no longer be used. But your CPU still suffers from the situation and cannot reach full multi-core bandwidth. This setting does not change the underlying problem for which the kernel setting has been introduced in the first place. It is a band-aid to make developers fix their programs.
This issue will most likely see lots of activity soon, since amd enabled this on am5 with the 6.13 kernel
On far cry 5, i'm loosing 300% of the framerate from this over a couple of minutes ( going from 100fps to 30) which makes it unplayable
I think gaming-focussed distributions and/or kernels should turn this mitigation off by default. It usually only makes sense for multi-user environments anyways.
This issue will most likely see lots of activity soon, since amd enabled this on am5 with the 6.13 kernel On far cry 5, i'm loosing 300% of the framerate from this over a couple of minutes ( going from 100fps to 30) which makes it unplayable
Yep, can confirm, just got 9950X3D this week and with Fedora 42 (kernel 6.14.0.rc7) the log is spammed with those messages when Steam is running:
kernel: x86/split lock detection: #DB: CHTTPClientThre/4214 took a bus_lock trap at address: <addr>
On far cry 5, i'm loosing 300% of the framerate from this over a couple of minutes ( going from 100fps to 30) which makes it unplayable
@Lifeismana How did you conclude that the cause was the bus lock detection? Did disabling the feature via the kernel commandline make the framerate recover to its prior state? Were you getting the warnings in dmesg about tasks besides CHttpClientThread?
The game runs in a separate process from Steam so bus locks in Steam shouldn't affect the game, if I understand the kernel feature correctly.
FYI to people debugging issues with games who are getting these log messages. In addition to the split_lock_detect kernel commandline flag, which has 4 values (off, warn, fatal, ratelimit:N), there's also a sysctl: kernel.split_lock_mitigate. This controls whether the warn mode of the split lock detector penalizes the task it detected the split lock in. You can do sysctl kernel.split_lock_mitigate=0 at runtime to disable that penalty. How to set it persistently is distribution-dependent, but try /etc/sysctl.conf or /etc/sysctl.d/.
x86/split lock detection: #DB: CHTTPClientThre/2152 took a bus_lock trap at address:
x86/split lock detection: #DB: CJobMgr::m_Work/2480 took a bus_lock trap at address:
x86/split lock detection: #DB: PlanetSide2_x64/75378 took a bus_lock trap at address:
the spam goes for thousands of line if i start planetside
There is nothing that Valve can do about that for games. And it's probably difficult to fix things inside CEF except updating it to a newer version. Usually, splitlock detect inside Steam components may actually be good because it throttles CPU-heavy misbehavior down and your game gets more fps in turn. But your game is also affected by this, so you should probably just turn it off. Add split_lock_detect=off to your kernel cmdline (check your distribution docs and forums on how to do that).
If you later suspect low fps may be due to splitlock usage, you can turn that setting back on to see if the game does that. But usually, on single user systems, splitlocks are not an issue. So it's safe to turn the mitigation off.
If you launch the game with https://github.com/FeralInteractive/gamemode it will turn it off while the game is running and then back on after the game exits. Doesn't fix Steam itself spamming though.
Jesus, it's been 4 years, how has this not been fixed yet?
Jesus, it's been 4 years, how has this not been fixed yet?
Boot your system with split_lock_detect=off on the kernel cmdline. This mitigation has no useful purpose on single user systems. It's a countermeasure for multiuser / virtualization hosts with untrusted users.
It's still a silly mistake that is easy to fix. It shouldn't just be ignored for 4 years.
it ain't an easy fix if the one that made/knows that part of the steam client isn't at valve anymore
That's the most likely reason
It's still a silly mistake that is easy to fix. It shouldn't just be ignored for 4 years.
AFAIK, the problem with the Steam client is the CEF code. This doesn't exist on newer versions, but porting the code base is a lot more work than "easy to fix"
Also, it can't be fixed in games anyways: Windows games are notorious about doing split locks. It's not an issue for the game as the "single user" of the system. Windows simply ignores this (and accepts the hardware performance impact). But in multiuser or untrusted environments, split locks can become a real issue and attackers could severely affect system performance if you spam split locks on purpose. Surely, even 3rd party components in the Steam client could probably be fixed. But no developer cares about the older games, they won't be fixed. So the issue remains. The real fix is: Kernels for desktop Linux probably should not enable split lock mitigation by default, especially if using a gaming-optimized kernel.
Thus, simply disable it, or ask your distribution to disable it for desktop. Without split-lock mitigation, the kernel will no longer penalize processes causing split locks (tho, the performance loss on the CPU side resulting from this will remain but there will be no extra penalization). Kernel devs enabled that mitigation by default to force open source projects into avoiding split locks, and that worked (by slowing down processes that cause split locks). It reduces an attack vector that doesn't apply to desktop systems.
Valve already fixed the split locks inside most parts of the Steam client - so you cannot say they didn't fix it. But a few remain, and that's most likely from third-party components like CEF.
This also means, that with split lock mitigation on, the kernel will slow down games that cause split locks. Valve cannot fix that. The "silly mistake" will remain in games. Good luck asking them to fix that...
Really, the simple fix is: Turn it off in your system configuration. It has no use on desktop. Valve aren't the ones doing that for you. At best, your distribution should do that.
Or rather, ask Microsoft to implement split lock mitigation in Windows. That could probably work and force game developers into fixing their games and engines. Thinking about this, I actually kinda like that idea: If MS implements this without an option to turn that off, it might bring more gamers to Linux - because we can turn it off. ;-)
This doesn't involve rewriting or porting anything. You just find the object that they are taking the split lock on, and move it over a few bytes in memory so that it is aligned. Easy peasy lemon squeezy.
This doesn't involve rewriting or porting anything. You just find the object that they are taking the split lock on, and move it over a few bytes in memory so that it is aligned. Easy peasy lemon squeezy.
Yeah, it's mostly recompiling with a better alignment. But that's not gonna happen on older games, or with some game vendors /Ub...cough..ft/ sry ;-)
What an interesting issue. I never noticed this before today, so I'm not sure the timeline quite matches. In any case, I see this problem with Steam, not a particular Steam game. So, it seems like there is still room for improvement in the client.
kernel: x86/split lock detection: #DB: CHTTPClientThre/1848217 took a bus_lock trap at address: 0xf3bcd5a4
Or maybe this is actually a regression in CHTTPClient.... Most prior complaints regard CJobMgr and other components, not so often CHTTPClient. So, could this actually be a new bug in CHTTPClient?
@PorcelainMouse
Or maybe this is actually a regression in
CHTTPClient.... Most prior complaints regardCJobMgrand other components, not so oftenCHTTPClient. So, could this actually be a new bug inCHTTPClient?
The original post from 2021 has a log message from CHTTPClient. While there may also be new cases of split locks in that component, it's not new in general.
This issue in 32-bit parts of Steam is not going away soon.
~ uname -r
6.15.2-arch1-1
Since kernel 6.2, a new sysctl tunable[2] of split_lock_mitigate = 1 is available (by default) to enable mitigation, instead of the older[3]split_lock_detect=on kernel parameter.
[42404.521899] x86/split lock detection: #AC: CHTTPClientThre/36692 took a split_lock trap at address: 0xf3df8c6f
[42406.707619] x86/split lock detection: #AC: CHTTPClientThre/36698 took a split_lock trap at address: 0xf3df8c6f
[46719.298512] traps: steamwebhelper[45141] trap invalid opcode ip:7fd41e8cb0af sp:7ffc902a8690 error:0 in libcef.so[66ca0af,7fd41a5ed000+a36d000]
Upon setting: echo "kernel.split_lock_mitigate = 0" > /etc/sysctl.d/99-splitlock.conf
The kernel will log Steam / Cef lock splitting, but without the sequential access penalty.
Affected as well. To make things worse, this split lock nuisance cause a nvme controller crash, so it is a pretty nasty s***
Here is an excerpt from a log:
2025-06-30T22:26:34.364607+03:00 odysei-desktop kernel: message repeated 447 times: [ x86/split lock detection: #DB: CHTTPClientThre/1661173 took a bus_lock trap at address: 0xed666c74]
2025-06-30T22:27:01.088597+03:00 odysei-desktop kernel: handle_bus_lock: 9 callbacks suppressed
2025-06-30T22:27:01.088614+03:00 odysei-desktop kernel: x86/split lock detection: #DB: CHTTPClientThre/1661173 took a bus_lock trap at address: 0xed666c74
2025-06-30T22:35:34.531609+03:00 odysei-desktop kernel: message repeated 39 times: [ x86/split lock detection: #DB: CHTTPClientThre/1661173 took a bus_lock trap at address: 0xed666c74]
2025-06-30T22:35:51.174874+03:00 odysei-desktop kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
2025-06-30T22:35:51.174889+03:00 odysei-desktop kernel: nvme nvme0: Does your device have a faulty power saving mode enabled?
2025-06-30T22:35:51.174890+03:00 odysei-desktop kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
2025-06-30T22:36:01.776607+03:00 odysei-desktop kernel: x86/split lock detection: #DB: CHTTPClientThre/1661173 took a bus_lock trap at address: 0xed666c74
2025-06-30T22:36:01.776620+03:00 odysei-desktop kernel: x86/split lock detection: #DB: CHTTPClientThre/1661173 took a bus_lock trap at address: 0xed666c74
2025-06-30T22:36:09.062609+03:00 odysei-desktop kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3204876392, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
2025-06-30T22:36:09.062615+03:00 odysei-desktop kernel: I/O error, dev nvme0n1, sector 3204876392 op 0x0:(READ) flags 0x80700 phys_seg 21 prio class 0
2025-06-30T22:36:09.062616+03:00 odysei-desktop kernel: nvme0n1: I/O Cmd(0x2) @ LBA 3204876648, 256 blocks, I/O Error (sct 0x3 / sc 0x71)
2025-06-30T22:36:09.062617+03:00 odysei-desktop kernel: I/O error, dev nvme0n1, sector 3204876648 op 0x0:(READ) flags 0x80700 phys_seg 2 prio class 0
@odysei Your NVMe controller crash is completely unrelated to the split lock detection. They merely take place somewhat near each other in time. Note that those log messages are separated by many seconds, which is a very long time when NVMe controllers are concerned. So they are not even that near each other in time.
Maybe not, 17 s is indeed a substantial gap (misread initially). Anyway, I will disable a split lock to compare.
Hello, I observed this as well:
- steam build 1751405894
- Split lock detection spam occurred for approximately 37 minutes (17:01 to 17:38) before lockup.
- impact: complete system lockup, hardware watchdog rebooted. this is the 3rd time I've observed this behavior in 6 months
- I will test the
split_lock_mitigate=0suggestion from the comments / archwiki
sys info
-` user@archlinux
.o+` ---------------
`ooo/ OS: Arch Linux x86_64
`+oooo: Host: MS-7D75 1.0
`+oooooo: Kernel: 6.15.5-arch1-1
-+oooooo+: Uptime: 1 hour, 12 mins
`/:-:++oooo+: Packages: 1282 (pacman), 9 (flatpak)
`/++++/+++++++: Shell: bash 5.2.37
`/++++++++++++++: Resolution: 2560x1080
`/+++ooooooooooooo/` DE: Hyprland
./ooosssso++osssssso+` Theme: Adwaita [GTK2/3]
.oossssso-````/ossssss+` Icons: Adwaita [GTK2/3]
-osssssso. :ssssssso. Terminal: kitty
:osssssss/ osssso+++. CPU: AMD Ryzen 9 9950X (32) @ 5.756GHz
/ossssssss/ +ssssooo/- GPU: AMD ATI Radeon Graphics
`/ossssso+/:- -:/+osssso+- GPU: NVIDIA GeForce RTX 5090
`+sso+:-` `.-/+oso: Memory: 3124MiB / 61872MiB
`++:. `-/+/
.` `/
Logs
# steam launch + immediate split lock
Jul 14 17:01:57 archlinux steamwebhelper[40157]: exec ./steamwebhelper -nocrashdialog -lang=en_US -cachedir=/home/user/.local/share/Steam/config/htmlcache -steampid=40132 -buildid=1751405894 -steamid=0 -logdir=/home/us>
Jul 14 17:01:57 archlinux kernel: x86/split lock detection: #DB: CHTTPClientThre/40207 took a bus_lock trap at address: 0xf3e58c54
Jul 14 17:01:57 archlinux kernel: x86/split lock detection: #DB: CHTTPClientThre/40207 took a bus_lock trap at address: 0xf3e58c54
# multiple steam processes affected
Jul 14 17:35:14 archlinux kernel: x86/split lock detection: #DB: CHTTPClientThre/45562 took a bus_lock trap at address: 0xf3de0c54
Jul 14 17:35:14 archlinux kernel: x86/split lock detection: #DB: CHTTPClientThre/41208 took a bus_lock trap at address: 0xf3de0c54
# system overwhelmed
Jul 14 17:35:22 archlinux kernel: handle_bus_lock: 10 callbacks suppressed
# last message before lockup
Jul 14 17:38:05 archlinux kernel: x86/split lock detection: #DB: CHTTPClientThre/41208 took a bus_lock trap at address: 0xf3de0c54
# [System becomes unresponsive, hardware watchdog triggers reboot]
Note: No kernel panic, OOM, or critical errors - system silently becomes unresponsive due to interrupt storm
split lock mitigation most probably does not cause your lockups...
Is the trap address or specific process useful?
[ 7296.665139] x86/split lock detection: #DB: CHTTPClientThre/3764 took a bus_lock trap at address: 0xf39ebc54
[ 7296.667946] x86/split lock detection: #DB: CHTTPClientThre/3764 took a bus_lock trap at address: 0xf39ebc54
[ 7296.747387] x86/split lock detection: #DB: CHTTPClientThre/3764 took a bus_lock trap at address: 0xf39ebc54
[ 7296.747402] x86/split lock detection: #DB: CHTTPClientThre/3764 took a bus_lock trap at address: 0xf39ebc54
[ 7297.058391] x86/split lock detection: #DB: CHTTPClientThre/3764 took a bus_lock trap at address: 0xf39ebc54
[ 7297.058513] x86/split lock detection: #DB: CHTTPClientThre/3764 took a bus_lock trap at address: 0xf39ebc54
➜ ~ ps -eLf | grep 3764
user 3618 3291 3764 0 50 08:36 ? 00:00:00 /home/user/.steam/debian-installation/ubuntu12_32/steam -srt-logger-opened -nominidumps -nobreakpad
Maybe? I've taken the liberty of wading through my /proc/{pid}/map.
# truncated from [journalctl]
Aug 16 13:45:10 bazzite kernel: x86/split lock detection: #DB: CHTTPClientThre/6133 took a bus_lock trap at address: 0xf378bc54
In my case, it seems that the issue lies within the Steam Linux Runtime (Scout)[?], specifically libtier0_s.so.
# truncated [cat /proc/6133/map]
f3779000-f37ac000 r-xp 00022000 00:22 946470 /var/home/bazzite/.local/share/Steam/ubuntu12_32/libtier0_s.so
Voting for this issue. It stays for a long time.
Of course it's possible to get rid of warnings using "split_lock_detect=off" but it doesn't solve the problem.
Also sometimes the system becomes less-responsible when steam downloads something and it's UI is on foreground. Maybe it's coincidence, I don't know.
Of course it's possible to get rid of warnings using "split_lock_detect=off" but it doesn't solve the problem.
Well, this conclusion is wrong in a special way: it removes the direct performance penalty applied by the kernel to misbehaving processes. It still doesn't fix the performance penalty that the CPU suffers during a split lock (and actually, the kernel penalty will prevent such processes from affecting the whole system performance).
BUT: Whatever is shown here, often comes from games itself. If split lock mitigation is on, this will penalize the games for misbehaving - and you don't want that. Thus, turning the detection off is the correct solution. Gaming-focused distributions should just do that, and people would probably stop complaining here. There is only so much that Valve could fix here in their own code (and most if not all has been fixed).
@TheFlagCourier identified a library that misbehaves within the scope of Valve's software distribution, they can fix that. But it will still hit you with certain games, especially some Ubisoft titles. There's nothing that can be done about it - unless Windows would start penalizing misbehaving games, too. Only then, game devs will fix this. But Windows doesn't have such a feature, so if you want to game on your Linux kernel, turn it off.
Newer kernels allow dynamically switching this off and on. In that case, Proton itself could turn it off while the game is running, and turn it back on afterwards. This is probably what we should ask Valve for.
If a split lock occurs, the CPU needs to lock the complete bus, which prevents concurrent access to the bus in multithreaded environments. Developers need to adjust their code to prevent this. Turning the detection off will just stop the kernel from complaining - so far your argument is correct. BUT: It will also prevent the kernel from slowing down processes that cause split locks, thus the overall system performance will slow down.
This is two sides of a coin:
- Turn detection off to prevent penalization of misbehaving processes. This will leave the system vulnerable to split lock attacks which can be used to slow the system down. Mitigating split locks is mostly important for multi user systems, e.g. cloud servers or virtualization platforms. It's not important for your gaming rig. You don't want your game to be slowed down artificially, in addition to the bus lock itself which occurs independent of that - always.
- Turn detection on to penalize misbehaving processes. This prevents an active split lock attack to slow down other processes in the system. It will also slow down your game if it is the misbehaving process. It will also allow developers to identify such processes. And it will solve a real problem where one cloud VM could slow down another, or some malicious process can slow down the system.
So repeating: you want it off. Split locks outside of the game are rare enough to not affect the game performance (even if not penalized). But split lock mitigation inside the game will considerably slow it down if you leave split lock detection enabled.
This continues to be an issue. In my case, I've read what @kakra wrote above, but am getting the spam with no games running. If the client is running, the errors flow, whether or not I'm running a game, or even HAVE run a game since booting up.
@Locklear93 If it doesn't affect the games itself, this is actually a good thing: It means that processes that slow down your CPU unintentionally, become throttled. This doesn't mean that Valve shouldn't fix the issues somehow but it's also not bad for your system performance as long as game executables aren't affected. If the log spamming is extreme, it might affect the system overall, though. In that case, you might just turn split lock mitigation in your kernel off, and let others deal with it.
Thank you for the comprehensive explanation, @kakra ! Yes, I switched it off, but a lot of people don't and it split_lock_detection is on by default in Ubuntu :-(
I think the kernel devs are working on a way to disable the mitigation per process or per cgroup. Thus, Proton could start games with such a flag set, and only the Steam client itself or other processes in the system would remain under active mitigation control.
@kakra
If the log spamming is extreme
That's my primary concern, yes. I've got my logs capped at 200 MB because of situations I've seen where journald got out of control, and countless lines of this spams out logs I may need.
This continues to be an issue. In my case, I've read what @kakra wrote above, but am getting the spam with no games running. If the client is running, the errors flow, whether or not I'm running a game, or even HAVE run a game since booting up.
@Locklear93 Can you pinpoint the logs to the Steam client? Or maybe some other process? @TheFlagCourier did a great job of hunting it down to a specific library.
@kakra I lack the experience to narrow it down inside the mapping, but am familiar enough to figure out what TheFlagCourier did to trace it to the mapping. There's a lot in my /proc/7595/maps, but anything other than /run/ or /tmp/ related stuff is pointing to one of several files belonging to Steam. Specific libraries are named, but different ones on different lines, and I couldn't begin to tell you which are the offender(s).
on my system it looks like the memory location the kernel is complaining about lives inside of this mapping:
f3c76000-f3ca9000 r-xp 00022000 00:23 4888587 /var/home/user/.var/app/com.valvesoftware.Steam/.local/share/Steam/ubuntu12_32/libtier0_s.so
the address is 0xf3c88c54
libtier0_s.so
This seems to be a 32-bit library specifically from the Steam client. A quick google search didn't reveal that it would be part of any open source project. But it could be seen in a lot of back traces from Steam crashes (or games that use Steam libs) in the past. So I suspect Valve is in control of it and they should fix the split locks in this file. OTOH, I think they are in the process of transitioning the Steam client to 64-bit which would probably also fix the split lock problem with the 32-bit version of this library (because it would be no longer in use).
yes, looking at the names the file exports gives us some mild indications of "this is made by valve", like
00032050 T Plat_InternalOverrideArgv
00032390 T Plat_IsChromeOS
000324e0 T Plat_IsGamescope
00026240 T Plat_IsInDebugSession
000322c0 T Plat_IsSteamConsoleMode
00032400 T Plat_IsSteamDeck_DoNotUse
00032370 T Plat_IsSteamOS
00032240 T Plat_IsSteamOS3
00032470 T Plat_IsTesla
00032550 T Plat_IsWSL
and some strong indications of "this is made by valve", like
00035130 T _ZN16SteamThreadTools12CThreadEvent3SetEv
00034b10 T _ZN16SteamThreadTools12CThreadEvent5CheckEv
00035010 T _ZN16SteamThreadTools12CThreadEvent5ResetEv
00034c60 T _ZN16SteamThreadTools12CThreadEventC1Eb
00034cb0 T _ZN16SteamThreadTools12CThreadEventC1EPKcbb
00034c60 T _ZN16SteamThreadTools12CThreadEventC2Eb
00034cb0 T _ZN16SteamThreadTools12CThreadEventC2EPKcbb
I was able to figure out that the place that causes the bus lock on my system is ThreadInterlockedExchangeAdd64+0x44, but due to omitted frame pointer in libtier_0.so and no dwarf debug symbols available, I can't get the caller that is actually using this function.
It's just a helper function to use the `lock cmpxchg8b` instruction
00035c10 <ThreadInterlockedExchangeAdd64>:
35c10: 55 push %ebp
35c11: 57 push %edi
35c12: 56 push %esi
35c13: 53 push %ebx
35c14: 83 ec 14 sub $0x14,%esp
35c17: 8b 44 24 2c mov 0x2c(%esp),%eax
35c1b: 8b 54 24 30 mov 0x30(%esp),%edx
35c1f: 8b 6c 24 28 mov 0x28(%esp),%ebp
35c23: 89 44 24 08 mov %eax,0x8(%esp)
35c27: 89 54 24 0c mov %edx,0xc(%esp)
35c2b: 8b 45 00 mov 0x0(%ebp),%eax
35c2e: 8b 55 04 mov 0x4(%ebp),%edx
35c31: 8b 4c 24 08 mov 0x8(%esp),%ecx
35c35: 89 c6 mov %eax,%esi
35c37: 89 d7 mov %edx,%edi
35c39: 8b 5c 24 0c mov 0xc(%esp),%ebx
35c3d: 01 c1 add %eax,%ecx
35c3f: 11 d3 adc %edx,%ebx
35c41: 89 0c 24 mov %ecx,(%esp)
35c44: 89 5c 24 04 mov %ebx,0x4(%esp)
35c48: 8b 1c 24 mov (%esp),%ebx
35c4b: 8b 4c 24 04 mov 0x4(%esp),%ecx
35c4f: f0 0f c7 4d 00 lock cmpxchg8b 0x0(%ebp)
35c54: 75 db jne 35c31 <ThreadInterlockedExchangeAdd64+0x21>
35c56: 83 c4 14 add $0x14,%esp
35c59: 89 f0 mov %esi,%eax
35c5b: 89 fa mov %edi,%edx
35c5d: 5b pop %ebx
35c5e: 5e pop %esi
35c5f: 5f pop %edi
35c60: 5d pop %ebp
35c61: c3 ret
which is something the code might be calling from about five bajillion places.
here's how i found that
perf probe -a "handle_bus_lock"
perf record -e probe:handle_bus_lock --call-graph dwarf -aR sleep 10
perf script
the perf script command also allows you to get a dump of the events raw, which also includes the user-land stack, but i haven't had a lot of success matching the values on the stack to anything helpful
Having this issue too
x86/split lock detection: #DB: CHTTPClientThre/1677386 took a bus_lock trap at address: 0xf3a13c84
getting spammed to my journals when steam is running (don't need to run a game) always the same address too. I'm on cachyos, not ubuntu. So you could say the arch distro family is also affected.
Of course it's possible to get rid of warnings using "split_lock_detect=off" but it doesn't solve the problem.
Well, this conclusion is wrong in a special way: it removes the direct performance penalty applied by the kernel to misbehaving processes. It still doesn't fix the performance penalty that the CPU suffers during a split lock (and actually, the kernel penalty will prevent such processes from affecting the whole system performance).
No it's not wrong, you're thinking of split_lock_mitigate which doesn't affect the warnings but does mitigate the performance issues associated with this problem. Most distros enable this by default though.
I was able to experience this issue milliseconds after Steam is finished updating:
kernel: x86/split lock detection: #DB: IPC:CSteamEngin/3138 took a bus_lock trap at address: 0xf3ac3cbf
kernel: x86/split lock detection: #DB: IPC:CSteamEngin/3138 took a bus_lock trap at address: 0xf3ac3cbf
kernel: x86/split lock detection: #DB: IPC:CSteamEngin/3138 took a bus_lock trap at address: 0xf3ac3cbf
kernel: x86/split lock detection: #DB: IPC:CSteamEngin/3138 took a bus_lock trap at address: 0xf3ac3cbf
kernel: x86/split lock detection: #DB: IPC:CSteamEngin/3138 took a bus_lock trap at address: 0xf3ac3cbf
kernel: x86/split lock detection: #DB: ThreadedValidat/12304 took a bus_lock trap at address: 0xf3ac3c84
kernel: x86/split lock detection: #DB: ThreadedValidat/12304 took a bus_lock trap at address: 0xf3ac3c84
kernel: x86/split lock detection: #DB: ThreadedValidat/12304 took a bus_lock trap at address: 0xf3ac3c84
kernel: x86/split lock detection: #DB: ThreadedValidat/12304 took a bus_lock trap at address: 0xf3ac3c84
kernel: x86/split lock detection: #DB: ThreadedValidat/12304 took a bus_lock trap at address: 0xf3ac3c84
Judging by the reports, the problem is old. Until now, I couldn't identify it in Ubuntu 25.04. In that version, I was using a Ryzen 7 5700G (Zen 3), which doesn't support lock detection. Now, I've updated my PC to a Ryzen 7 9700x, with Ubuntu 25.10. It seems that kernel 6.17 has tightened the detection and logs. After some research, it seems this might be related to Chromium and some libraries that Steam uses, but Chromium itself has already been fixed to avoid this lock. Furthermore, Zen 4+ processors now have this detection at the hardware level, and, along with the tightening of kernel policies, this results in a loss of performance and a waste of space due to the endless number of logs, significantly reducing the performance of other applications besides Steam, which is now penalized by the kernel. The only solution here was to disable detection at kernel startup (which I wouldn't want to be permanent, since it's a useful feature for protecting the rest of the system from poorly written applications). Will we see a fix for this?
Here, my log is full of messages like:
x86/split lock detection: #DB: CHTTPClientThre/37525 took a bus_lock trap at address: 0xe9dddc84.
They are always very similar, reaching more than 100 per minute. This has also made using Steam Link unfeasible. Thousands of logs of this type start the moment I try a remote connection. Until disabling kernel-level detection, using Steam Link was unfeasible, in addition to all the overhead from the logs, which impact the performance of the system as a whole.
Who would we need to talk to address this issue? Steam has this whole thing of being open to Linux, but this affects (even if silently) various devices with different kernel versions and distributions. Since the new Steam Machine uses Zen 4, this will also be a problem if it keeps this detection active (which is the kernel default).
The solution here was to use:
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash split_lock_detect=off" in /etc/default/grub.
The problem persists even after disabling lock detection, it's worth noting. This will only stop the generation of huge logs about the occurrence of locks, but the problem continues. Besides preventing the kernel from penalizing the process creating the locks, it will make the process silent at the user level. The processor will continue to suffer from the locks, and overall performance will be degraded in the same way. The solution needs to be a fix in Steam. I haven't seen any process (I use the PC for work, with various IDEs, tools, browsers, etc.) that does this.
@TTimo @mikela-valve @kisak-valve @frankc-valve @EricS-Valve @johnv-valve @davidw-valve @Plagman
Same here, spams:
x86/split lock detection: #DB: CHTTPClientThre/6830 took a bus_lock trap at address: 0xf3a62634
Running Steam v1766451605 on Kubuntu 24.04 64-bit
Linux Kubuntu-Event 6.18.2-x64v3-xanmod1 #0~20251218.g9f068d0 SMP PREEMPT_DYNAMIC Thu Dec 18 21:27:29 UTC x86_64 x86_64 x86_64 GNU/Linux
(but also with default distro kernel)
Seems not to happend with default Linux Kubuntu-Event 6.14.0-37-generic [#37](/issue/ValveSoftware/steam-for-linux/37)~24.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Nov 20 10:25:38 UTC 2 x86_64 x86_64 x86_64 GNU/Linux kernel (will keep verifying anyway, I swear I got the error some time ago without xanmod kernerl)
And yes, happens with every kernel I tried, Liquorix, XenMod, default .
The problem persists even after disabling lock detection, it's worth noting.
Indeed. As explained in article https://lwn.net/Articles/790464/ the underlying issue is hardware related and the issue is triggered by user mode program (Steam client code by Valve in this situation) using atomic assembly operation (lock cmpxchg8b) on misaligned memory. x86 and x86-64 CPUs can execute this kind of code without logic problems but it will cause HUGE performance penalty because CPU must execute "stop the world" logic on the whole CPU instead of executing code as usual to maintain implicit memory synchronization promise that the hardware platform ISA promises.
Since this "stop the world" issue can be used to artificially slow down other processes running on the same CPU it can be used to cause partial denial of service attack if given code is executed by attacker controlled code. To reduce damage from such attacks, Linux artificially slows down processes using this potential attack vector and logs the event to system log to make it explicit why this slowdown happens.
When you use kernel flag split_lock_detect=off you tell the kernel that this attack vector defense + reporting should be turned off but the underlying performance issue caused by hardware implementation is obviously still in action. You're just hiding the issue partially instead of fixing the underlying issue – namely using atomic instruction on misaligned memory.
If you want to just turn off the artificial slowdown (basically disable attack vector protection) but still get the logs about the issue, you can use kernel flag split_lock_mitigate=0 instead.
If you think "stop the world" logic causing nondeterministic latency in GC languages is bad, this is the same thing but for the whole operating system at hardware level and will obviously destroy any low latency things such as jitter free frame updates.
If Valve tried to execute the same logic using ARM native assembly the process would end with alignment fault (a synchronous data abort). ARM doesn't emulate the required atomic operation with reduced performance but crashes the whole process instead! In addition, ARM doesn't have implicit memory synchronization either.
The only reason Steam client has this issue in the first place is that the operating system that Valve used to create this code didn't have defences against the problematic use case and the original developer didn't notice that their code is working poorly.
The correct fix is to allocate required atomic structure in memory address that's divisible by 8 and this cannot be nicely fixed without Valve fixing their own source code. It might be possible to workaround the issue by using a wrapper code that changes all allocated memory blocks to be offset by suitable byte count to end up with the the locked 64 bit address in 8 byte boundary but this would highly probably result in other memory accesses to be misaligned instead and it would slow down the client overall but avoid this problematic atomic instruction slowdown.
I bet the original developer could easily fix the whole issue if they just were aware about this issue.
And once Valve fixes this issue, the runtime performance of the client should improve on all x86 and x86-64 platforms regardless of the operating system used.
https://github.com/ValveSoftware/steam-for-linux/issues/8003#issuecomment-3690306484
Barely understood your delightful answer :-p I can tell that if you sell me an old car with those lovely words I would pay you double 8-)
(thanks for trying to explain difficult stuff, really!)
And once Valve fixes this issue, the runtime performance of the client should improve on all x86 and x86-64 platforms regardless of the operating system used.
I think that Valve already fixed most if not all code in the Steam Client itself. But the Steam Client is still 32 bit, and it seems to run in a very old Steam Runtime (I think this is what AppId = 0 is for), and that has very old 32 bit libs that Valve probably can't or doesn't want to fix (another user here identified the problematic 32 bit libs). On another note, I think the Steam Client is in the process of migrating to 64 bit, hopefully soonish - and that would explain why Valve doesn't want to fix issues in very old 32 bit libs, and that's fine. As far as I know, it already did switch to 64 bit for Windows. So we might finally get this fixed. If any logs remain after this migration, those would be real issues that still need to be fixed, should it be in the client or runtimes. Any logs produced by game executables would remain and cannot be fixed by Valve.
@mikkorantalainen btw, I love your explanation - very well done.
Hi, dont know why or how... but if I add split_lock_detect=fatal to my kernel boot up parameters, I dont get any dmesg split_lock errors as before.
When I start steam client after booting, it closes one time, I open it once again and works just fine WITHOUT dmesg errors (sorry made a type error before) . I am in XanMod kernel.
split_lock_detect=fatal will kill offending processes. That may as well hit a random game if you keep that setting.
split_lock_detect=fatalwill kill offending processes. That may as well hit a random game if you keep that setting.
Sorry i correct my last post, I get no errors in dmesg with split_lock_detect=fatal (will test more ingame later), can it be that Steam "detects" the split_lock_defect option and disables the offending old x86 code? (sorry too much uneducated guessing :-()
Well, your Steam client got killed on the first try - so it probably works. I don't think there's something like "disabling the offending code" - in that case it wouldn't need to be there in the first place. The kernel should probably log it but just in case, I think you'll get a core dump (see coredumpctl) or Steam catches the core dump and uploads it itself on the next start (should be visisble in one of the Steam log files).
It's still an interesting observation that Steam fails the first start but succeeds on the second start.
Well, your Steam client got killed on the first try - so it probably works. I don't think there's something like "disabling the offending code" - in that case it wouldn't need to be there in the first place. The kernel should probably log it but just in case, I think you'll get a core dump (see
coredumpctl) or Steam catches the core dump and uploads it itself on the next start (should be visisble in one of the Steam log files).It's still an interesting observation that Steam fails the first start but succeeds on the second start.
Hi, yup, strange... I still have no dmesg with the "split error" and run several games yesterday without problem (Steam running in background)
You were right @kakra I got a coredump first time:
[2025-12-26 11:17:40] /tmp/dumps/crash_20251226111740_22.dmp
[2025-12-26 11:17:42] crash_20251226111740_22.dmp[9191]: Finished uploading minidump (out-of-process): success = yes
[2025-12-26 11:17:42] crash_20251226111740_22.dmp[9191]: file ''/tmp/dumps/crash_20251226111740_22.dmp'', upload yes: ''CrashID=bp-da10fa95-9f7a-4e7e-ba5c-2176f2251226''
(if it is useful, I can upload steam logs from empty to what happens on its first run after the crash)
Well... strangely, maybe after kernel update, mesa or ROCm stuff installed, now Steam wont open anymore if I keep the split_lock_detect=fatal kernel paramenter.
Behaviour the same, opens and sometimes closes sooner or later if you click some inside Steam client.
Days ago I have to open it two times, first closes, second just worked :-( so... lost here, will try to learn more about what @mikkorantalainen explaing about split locks.
@sebadamus that's the expected outcome if the process misbehaves when fatal is configured. The likely explanation is steam client update or one of the libraries.
Fresh install of ubuntu with 9700x and AMD 9070 xt. Having this problem and steam wont work properly.
Fresh install of ubuntu with 9700x and AMD 9070 xt. Having this problem and steam wont work properly.
See https://github.com/ValveSoftware/steam-for-linux/issues/8003#issuecomment-2974719562
Jesus, it's been 4 years, how has this not been fixed yet?
Boot your system with
split_lock_detect=offon the kernel cmdline. This mitigation has no useful purpose on single user systems. It's a countermeasure for multiuser / virtualization hosts with untrusted users.
I'd say Steam/Valve should correct this.
I just installed Steam on a Suse Tumbleweed with update 20260216-0.
Linux newmachine 6.18.9-1-default #1 SMP PREEMPT_DYNAMIC Fri Feb 6 18:53:03 UTC 2026 (6d9f8a8) x86_64 x86_64 x86_64 GNU/Linux
I did not even play anything, just started Steam and got:
...kernel: x86/split lock detection: #DB: CHTTPClientThre/66005 took a bus_lock trap at address: 0xf37766d4
in the journal.
Btw. you can add "Distro Family: Suse" here.
Apropos:
"The problem with atomic operations that cross cache-line boundaries is that the system bus must take special measures to ensure that both cache lines are simultaneously protected from concurrent access. In practice, that means locking the bus for the duration of the operation, which can stall every other processor in the system. A malicious program executing a tight loop with a split-lock operation can destroy the performance of the system as a whole. For this reason, split-lock operations have long been frowned upon...
The definitive answer, though, came from Thomas Gleixner, who pointed out that slowing down split lockers by default is the only choice that distributors could make; anything else creates an easily exploitable denial-of-service vulnerability. So the slowdown needs to remain: "Attack vector prevention has precedence over broken applications". "
https://lwn.net/Articles/911219/
I did not even play anything, just started Steam and got:
...kernel: x86/split lock detection: #DB: CHTTPClientThre/66005 took a bus_lock trap at address: 0xf37766d4
in the journal.
Same.
I'll try to see if I get the same issue with the xanmod kernel (just in case its patch set can change anything).
Note also that I never noticed those errors in dmesg logs with kernel 6.12.47 and 6.12.63, but I started to see them with the kernel 6.18.12 that I compiled today.
I have identified the same problem on Arch Linux, running Niri with DMS
-`
.o+` root@Trooper
`ooo/ OS: Arch Linux
`+oooo: Kernel: x86_64 Linux 6.18.16-1-lts
`+oooooo: Uptime: 1d 6h 6m
-+oooooo+: Packages: 1259
`/:-:++oooo+: Shell: bash 5.3.9
`/++++/+++++++: Resolution: No X Server
`/++++++++++++++: WM: xwayland-satellite
`/+++ooooooooooooo/` GTK Theme: Adwaita [GTK3]
./ooosssso++osssssso+` Disk: 4.0T / 20T (21%)
.oossssso-````/ossssss+` CPU: AMD Ryzen 9 9900X 12-Core @ 24x 4GHz
-osssssso. :ssssssso. GPU: Advanced Micro Devices, Inc. [AMD/ATI] Navi 32 [Radeon RX 7700 XT / 7800 XT] (rev c8)
:osssssss/ osssso+++. RAM: 36202MiB / 63938MiB
/ossssssss/ +ssssooo/-
`/ossssso+/:- -:/+osssso+-
`+sso+:-` `.-/+oso:
`++:. `-/+/
.` `/
I can confirm I did not have this prior to the upgrade to kernel 6.18.16-1-lts:
#> grep -rin linux-lts /var/log/pacman.log | grep 6.18.16-1
9710:[2026-03-09T16:24:40+0100] [ALPM] upgraded linux-lts (6.12.74-1 -> 6.18.16-1)
9711:[2026-03-09T16:24:41+0100] [ALPM] upgraded linux-lts-headers (6.12.74-1 -> 6.18.16-1)
#> time journalctl -U '2026-03-09T16:24:41' | grep -c CHTTPClientThre
0
real 0m10.148s
user 0m10.047s
sys 0m0.243s
#> time journalctl -S '2026-03-09T16:24:41' | grep -c CHTTPClientThre
1171
real 0m0.083s
user 0m0.071s
sys 0m0.014s
Note: my journals go quite a long back:
#> journalctl --list-boots
IDX BOOT ID FIRST ENTRY LAST ENTRY
-56 0dd74cf7996c40fc9bc07191f0f48ae5 Mon 2025-07-21 23:50:40 CEST Wed 2025-07-23 16:25:27 CEST
[...]
using steam 1.0.0.85-5:
Steam Beta Branch: Stable Client
Steam Version: 1773099986
Steam Client Build Date: Tue, Mar 10, 2026 00:19 UTC -08:00
Steam Web Build Date: Tue, Mar 10, 2026 00:36 UTC -08:00
Steam API Version: SteamClient023
This should be fixed once the Steam Client on Linux switches to 64 bit. The split lock happens in 32 bit components of an old Steam Runtime, it seems. Hopefully, this switch happens sooner than later.
Until then, people should set split_lock_detect=off on the kernel cmdline under one of two conditions:
- games causing split locks, because then games will be slowed down as part of the split lock mitigation (Valve cannot do anything about this, rather game devs would need to fix it)
- split lock logging overwhelmingly spams the system logs causing IO overhead to be higher than the split lock mitigation impact or the split locks itself
Personal desktop systems are usually not subject to these kinds of denial of service attacks, so it's safe to enable the work-around. You should not enable it on systems running potentially untrusted code, like VM/container hosts or web hosting infrastructure, where anonymous access from internet can directly cause CPU activity on the host. Again, on a personal desktop system or other closed/walled garden systems, it is safe to disable the mitigations (read: enable the work-around).
The only downside then is, that if some software unexpectedly causes split locks, it would go unnoticed and you're not notified that the causing software component will cause slow downs of other software components due to an inefficient CPU bus lock situation.
Once Steam Client switches to native 64 bit, the work-around can be removed.
IMHO, Valve has two options here:
- fix the 32 bit component (it has been identified earlier in this thread but might not be the only one) by aligning the lock properly in memory
- migrate the client to 64 bit
Both seem easy but I'm pretty sure it would have already been solved if it had been that easy. Adjusting old 32 bit components can cause a lot of spurious follow up bugs or behavior changes, so it might explain why this component hasn't been simply fixed. Switching the client to 64 bit might need a lot of edge case testing and fixing, so it has a big quality control and testing overhead.
Hi,
For more information why is that and what is that: https://lwn.net/Articles/911219/
I only recently started getting these in my log, likely from a kernel update in Debian unstable. I rarely have had some performance issues so interesting to see are the errors related to that, and thanks for linking potential workarounds here. Just wondering is it common that these messages might appear without significant performance issues accompanying?
I only recently started getting these in my log, likely from a kernel update in Debian unstable. I rarely have had some performance issues so interesting to see are the errors related to that, and thanks for linking potential workarounds here. Just wondering is it common that these messages might appear without significant performance issues accompanying?
The messages are actually there to tell you that a performance degradation has been mitigated. This means:
- The causing process is the one which needs to be mitigated, to not throttle performance of other process. This is because split locks cause bus locks on modern CPUs which degrades multithreaded performance. A potential attacker could use this to reduce the performance of the whole system. The mitigation is to throttle/pause the causing process. This is good as long as the causing process isn't your game (or an external process which the game has to wait on).
- It is mostly useful for multi-user systems or if you cannot or don't want to trust the code executed, e.g. cloud providers which run VPS for random customers. So its usefulness is limited on single-user desktop systems.
- If it is the game itself that is causing this, you should turn mitigation off because it will throttle the game. Today, you will most likely only encounter it with 32 bit games.
- If it is the Steam client, it should be fixed once the 64 bit client is pushed to public (I think it's in preview now). Some old 32 bit components of Steam still cause split lock situations. These are mostly harmless and don't affect your games, even better: The mitigation successfully preserves better game performance because too frequent split locks will be avoided.
- If split lock logging becomes too spammy, the IO overhead may start causing more performance loss than just not mitigating the split locks at all.
So YMMV, depending on your situation. If you don't see the logs often, and if you don't see them often while gaming, everything is fine. But if the logs are very spammy, you should silence them for now. There are two ways: Either turn mitigation off (performance impact depends on what process causes it), or just turn logging off (prevents performance loss by IO overhead).
Once the 64 bit client is released, re-evaluate.
If it is the game itself that is causing this, you should turn mitigation off because it will throttle the game. Today, you will most likely only encounter it with 32 bit games.
Or those games could be executed in some sort of "split lock" sandbox.
What I'm thinking about is that any binary code that would cause split lock should be recompiled (yes recompiled, don't laugh) into some sort code that wouldn't cause split lock.
You can see for instance FEX-Emu that feature such a recompiler (it is able to recompile x86-64 into arm64):
https://fex-emu.com/
Or those games could be executed in some sort of "split lock" sandbox.
The split lock is cause by CPU internal state so there's no way to run any kind of sandbox that executes the same code without affecting the rest of the system. If you cannot fix the software you're running and you trust all the software running on your computer, use the kernel flag split_lock_detect=off to workaround the performance issue caused by the split lock detection (which intentionally penalizes the process that causes split locks to avoid slowing down the rest of the system because of poor code).
If your system is running only single user tasks (e.g. desktop + game) then slowing down the system to improve throughput of other programs doesn't make sense and it's better to turn off the mitigation to make the poorly written software run as well as possible on your hardware. (That is, turning off the mitigation doesn't fix the software but that's the best thing you can do performance-wise since you cannot fix the broken software.)
@mikkorantalainen I don't think you understood my statement.
The split lock is cause by CPU internal state
I already know.
so there's no way to run any kind of sandbox that executes the same code without affecting the rest of the system.
I already know.
[...] to make the poorly written software [...]
[...] since you cannot fix the broken software. [...]
Who said you can't fix it?
What I was really talking about is recompiling the poor code ahead of time in some code that wouldn't cause split lock. The example I gave was that nowadays recompilers allows executing games compiled for a specific architecture like x86-64 on another architecture like arm64. The one I FEX-Emu:
On the technical side, FEX features an advanced binary recompiler that supports all modern extensions of the x86(-64) instruction set, including AVX/AVX2.
To my understanding executing a "recompiled" software isn't completly different from executing a software in an emulator.
Here's another example of a recompiler:
https://github.com/rexdex/recompiler
This one allow recompiling binary code compiled for the Xbox 360 PPC64 CPU into binary code compatible with Windows.
Basically, what I was thinking about would be recompiling "poor x86_64" binary code into "good x86_64" binary code. That "good x86_64" wouldn't cause split locks and would be as efficient as the original code that is causing split locks.
since you cannot fix the broken software. [...]
Who said you can't fix it?
"You" as in end user trying to run the software. If the vendor of the software cannot bother to recompile the software, we (as the end users) cannot fix the software. We either use the workaround to turn off the mitigation or suffer from the broken software getting even slower in return for preventing slowing down the rest of the system.
The split lock is cause by CPU internal state so there's no way to run any kind of sandbox that executes the same code without affecting the rest of the system. If you cannot fix the software you're running and you trust all the software running on your computer, use the kernel flag split_lock_detect=off to workaround the performance issue caused by the split lock detection (which intentionally penalizes the process that causes split locks to avoid slowing down the rest of the system because of poor code).
Probably not since it's down to the kernel level, but:
Could we flag, or mark in some way the specific binary / process off the detection of split_lock, but not everything?
I understand the concept (I think) but that's too low a level for me ^^
Could we flag, or mark in some way the specific binary / process off the detection of split_lock, but not everything?
I'm not sure what this should solve. The causing process will still slow down the CPU for all processes, no matter if you disabled or enabled the mitigation, no matter if per process or system wide. The only difference is that the causing process will no longer be throttled, thus it is causing a higher impact on the whole system.
That being said: I'm pretty sure you can dynamically enable and disable mitigation in current kernels: You could just disable it while running a game, e.g. by using a wrapper script which first stores the current state and disables mitigation, then after the game exited, it would restore it.
I don't think that a "per process" control would do what you expect: Why would you want to disable mitigation for a game, if really the causing process is not the game, e.g. an old Steam runtime component running as a background process?
Additionally, on the other idea of dynamic recompilation which seems tempting: I think the overhead you get from that is much worse than the performance penalty from split locks - no matter if they are mitigated or not. Dynamic recompilation is better than emulation but it is not free. And it is also very complicated.
I think it's just almost always better to turn off split lock mitigation on single user systems which are meant to play games. We shouldn't try to overengineer stuff and create complicated solutions for simple problems. And split locks on single user gaming systems are a simple problem because it does affect only the single user of the system. It's not like your system is a shared cloud hosting platform where split locks could be exploited to cause denial of service attacks.
since you cannot fix the broken software. [...]
Who said you can't fix it?
"You" as in end user trying to run the software. If the vendor of the software cannot bother to recompile the software, we (as the end users) cannot fix the software. We either use the workaround to turn off the mitigation or suffer from the broken software getting even slower in return for preventing slowing down the rest of the system.
I don't think you understand my point. I never said the vendor of the software would have to fix the broken poorly written source code of the software that are causing split lock and then publishing a new version. I said that we as end users could create and use locally our own "binary code to binary code" recompiler (like FEX-EMU or rexdex recompiler -- they don't need the source code of the original software) that would be able to rewrite (from binary code, not from source code) the part of those software that are causing split lock.
use the kernel flag
split_lock_detect=offto workaround the performance issue caused by the split lock detection
Your reasoning is correct but the statement is technically not: having split lock detection on actually prevents a performance issue which affects the whole CPU, turning it off will cause performance issues. The resulting effect is that the process that causes it will be heavily penalized to mitigate slowing other processes down - simply by artificially reducing the rate at which split locks occur. Yes, this causes a performance issue for the causing process itself while maintaining better performance for all other processes. The other downside is that the excessive logging of the incident may cause IO overhead in the system logger, making the system appear slower. On the average gaming desktop system, the performance overhead caused by split lock mitigation and logging can actually be higher than the split locks itself, especially if an affected process actually interacts with a process you expect performance from, e.g. the game communicating with Steam services which become mitigated. I'm not sure if the latter is a real problem.
Also, I think this is mostly caused by old 32 bit components, e.g. using an old Steam runtime, thus the 32 bit Steam Client itself may be affected, and native games running inside old Steam runtimes may be affected. Setting such games to a current Proton runtime prevents running the native Linux version of such games, and is probably a solution which comes closest to the idea of "dynamic recompilation" which has been suggested before.
If I may, all the conversations here, starting from the opening post, refer the steam binary under Linux, and that this process generates split lock.
In a server, that steam process should be used isolated, or be the main service, either way, split lock should then be isolated (or does it impact a hypervisor as well?) or, as the main service, you want it to have the best performance, even if it impacts the rest, so split lock is not a concern (albeit the split lock impact on itself).
On a gaming computer, as Kakra suggested, the kernel flag is a thought to have, as this remains the main process, and just like my previous point, would be the focus.
If I suggested a single process exclusion from the split_lock_detect flag, it is for that remaining case (like my computer) running my own little cloud, but still playing on it as well:
- I do not care if a split_lock occurs while I play (or at least, I want that process to be able to use available resources and not be slowed down by overwhelming spams)
But, if a new process creates split lock, outside the one I specified, I want to know that.
Note: I am not suggesting that a solution for split_lock is not required, just a way to diminish the impact on some individual, targeted process / use case scenario and still be able to detect any other process triggering split lock.
Of course, I would much prefer if no split lock occurred, but that is outside my control.
Side question: if you recompile a binary, it would definitely have a cost. But, once that is completed, wouldn't that cost be removed (you get the newly compiled binary, without split lock)? OC any new version would require a recompile, until the source of that split lock is dealt with.
In a server, that steam process should be used isolated, or be the main service, either way, split lock should then be isolated (or does it impact a hypervisor as well?)
It affects the CPU on the hardware level, thus it affects the hypervisor and also neighbor VMs - and exactly that is the problem why the Linux kernel takes such an aggressive approach to mitigate that. The target of split lock mitigation has never been desktop systems, it always targeted cloud hosters, multi-user server systems, etc. As far as I understood: A split lock is a lock taken across different cache lines (due to bad data alignment), which forces the CPU to lock the whole bus. That in turn prevents other cores / threads from properly progressing if they need to do anything that needs cache coherency (memory barriers, independent locks in other threads, etc) while the lock is taken.
So, given you have a malicious actor in a VM causing thousands of split locks on purpose, it will slow down the whole machine, affecting other VMs on the same host, and affecting the host itself. That's why split lock mitigation pauses the process which caused the split lock for a few milliseconds, essentially reducing the rate of split locks by a factor of 100, 1000 or even more.
Split lock mitigation is not there to affect your desktop / game performance or make things better or worse there just because some old library causes split locks as a side effect of running on a modern CPU. It is there to prevent large scale denial of service attacks on cloud services.
Also, the problem isn't new - CPUs are affected by that since years, probably since we got 64 bit CPUs. But younger CPUs introduced a feature to actually report the split lock situation to the OS kernel so it can do whatever it wants to mitigate the issue. Turning mitigation off will just pretend that your CPU doesn't report split locks - just like your previous CPU probably did. No one ever cared about split locks until recent kernels mitigated what recent CPUs are now able to report.
But, if a new process creates split lock, outside the one I specified, I want to know that.
That's why you may want to log that, and even mitigate that: A process outside of the game would be slowed down if it causes a split lock, preventing it from taking another split lock within the next few milliseconds, thus mitigating the effects of a CPU performance slow down due to repeated split locks.
running my own little cloud
If you mean by that: running something like NextCloud or Syncthing on your machine, then no fear: These are not the kind of services affected by split locks. The problem comes when users of such services can execute their own code, e.g. if you run a VM where people can login and execute custom code. In theory, it can also happen if a malicious actor runs specially crafted javascript in your browser (via a malicious website or browser plugin), tho I think the latter attack vector is much more unlikely because it serves no real purpose for an attacker - at least on your typical gaming setup.
I highly doubt that you run cloud service VMs on your gaming rig where random internet users (potentially malicious users) could login and do stuff like executing code they compiled or downloaded.
OC any new version would require a recompile
If there happens such a thing as "any new version", then split locks wouldn't be the problem in the first place: If some software gets an update, it will most likely be compiled or programmed in a way to not cause split locks. Modern compilers and programming libs usually take care of that automatically. This has been different in the old 32 bit days, when data alignment has been a bit more lax and not performance critical.
BTW, for the same reason you probably want to disable all the other server-grade CPU security mitigation like SPECTRE, RowHammer etc on your gaming system, because those mitigation cause really high performance costs. A kernel which enables all memory security mitigations, especially on older CPUs, will make the system feel much slower, and have a real performance cost while gaming. Modern CPUs have hardware acceleration for most memory security mitigations but the cost is still there and real.
Thank you for this in-depth explanation !
That's why you may want to log that, and even mitigate that
That was kind of my point: I want to know whether a process generates split-lock, but I want also be able to accept and silence the log for that process:
- any process generating split-lock -> logged
- a specific list of user approved -> no log
But It doesn't seem feasible?
running something like NextCloud
No, it's more complex than that, but still private, so no risk indeed (trying to create a complete private network with DHCP/DNS/forgejo/Talos/openvswitch on incus with Ansible).
If there happens such a thing as "any new version"
So you're suggesting that the next Steam version will not have the same split-lock problem, sadly I doubt it, until they release a 64 bit version.
But it was more of a question: Could we recompile a binary (once) to remove this split-lock as F3llFr0mTh3Sky suggested?
That means that each call would need to be rewritten to be made correctly: that sounds more like an encapsulation?
Too low level for me as I said ^^
Posting this again as some appear to have missed it - https://github.com/ValveSoftware/steam-for-linux/issues/8003#issuecomment-2974719562
That was kind of my point: I want to know whether a process generates split-lock, but I want also be able to accept and silence the log for that process:
- any process generating split-lock -> logged
- a specific list of user approved -> no log
But It doesn't seem feasible?
This sounds like a good approach, so we can have the good of both worlds: Stop performance degradation by known "good" processes spamming the logs (and which we cannot fix) but still get informed of remaining processes which need fixing.
So you're suggesting that the next Steam version will not have the same split-lock problem, sadly I doubt it, until they release a 64 bit version.
You can already try it by opting in to the beta client, then enable the experimental SteamRT3 client in the UI settings. But while it mostly works now (systray icon works now, running games with custom commands works now) I still found some flaws (some games cannot play videos and show the placeholder test video instead). But it would be worth a try to see if the log spamming is gone then. If it is, we just need to wait until all the remaining edge cases are ironed out.
But it was more of a question: Could we recompile a binary (once) to remove this split-lock
I think this is very complicated. Many games probably have encrypted code and anti-tampering measures, let alone complex dependencies between all the various data stored in RAM. Doing a recompile (however that might work in theory) may even accidentally fix behavior bugs that the game relies on. Games actually have a lot of bugs that wine already needs to work around, like use-after-free or misused locking, which happens to work just fine on Windows but not in a clean-room implementation like wine. Wine actually has to implement some buggy behavior on purpose. Recompiling a binary will most likely change that behavior.
The idea, however, is not completely theoretical: If you don't have the source, you could still argue that the binary code could serve as some kind of source code - and that's actually how .NET or Java work. Java (and .NET which is more or less a Microsoft implementation of the Java idea) compile source code to an intermediate binary representation which is an abstract representation of the code just before converting it into native CPU code. Such programs thus run inside a VM (JVM for Java) or use JIT compilation (I think .NET does that): It's quite a simple process at that point because all the heavy lifting of compilation has already been done by the compiler except the last step: converting code to native and optimized CPU instructions.
The problem here is: What has been suggested is already past that in-between point - it is already native code, including all the CPU-specific optimizations and register elimination. It will make the process complicated again because a lot of useful meta information from the code is gone at this point - no more knowledge about variables or their purpose etc, no more information about memory layout or alignment, or rather why it is there.
But yes, software like FEX probably does exactly that. But the effort is huge. We should just wait until Steam fully migrated to 64 bit. Next is wine which will have a new native 64 bit mode where it no longer needs 32 bit libraries but maps 32 bit programs into native Linux 64 bit address space. This means you will no longer have to provide native 32 bit libs in Linux, the old Steam 32-bit runtimes are no longer needed. It should all help getting rid of the remaining split locks.
Of course, if you continue to run very old native 32-bit Linux games, the situation may still exist. I'd they: Either run them in Proton instead (they will most likely run better there because libs are better optimized and can use modern features), or only then re-evaluate if some FEX-like approach would be feasible.
The kernel will log Steam / Cef lock splitting, but without the sequential access penalty.
@Strykar I think you can disable the logging separately.
There's also a rate limiting now: https://www.kernel.org/doc/Documentation/arch/x86/buslock.rst
You can actually say now: "okay, up to 5 incidents per seconds are okay, beyond this penalize the process". I wish there would be a rate limiting for logging, too, because that's what more likely impacts gaming performance. After all, this incident will happen on Windows, too, if games cause this, and they are designed around the performance impact that this has. I don't think that Windows has an active mitigation for that, so turning mitigation off would come closest to how Windows behaves.
The document suggests that a rate limiting of 1000 would have a performance impact of 0.35% with a 2 Ghz core (IOW, bus locks up to a 1000 incidents per second have only 0.35% performance impact). So I think anything in a range of 5-1000 could be a good start. That way, you should be able to stop the log spamming but still have a log if something really goes crazy in your system.
I apologize if someone else already brought this up, but just in case, I would just like to add:
The process and memory address repeatedly spamming the console, shows the bus lock instruction comes from the shared object located here:
~/.local/share/Steam/ubuntu12_32/libtier0_s.so
You can test this yourself, plugging in the PID and memory address from the dmesg log:
pmap -pA OFFSET PID
I apologize if someone else already brought this up, but just in case, I would just like to add:
Thanks, and no worries. Yes, it has already been identified. It's an old 32 bit dependency in Steam which is probably not easy to fix, and once Steam switched to 64 bit, this lib should no longer be used.
This isn't just log noise — it's measurable gameplay stutter in CS2.
CHTTPClientThread (steamclient.so, loaded inside the cs2 process) performs
misaligned atomics that trigger x86 bus_lock traps in bursts. On default kernel
settings this causes periodic multi-second fps degradation; even with
kernel.split_lock_mitigate=0 the trap storms still correlate with frame spikes.
Measured data (MangoHud per-frame CSV + kernel journal, 400s CS2 session):
- Hardware: Ryzen 7 9800X3D, RX 7900 XTX (RADV), kernel 6.18.34 LTS (CachyOS), KDE
Wayland - 745+ bus_lock traps logged in ~6.5 min, in bursts — handle_bus_lock: 586
callbacks suppressed in a single ~5s window - All traps from CHTTPClientThre, always the same address (0xf37a7c54)
- Median frametime 3.12ms (fps_max 320); 16 frametime spikes of 10–24ms captured;
the 3 largest spikes (19.8ms, 22.4ms, 21.9ms) each coincide with logged bus_lock
bursts (±0.7s); kernel log ratelimiting hides traps near the others - With default split_lock_mitigate=1 the impact is far worse: the kernel sleeps the
offending thread, producing 1–2s fps drops every 5–10s whenever the HTTP client is
active (matchmaking heartbeats etc.)
Workaround: booting with split_lock_detect=off (removes the traps entirely). Most
competitive players on Linux will never diagnose this — it just feels like "CS2
stutters on Linux."
Fix request: align the atomic operand in CHTTPClientThread's hot path so it doesn't
cross a cache line. One struct alignment fix in steamclient.so would remove the
issue for every title that embeds the Steam client library.
@aticicaner I think this is a good analysis, especially how log ratelimiting hides nearby traps, and the differences between different mitigation and detection options for the kernel. My take is still: A gaming focused kernel should turn detection completely off by default. It's rather a debugging tool for developers.
But I don't think the solution is as easy as aligning the atomic operand: The problematic library lives in 32 bit space and is very old, it probably needs to retain compatibility with old native games. It is part of an old Steam Runtime. Sure, the Steam Client should probably not use such an old library - but that is on its way to being fixes once the Steam Client switched to 64 bit. It would be interesting to see if this is fixed when you switch to the 64 bit preview client in Steam.
OTOH, I wonder if rebuilding the problematic library within the Steam Runtime build environment, including a proper patch, could really solve this. But in the end, it's probably based on a really old GCC toolchain which used different aligning by default anyways. So it never will have the full performance potential on modern CPUs. From my experience, changing alignments in old 32 bit software usually leads to crashes somewhere else in the stack. This has been an issue in Gentoo where various 32 bit solibs from Steam Runtime or native games caused crashes because Gentoo switches to modern alignment with updated GCC toolchain. I had to manually force back the old alignment to make it stable again. Luckily, this is fixed these days. So yes, it is probably possible to fix the alignment, but it's also probably not as simple as fixing this single atomic operand...
My initial suggestion about 5 years ago (I don't recall the exact comment, it may not be in this thread) has also been "simply recompile with proper alignment" - but I've made up my mind since then and it is probably not that simple. I'm pretty sure it would be fixed already if it were that simple.
That said, I wonder if the non-native CS2 version (running it through Proton instead) would show the same behavior. Overall FPS may be lower but the stutters may go away? Maybe CS2 should be raised to a new Steam Runtime version? I think you can force specific Steam Runtime versions as the compat layer, too, not only Proton.
This has to be restated: Steam Beta with SteamRT3 enabled solves this issue. We just have to be patient while that reaches the stable branch. Nothing else to do from this issue perceptive since it goes out of scope.
Your system information
CPU:
model name : 11th Gen Intel(R) Core(TM) i7-1165G7 @ 2.80GHzGPU:
52:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)Driver:
460.91.03Please describe your issue in as much detail as possible:
While Steam is open, my syslog is spammed with
x86/split lock detectionthus:If I close Steam, it stops.
Steps for reproducing this issue: