Why open this issue on the dxvk tracker? You wrote a whole lengthy post yourself to point out that this issue isn't exclusive to dxvk and doesn't happen with dxvk on windows drivers.
Why open this issue on the dxvk tracker? You wrote a whole lengthy post yourself to point out that this issue isn't exclusive to dxvk and doesn't happen with dxvk on windows drivers.
Because
If windows dxvk isn't affected then this seems something about linux in general
The only thing that helped was using WINED3D instead of DXVK when OpenGL was used. WineD3D Vulkan backend has the same impact.
Likely something with the vulkan drivers/presentation
It was discussed a bit on Discord* and there are some things dxvk probably could do better in in regards to eGPUs (afaiu the devs. Correct me if I'm wrong).
So the issue should be fine to track it.
Edit: meant Discord and not GitHub
If windows dxvk isn't affected then this seems something about linux in general
The only thing that helped was using WINED3D instead of DXVK when OpenGL was used. WineD3D Vulkan backend has the same impact.
Likely something with the vulkan drivers/presentation
I took an entire month of testing and it's still strange because it happens on 99% of times when dxvk is involved, the performance drop in Windows was just not as hard as it was on Linux, considering the NVIDIA card had less benefits overall with Vulkan. On discord we were talking about how much DXVK had an impact for PCIe bandwidth
Also, I may be CPU bottlenecked when the eGPU was used since some games (BG3 Vulkan for example and being one exception, still only on linux) were on the edge, I don't play new games overall so I was not able to create a decent series of tests. VKD3D was affected as well anyway.
Sorry I misclicked again the close with comment, I'm typing and reading from my phone in the most uncomfortable way ever🤦♂️
I had a bit of free time and decided to really get into figuring out the issue.
There seems to be a lot of different opinions on the issue especially on discord that's why I'll just show the tests I made.
As I said, I managed to get an RTX 3060 and the same exact behavior is shown, same games show the same performance issue, with the same exceptions, being DOOM Eternal working surprisingly well, overall DXVK/VKD3D games are much more affected rather than native Vulkan ones, even if BG3 shows much worse performance on both DXVK and Vulkan, still only on Linux.
Out of curiosity, I also tried a desktop with the AMD card and even forced PCIe 1 x4 speeds (and re-forced by the kernel module option) and the performance is infinitely better than anything tried since now. They are playable compared on Linux with the TB3 eGPU
For comparison, the RX 7800 XT on the desktop:
Cyberpunk 2077:
103 fps~ Ultra settings, PCIe 3 x16
80 fps~ Ultra Settings, PCIe 1 x4
110 fps~ Low Settings, PCIe 1 x4
Also whats really strange is compared to the eGPU, on the desktop with limited pci bandwidth changing video settings did actually affect the performance by a lot.
Also tried disabling/enabling Resizable BAR and above 4G decoding on the bios, and didn't change that much
Laptop
Cyberpunk 2077: (Already posted the results)
24 fps~ Ultra settings, PCIe 3 x4
24 fps~ Ultra Settings, PCIe 1 x4
30 fps~ Low Settings, PCIe 1 x4
ReBar not avaiable on the laptop BIOS options as enabling or disabling above 4G decoding
The only noticeable difference was an higher CPU usage in both when lower PCIe speeds were set.
My laptop has an i7 11375h while the desktop has a much better CPU (Ryzen 5 5600) but still, even windows with dxvk runs much better.
Either the BIOS really doesn't downgrade the PCIe speed, even if I confirmed it by looking at lspci
How could it be? I'm speechless at this point, I don't even know how to verify the bottleneck here, maybe there really is something wrong with the thunderbolt driver itself rather than the everything else already thought.
Maybe you can try downgrading the PCI speeds and share the results on a weaker CPU?
Just to confirm, when using the eGPU, is the monitor plugged directly into the eGPU? Also, when on Linux, what DE and are you using Xorg or Wayland?
I'm not sure if that's specific for SnowRunner, but enabling/disabling d3d11.dcSingleUseMode has a huge effect on my eGPU setup (TB3 + Sonnet Breakaway Box 650 + RX6700XT on Framework 13 13th Gen Intel, Ubuntu 23.10, GNOME Wayland running on iGPU, game is running on external monitor plugged into the eGPU):
d3d11.dcSingleUseMode=False (was set to that in dxvk.conf by itself for some reason) on Low settings is around 20 fps with 40% GPU load:
d3d11.dcSingleUseMode=False on Ultra settings is around 15 fps with 50% GPU load:
d3d11.dcSingleUseMode=True on Low settings is around 45 fps with 70% GPU load:
d3d11.dcSingleUseMode=True on Ultra settings is around 38 fps with 70% GPU load:
I'm not sure what this setting changes exactly, but it has an observable effect on both GPU load and game FPS.
That dcSingleUseMode=False is slow is expected, but it has nothing to do with this issue. It is set to false by default only for 5 games in total, so those probably have been found to rely on it at some point and will break without it.
I have noticed that the problem occurs mostly when the game uses an excessive number of draw calls (more than 4000-5000 draw calls per second).
The worst case I have seen is on Civilization VI. During the graphic benchmark, peaks of 14000 draw calls per second (!) are reached. With an Nvidia RTX 2060 Super eGPU I barely reach 15 fps using DXVK (even on Windows) compared to 70 fps with native DX11.
Can't DXVK be optimized to cope with these situations of large numbers of draw calls saturating the limited eGPU bandwidth?
Thank you.
Something tells me that on Windows, the GPU driver knows that an eGPU is being used, and optimizes certain things for this case.
I don't have time (and don't want to install Windows on my hardware) to test this, but I think what can be done to verify this is:
In my tests, 2 and 3 are pretty much the same, and I did not measure 1. Something tells me that's the case though.
If that's the case, I don't think much can be done on DXVK's side, it should be Mesa's problem.
I want to chime in as this has been bothering me for a few years now.
I use a Razer Core X Chroma eGPU enclosure. I've gone through multiple GPUs - GTX 1080Ti, RX 580, RTX 3090, and currently - RTX 4090.
I've always had issues under Linux. The kernel version didn't matter, the GPU driver version didn't matter. The performance is just worse. Except for a few games or some lightweight games that don't utilize a lot of performance.
The problem might be coming from the Thunderbolt stack in the kernel and not from DXVK, as discussed here: https://gitlab.freedesktop.org/drm/amd/-/issues/2885#note_2295154
Relevant Bugzilla thread: https://bugzilla.kernel.org/show_bug.cgi?id=218525
Even if it isn't related, here's my data if it's of any use.
I currently have dedicated around 700GB for a Windows partition so here's my tests:
One thing that ran close to Windows' performance while running on OpenGL natively on Linux was Unigine Heaven benchmark. The results are posted in the Bugzilla thread linked above.
Tested with Windows 10, Windows 11, Gentoo, Debian 12, Fedora 39, OpenSUSE Tumbleweed, Nobara 39, ArchLinux. The same results were observed on all Linux distributions.
Kernel versions varied between 5.15 and 6.7.9. Nvidia driver versions varied between 515 and 550.
I use Lutris and Steam to play games, with the exception of FFXIV which now has a native Linux launcher obtained either via Flatpak or native compilation.
I am upping the thread because I found the solution and it might be helpful to others.
Use the environment variable DXVK_CONFIG=d3d11.cachedDynamicResources=a
(or add d3d11.cachedDynamicResources=a to the dxvk.conf file as described in the DXVK wiki) and magically the gpu usage and the fps will raise up :)
@gipawu Which games did this work with for you?
@K0bin I have tried F1 2020 (DX11 mode), Civilization VI (DX11 mode) and eFootball PES 2020, all three with positive results.
I am currently unable to test other games.
The suggestion from @gipawu made Guild Wars 2 actually playable on Linux with an eGPU. It's definitely not as performant as on Windows, but before the fix I was getting 15-18 fps max no matter which settings I chose. Now I'm getting 35-45 fps.
Update:
I managed to get above 70 fps in some open world areas, but the Mage Tower area drops the fps to 15-20, and I can get more in that specific area on integrated graphics. Still not ideal, but way better than before overall.
I also managed to test FFXIV - it also received a lot of improvement. The FPS is way more stable and doesn't drop as low as before.
WoW didn't see any performance improvement whatsoever.
Baldur's Gate 3 also received some small improvement here and there.
Overall pretty happy with the results!
The suggestion from @gipawu made Guild Wars 2 actually playable on Linux with an eGPU. It's definitely not as performant as on Windows,
I noticed you mentionining in the kernel bugzilla that switching to AMD has removed this performance bottleneck.
Is it still the case or is there a performance drop on AMD as well?
The suggestion from @gipawu made Guild Wars 2 actually playable on Linux with an eGPU. It's definitely not as performant as on Windows,
I noticed you mentionining in the kernel bugzilla that switching to AMD has removed this performance bottleneck.
Is it still the case or is there a performance drop on AMD as well?
Moving from a TB4 laptop with an i7-12700H to a Framework 13 AMD Ryzen 7 7840U and USB4 definitely improved the performance of games. Running the same Gentoo setup (with rebuilt packages after the switch), I got from non-playable, to more than just playable. Phasmophobia is another game I couldn't play on Intel, but I can comfortably play on AMD. Baldur's Gate 3 is the other one that was literally garbage on Intel.
Overall I'm not that close to Windows numbers, but I get close in most occasions.
Edit: To clarify, all the time I've been using an Nvidia RTX 4090. I'm tempted to sell it and try with an AMD RX7900XTX, but I doubt it'll happen anytime soon.
I am upping the thread because I found the solution and it might be helpful to others.
Use the environment variable
DXVK_CONFIG=d3d11.cachedDynamicResources=a(or addd3d11.cachedDynamicResources=ato the dxvk.conf file as described in the DXVK wiki) and magically the gpu usage and the fps will raise up :)
Nice catch, looks like (at least for a couple of games) the issue is completely gone.
Here are some results
Killing Floor 2: From 41~ Fps to 142~ Fps
Far Cry 5: From 48 fps~ to 92~ fps
Gpu usage is stable and over 80~90%
However, that flag, as the name suggests, doesn't seem to work on older games like Half Life 2 (DirectX 9), which for some reason still runs like garbage
For d3d9 and d3d8 games you have to use d3d9.cachedDynamicBuffers = True
For d3d9 and d3d8 games you have to use
d3d9.cachedDynamicBuffers = True
Yeah you’re right, I missed it in the documentation. Anyway definitely helps, its still strange to see how bad it runs considering its a 20yo game, but thats at least 6x better than before
from 30 to 180 fps with that flags
Hello everyone, I’m reviving this thread to report that through bisecting I noticed that commit [dxvk] Use new access helpers for resource relocation introduced severe stuttering and performance regressions with eGPUs, especially in situations of exhausted VRAM.
Just before that commit, it was working well with the previously mentioned configuration (d3d11.cachedDynamicResources=a).
I hope you’ll be able to identify and resolve the issue, for us eGPU users. Thank you.
That specific commit quite literally doesn't do anything that even can introduce stuttering. It just changes the way barriers are emitted when defragmenting VRAM, which was necessary due to some changes w.r.t. image layout handling. I really cannot see a universe where it would regress performance in any way whatsoever.
If you're running out of VRAM on an eGPU you're quite literally asking for shit perf anyway, and there's really nothing we can do about that. We don't have enough control over memory management to even know if any of our Vulkan memory allocations are resident in VRAM or not.
If you're 100% certain that that particular commit is problematic 100% of the time and isn't just randomly slow due to driver jank elsewhere, then we're going to need more details (game, gpu in use, etc) to even have a chance to figure anything out.
I can confirm that commit is problematic; I can reproduce the issue 100% of the time, when VRAM is exhausted or close to it (I also played a little with dxvk.maxMemoryBudget config to test it).
I know being in a situation with exhausted VRAM isn’t ideal, but from DXVK version 2.7 onwards, with VRAM eviction, the problem was largely alleviated. On second thought, I’m not sure if the problem is exclusively related to eGPUs or not; unfortunately I can’t do any test on a desktop system.
By the way, the game I tested is Assetto Corsa Competizione, with a Nvidia 2060 Super GPU 8 GB VRAM (580.119.02 driver version)
Let me know for any additional info may help you to analyze the issue. Thank you again.
Well here's the analysis: We transition images back to their default layout, while ignoring any subresources that are still in UNDEFINED layout. In other words, the barriers are a no-op on Nvidia, just like before.
From the DXVK point of view it's entirely working as intended, and it can't be changed either without breaking things.
Maybe this is an ugly hack, maybe not, but I managed to fix the eGPU stuttering issue with this patch dxvk-egpu.patch
I’d like to get your feedback, and see whether it could be integrated upstream, possibly by implementing a proper eGPU-specific code path or through a dedicated configuration option.
Thank you.
The real question is, why does that fix anything?
I don't think the patch does the right thing w.r.t. keeping clears, barriers and layout transitions in the correct order (this is really nasty to get right), but that would mostly only affect render targets from the last active render pass anyway so I don't see it mattering much. Everything else should be functionally identical.
We'll need to fully understand what's going on here, I'm not going to hack around this in broken ways since that's just going to create maintenance nightmares with code that we literally can't ever change without introducing regressions.
DXVK_CONFIG=d3d11.cachedDynamicResources=a`x2 2024-11DXVK_ASYNC=1x1 2023-10RADV_PERFTEST=dmashadersx1 2023-10RADV_PERFTEST=nosamx1 2023-10
System information
Asus TUF F15 FX516PM
Vulkan Drivers tested
Workarounds tested
RADV_PERFTEST=dmashaders
RADV_PERFTEST=nosam
DXVK_ASYNC=1 (no changes)
Summary
Its pretty much known for a fact eGPUs on Linux are a total nightmare, especially with newer cards, looks like it was not noticeable enough on older ones like the RX 550
I also tested with an NVIDIA card (RTX 3060) and the issues are pretty much the same.
The most noticeable thing about this, other than the really bad fps and low GPU usage, is the single core CPU usage spiking at 100% (not always the same core) when DXVK is used.
However, I'm sure DXVK on Windows with the same setup doesn't seem to affect performance that much in such games.
Slightly OT, there is something fishy also on vulkan only games as well as they seem to have performance issues even when DXVK is not used (ex. Baldurs Gate 3 on VK, performs decently on Windows even with the eGPU, or the integrated discrete NVIDIA even on Linux ) so I am tracking this issue even on mesa and the kernel driver where it has to be confirmed there may be some problems regarding the actual pcie speed detection, at least for AMD cards
However, its not always the case since some games perform good as Windows, taking for example DOOM Eternal, which works very very nicely.
I must also say making the eGPU running at apparently full PCIe 3.0 4x speed, a kernel parameter must be added
options amdgpu pcie_gen_cap=0x40000Forcing the limit to 1.0 x4 speeds doesn't seem to change that much in my case.
In the following
Comparison of Cyberpunk 2077 between forcing PCIe 1.0 4x speeds vs. 3.0 4x
https://gitlab.freedesktop.org/mesa/mesa/uploads/a431d19be845fd639f7a5989578e2aa0/immagine.png
Using an OpenCL Benchmarking tool, I have apparently confirmed at least with compute workloads, the PCIe bandwidth is correct and changes accordingly to that kernel parameter.
Speaking back to Linux, the issue is very much present in the case even on older games like Half-Life 2, Far Cry 5, Killing Floor 2
No matter the distribution, kernel, mesa drivers, dxvk version, etc...
The only thing that helped was using WINED3D instead of DXVK when OpenGL was used. WineD3D Vulkan backend has the same impact.
Software information
Half Life 2: No more than 130-140~ fps, even on low settings, low resolution, without frame limiter
Killing Floor 2: 35 fps when things are busy on screen, even on low resolution, low settings
Far Cry 5: 40 fps, even on low resolution and low settings.
Cyberpunk 2077: builtin benchmark does not' show more than 39-40 fps, no matter the video settings.
Ramping up the setting on such games, do not change the results at all.
CPU usage was also overall higher even on Vulkan Only games.
Apitrace file(s)
Just in case I made one for HL2, but doesn't make sense to post it.