protonscr

Precompiled shaders are pruned by nvidia driver, causing stutter from cold cache effects in games with excessive numbers of shaders

steamopen NVIDIA drivers
ValveSoftware/steam-for-linux#11392 · opened 2024-10-27 by ryao · updated 2026-08-26 · 18 comments · github
1 matching comments, n / p to jump
Rryao 2024-10-27 github

Your system information

  • Steam client version: 1726604483
  • Distribution (e.g. Ubuntu): Gentoo
  • Opted into Steam client beta?: No
  • Have you checked for system updates?: [Yes/No]
  • GPU: Nvidia

Please describe your issue in as much detail as possible:

A friend complained to me about Overwatch 2 suffering stutter from constant shader compilation with the nvidia driver, so I looked into it. It appears that the nvidia driver is pruning the shader cache, causing it to compile shaders ad infinitum during game sessions, which causes stutter.

If we launch Overwatch 2 withDXVK_HUD=compiler %command%, we can see when it compiles shaders, which is quite often. If we view CPU utilization on another screen (or by alt-tabbing) in htop or another tool, we can also see that CPU utilization is often pegged at 100%.

The nvidia driver historically limits the size of its shader cache to 128MB, but in the 460 driver, increased this to 1024MB:

https://www.nvidia.com/download/driverResults.aspx/167671/en-us/

Under the assumption that we were hitting the shader cache limit, I had my friend add __GL_SHADER_DISK_CACHE_SIZE=10737418240 to his launch options to increase his shader cache size to 10GB as per:

https://download.nvidia.com/XFree86/Linux-x86_64/550.127.05/README/openglenvvariables.html

After a 2 hour session, my friend reported to me that the excessive shader compilation had stopped and his shader cache was 5.7GB, which far exceeds the 1GB default limit.

I actually joined him in the session, and confirmed excessive shader compilation. Before setting __GL_SHADER_DISK_CACHE_SIZE, I could not launch Overwatch 2 without excessive CPU utilization (all cores being pegged) while DXVK claimed shader compilation was being done. Additionally, the excessive CPU utilization was so bad on my machine that discord had static noise both from what I heard when my friend spoke and reportedly, when he heard me speak. After setting __GL_SHADER_DISK_CACHE_SIZE and joining my friend's session, my shader cache grew to 4.3GB by the time the session ended. Immediately after relaunching Overwatch 2, I can see my CPU utilization is low, which did not happen until setting __GL_SHADER_DISK_CACHE_SIZE.

In order to be absolutely certain of what is happening, I would want to be able to tell steam to re-run fossilize replay and watch the size of the shader cache, but I cannot find a documented way to do that no matter how much I look. Without that, investigating this further would be a major time drain, but I think I have enough information to make a report and hopefully someone at Valve could advise on how to force fossilize_replay to run on a game so people can gather more data points efficiently.

That said, based on what I have observed, one of two things seems to be happening under the default settings:

  1. The nvidia driver is actively pruning the shader cache that fossilize_replay tries to prime as it runs
  2. The nvidia prunes the primed shader cache at game launch.

Either way, the shader cache on Nvidia hardware is being actively pruned to keep it within the default limit, making it useless for games with many gigabytes of shaders under the default settings, as cold cache effects will persist due to the default cache size limit.

Steps for reproducing this issue:

  1. Get a Nvidia machine.
  2. Enable steam shader precaching
  3. Install Overwatch 2 (other games may also have this issue)
  4. Set DXVK_HUD=compiler %command% as launch options to Overwatch 2
  5. Play Overwatch 2 for an hour
  6. Observe excessive CPU utilization and shader compilation, plus stutter
  7. Restart Overwatch 2 and play it for an hour
  8. Observe excessive CPU utilization and shader compilation, plus stutter
  9. Add __GL_SHADER_DISK_CACHE_SIZE=10737418240 to launch options
  10. Restart Overwatch 2 and play it for an hour
  11. Observe excessive CPU utilization and shader compilation gradually go away, along with associated stutter
  12. Restart Overwatch 2 and play it for an hour
  13. Observe there is no excessive CPU utilization or shader compilation
Kkisak-valve maintainer 2024-10-27 github

Hello @ryao, this issue should be reported to your video driver vendor.

Kkisak-valve maintainer 2024-10-27 github

Thinking about this a bit more, Steam sets __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 when running games and the NVIDIA driver should respond to that environment variable by disabling the driver's disk cache limits.

Rryao 2024-10-27 github

@kisak-valve You are right. Interestingly, I just started overwatch again and it is compiling shaders. Thanks to the DXVK_HUD=compiler, I see Compiling Shaders... (44%) at the bottom. That did not happen an hour ago.

Before I filed the issue, I had tried unchecking/rechecking "Enable Shader Pre-caching" to try to force steam to run fossilize replay, but it did not seem to work. I observed no change in the nvidia cache. I then started Overwatch 2, observed low CPU utilization and opened the issue. Now an hour later I start Overwatch 2 again and suddenly see high CPU utilization from DXVK doing shader compilation. The shader cache is also growing. This non-deterministic behavior does not really make sense to me.

It would be helpful if there was some way of running fossilize replay on demand so that the behavior could be studied. That is the reason I opened this issue despite my research being half baked. I don't have the time to investigate this for my friend unless I can prepopulate the cache to study the behavior from different states. He had switched to Linux on my recommendation and is understandably annoyed that Overwatch 2 has some stutter issues. Initial observations show that it is shader disk cache related, but more experiments are needed to understand the behavior and being able to invoke fossilize replay would make those easy to do.

Rryao 2024-10-27 github

Overwatch 2 somehow stopped rendering new frames from excessive alt-tabbing so I killed it during DXVK shader compilation and restarted it. It now isn't compiling shaders... this behavior is bizarre. :/

My cache is now 7.5GB in size.

Rryao 2024-10-27 github

I just played a game after changing some graphical settings and the on-disk shader cache shrank from 7.5GB to 4.5GB afterward. There is some kind of pruning happening, even though __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 is set. :/

Rryao 2024-10-28 github

I reported this to Nvidia. It would be helpful if Valve documented how to trigger fossilize replay on demand to prime the cache to make it easier to observe the cache pruning bug.

Ppeterkohaut-nv 2024-10-29 github

I filed internal bug 4934720.

Please, can you share which driver are you using? Or ideally proton.log or dxvk log.

Just to verify, are you setting both __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 and __GL_SHADER_DISK_CACHE_SIZE=10737418240 ?

Rryao 2024-10-29 github

@peterkohaut-nv Thanks, but this was filed at Nvidia yesterday as internal bug 4932793. I did not get the bug number to post here until about an hour ago. A number of details are already there, including nvidia-bug-report.log.gz files.

This was observed on Kubuntu 24.04 running 550.107.02, and on Gentoo running 560.28.03.

The cold cache effects were observed when only __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 was set. They disappeared soon after adding __GL_SHADER_DISK_CACHE_SIZE=10737418240. Unfortunately, I don't have DXVK or proton log files at the moment. I will try to get them for you.

Right now, from what I have been told, there seem to be two issues:

  1. When __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 is set, new shaders are not added to the disk cache if the cache size exceeds __GL_SHADER_DISK_CACHE_SIZE. This is likely the bug that prompted filing this report. This behavior is contrary to what everyone outside Nvidia thought that __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1 did. Also, coincidentally, __GL_SHADER_DISK_CACHE_SKIP_CLEANUP being set triggers the skip cleanup behavior, and the value is actually ignored, so setting to 0 would not override Valve's setting it, which complicates testing.
  2. When __GL_SHADER_DISK_CACHE_SIZE is set and is bigger than the disk cache size, sometimes the disk cache will shrink across game sessions. This was observed after filing this issue and we do not know how to reproduce it reliably. Yesterday, I had thought it was related to 1, but that apparently is not the case.
Rryao 2024-10-29 github

@kisak-valve It would be helpful if Valve provided a way to force steam to trigger fossilize replay to prime the disk cache between runs to help figure out how to reproduce 2. Is there any chance you could put us in touch with someone who could say how to do that?

Aamrit1711 2025-02-24 github

Hi @ryao
Could you please help to test with NVidia latest released driver 570.86.16 and share test results.
If issue still persists, could you please share nvidia bug report.
We are unfortunately unable to duplicate issue locally.
Also it would be good to know the exact repro steps with expected output and observed output so that I can match exactly the same in my test environment.

Aamrit1711 2025-04-01 github

Hi @ryao
Could you please help to update on my latest comment.

Aamrit1711 2025-04-07 · hidden on GitHub github

Hi @ryao
Could you please help to update on my latest comment.

Aamrit1711 2025-04-14 · hidden on GitHub github

Hi @ryao
Could you please help to update on my latest comment.

Aamrit1711 2025-04-21 · hidden on GitHub github

Hi @ryao
Could you please help to test with NVidia latest released driver 570.86.16 and share test results.
If issue still persists, could you please share nvidia bug report.
We are unfortunately unable to duplicate issue locally.
Also it would be good to know the exact repro steps with expected output and observed output so that I can match exactly the same in my test environment.

Kkisak-valve maintainer 2025-04-21 github

Hello @amrit1711, please stop spamming this issue report with duplicate requests.

Bbighead31 2026-07-19 github

I saw this issue was still open and I wanted to post because this is still happening in 2026. I will post both here and the related NVIDIA forum. (ValveSoftware/steam-for-linux#11392 | internal NVIDIA bug 4932793)

I think this must be happening to most people attempting to play Overwatch, or any game with really large shaders, on most linux distros with NVIDIA gpus. From my research, AMD gpus have a bigger shader cache allocation by default and don't have this issue. I tried many launch options, using DX12, and different proton versions and other things in a attempt to get this working. The solution below (See "Fix:") was the only thing that worked.

System:

  • GPU: RTX 3060 Ti
  • Driver: 610.43.03 (also happened on earlier 5xx drivers)
  • OS: Fedora 44 (x86_64)
  • CPU: Ryzen 5700G
  • Game: Overwatch 2 via Steam, GE-Proton11-1
  • Steam Launch Options: DXVK_HUD=compiler - Displays a message when shaders are compiling
    DXVK_CONFIG="dxvk.trackPipelineLifetime = True" - Fixes a separate memory leak issue
    PROTON_ENABLE_NVAPI=1 - DLSS Compatability
    gamemoderun - for linux gamemode
    %command%

Reproduction:
Every game launch triggers a full shader recompile — 100% CPU for 15+ minutes on all cores. The game is unplayable during this period. After the compile finishes, gameplay has occasional hitching as additional shaders compile on-demand. This happens on every boot, not just after driver/game updates.

Steam's shader pre-caching also does not work: it takes ~3 hours to complete, has zero effect on the in-game compile, and re-runs the full 3-hour process on every subsequent boot. Steam's precache was about 20GB on my machine and persisted, but it still wanted to rerun the preaching each time the game started.

What I believe to be the root cause:
The NVIDIA driver's shader disk cache (~/.cache/nvidia/GLCache/) is limited to 1GB by default. Overwatch has a lot of shaders, which may or may not exactly be optimized, and it's shader cache exceeds this limit. From my monitoring of the folder during shader compilation, it builds until it reaches exactly 1GB and then is capped there (even though the game reports shader compilation). My guess is it continues to replace existing shaders as it continues to compile. The game then runs with poor performance and frequent hitching. My guess is it continues to compile missing shaders, and replacing old ones at the same time, staying in a bad loop.

I was also able to capture the deletion of the shaders upon game boot with: inotifywait -m -r ~/.cache/nvidia/GLCache/ -e delete --format '%w%f %e'
I'm assuming the cache is invalidated due to it not being imcomplete and rewritten over in the above mentioned loop, so the game just recompiles from scratch.
System reboots seemed to mostly clear out the shader directory when I checked on the folder.

Fix:
Setting this system-wide in /etc/environment and rebooting fixed the issue completely:
__GL_SHADER_DISK_CACHE_SIZE=10737418240
__GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1
The game had to compile shaders for about 15 minutes as usual, but now there are no issues and no hitching in the game. This fix persists even when restarting the game, or rebooting the machine. I also tested afterwards removing this entry and it reproduced the problem.

Note: This must be set system-wide. Setting them in Steam's per-game launch options had did not have the same effect. The __GL_SHADER_DISK_CACHE_PATH variable in launch options also did not help. Additional details below:

Steam Launch Options Effects:
10 gb cache size only - Would grow above 1GB but be cleaned out between game startups and system reboots.
10 gb cache size and skip cleanup - Would grow above 1GB and persist between game startups. Gets cleaned with system reboot though.

/etc/environment effects:
10gb cache size only - grows above 1GB and persists between game start ups. Restarting the machine left the cache in place, but booting the game caused it to compile shaders again. The old shaders seemed to remain, but they were ignored.
10 gb cache size and skip cleanup - Fixes the issue as described above

EenricoKoschel 2026-08-09 github

@bighead31 Thank you for the write-up! This fixed the issue for me, no more hour long 100% CPU usage by fosilize-replay. But sadly not for Overwatch. I have accepted that Overwatch is not playable for me on Linux.

FFuzzyQuils 2026-08-26 github

@bighead31 Thank you for the write-up! This fixed the issue for me, no more hour long 100% CPU usage by fosilize-replay. But sadly not for Overwatch. I have accepted that Overwatch is not playable for me on Linux.

Overwatch has quite a high number of shaders using D3D11 Stream Output (also known in OpenGL/Vulkan as Transform Feedback) which can't be fast-linked when using VK_EXT_graphics_pipeline_library and I've suspected for a long time that this is the source for nearly ALL of the hitches experienced on NVIDIA systems on Linux not caused by the game itself. (Blizzard have dropped the ball on optimisation lately with microstutters even appearing on Windows)

For NVIDIA employees: A pretty consistently reproducible case to trigger the slow link path that causes stutters on a cold cache is to do the following:

  • Open Overwatch, let the CPU eventually come to rest.
  • Open the Heroes tab, select Moira and click "Customise"
  • Open the highlight intros section, select "Vanishing Act"
  • Observe that the game comes to a complete stop for ~2 seconds every time Moira teleports. This will only happen once, with subsequent playback attempts being completely smooth