I also experience the Shader Cache building everytime I try to start the game after a restart of steam.

confirmed on another games, everytime rebuilding

I have the same problem with those shaders. Week after week seems to be worse.

Same issue here on Linux.
I have enabled Shader precaching and allow background processing of Vulkan shaders.
My procedure for now is to let steam fully precache it and disable background processing after that. Everytime when it wants to precompile when starting a game I quit the process. Hopefully the issue is solved soon.

I'm having this issue on steam for Linux. The worst offender is Rise of the Tomb Raider. Shader caching can take 20-30 minutes, every single time I launch the game.

Can confirm this issue on Linux and it's a major pain. Haven't timed it but definitely feels like a 10-20 minute wait every time I try launching a particular Valve game, especially after an update. Considering many games push daily or semi-daily updates through Steam, this is a no-go and it's making me opt out Vulkan and into OpenGL when that's easily done in-game.

EDIT: I don't think it's opting out of OpenGL that stopped this problem but disabling the pre-caching. Haven't tried to reproduce.

I am sure this feature is well-intended as a quality-of-life feature but it's absolutely killing Vulkan support for me. My suggestion is to consider disabling it by default until a proper implementation is achieved because this is not on-par with Steam standards.

Imagine someone having to run 15 minutes of shader pre-compilation on battery on a laptop. I'm sorry but this is unacceptable.

Just for the record, if this pre-shading step took the same time it took today but happened once and as part of the installation of a product rather than as part of launching it, I would be 100% fine. I can leave the game downloading, installing and pre-caching as long as it needs to but waiting over 10+ minutes (or double that) when I try launching a game is not what I expect from Steam or any game.

I would say the same for game updates but as I said in my last comment, games can push very small updates multiple times a day and waiting 20 minutes for each of those (especially if updates are running on the background) is also not par for the course.

Issue started occurring for me a couple of days ago, too. Cache starts building every time I launch the game. No game/driver updates that might trigger it. The biggest offender so far is Warframe (for some reason, it takes a whopping 6 hours for Warframe's cache to be built, and this is successful scenario - most of the time it will hang at about 60% and won't progress after that).

Grand Theft Auto V does this too. Every darn time and hogging the CPU severely while at it. Valve, this is a SEVERE ISSUE with the Steam client and should not be left without any action for 2+ months. Give it some priority.

I have a lot of games installed - and a lot of shaders get processed every time i start the machine.
Every 2nd start of dota (my most played game) it processes again.
Even 1 time it happened that dota crashed and then the shader processing went so long that I got an abbandon.
Very annoying.

Same issue, specifically with Warframe. At this point I've disabled "Allow background processing..." as it happens after every restart, and after every time I play the game. Only uses a few threads, for some reason really likes using swap despite plenty of memory, and seems to really churn on my M.2.

Its just a bug

How does it being a bug change the fact that I'm suggesting an improvement over the current implementation?

EDIT: why did byte1024 delete his comments, too many thumb downs?

Same here. Processing shaders on Dota 2 takes about 40 min and start at every system boot. Worth to add other games (mostly steamplay/proton) affected too.

Same here, on my crappy laptop with hybrid graphics Rise of the Tomb Raider takes like one hour to process Vulkan shaders, every time I start it.

Same here, almost every time when I start Dota2, steam will rebuild it again.

Every time I start GTAV it builds shaders again. This process takes about 20 minutes.

Same here - nvidia 1060 - driver : 455.28

Shadow of the Tomb raider takes about 30 mins to start

Every time.

For me - I resolved this by removing Nvidia 455.28 and replacing with 450.80.02 - after i had to download the entire shader cache once more (i disabled then re-enabled pre-caching in steam) - after then it hasn't re-downloaded again (so far)

So is this purely a nvidia driver with 455.x series ?

So is this purely a nvidia driver with 455.x series ?

This is problem also with AMD GPU.

My uneducated guess is that this is a problem with the implementation/usage of Fossilize. I'd be surprised if this a driver issue at all.

One day gaming on linux is going to have less input latency then windows, and all the pro gamers / streamers will switch to linux. Then all the game developers and distribution platforms will finally start releasing decent software, drivers and tools. No excuses.

(same issue BTW on dota 2)

I can confirm that problem exists with nvidia-455, but NOT with nvidia-450 (with the older driver it processes the shaders once, but not once every start as with the nvidia-455). That surely does not mean that it's a driver bug, but it can probably help fix the problem as it definitly seems to be a bug not a desired "feature"

I have it with rx580 on amdgpu driver. It's not about nvidia drivers

Same here on Nvidia driver 455.38. I actually use OpenGL for dota (launch with -gl), but it still wants to compile the Vulkan Shaders when i start the game (i haven't even installed the DLC which supposedly adds the Vulkan Support). When i skip, the compilation sometimes does not get cancelled and i have to kill the fossilize-threads manually to avoid the high CPU-load.

Same here on Linux, this really annoying

Please give us a way to disable this feature !

Another confirm with Nvidia 455.x here.

This happens with me too with the latest AMD mesa drivers built from git. Currently on 20.3.0_devel.130194.fdbc45d1d46-1.

Also facing this issue with a GTX 1080 Ti - Nvidia 455.38

Just another confirm.
Arch Linux, GeForce GTX 1050 Ti, Driver: 4.6.0 NVIDIA 455.38

Same with gtx 670 on arch linux

Same for me on Ubuntu, Nvidia 455.38. 20-30 minutes for each startup after upgrading from 450.

EDIT: Confirmed downgrading to 450 solved the issue.

Same issue here Arch Linux kernel 5.9.8. Nvidia 455.38 driver Dota 2 take about 10 mins to cache everytime I launch the game

Same for Pop!_OS 20.10 64bit, kernel 5.8.7, nvidia driver 455.38. I noticed it a few days ago, for Subnautica Below Zero (only short amount of time needed) and Witcher 3 (took about 10min).

For myself, at least at the moment, I'm no longer having the issue using 455.45.01 on Arch. After some initial churning I can leave pre-caching/fossilize enabled and not have it constantly try to rebuild.

Same here for Arch Linux with Nvidia GTX 850M 4GB (driver ver. 455.38-7). Why would something like this get tied to the graphics driver? Or why would Steam invalidate the latest shader cache and recreate it again?

same here with rise of the tomb raider. i have nvidia 970 with 455.26.01 driver with archlinux. this fills up my 32 gb of ram and my 1700x cpu goes to 100% load for a lot of time. even mouse froze.
i'm curious why its not save on the disk. i mean its the same video driver for more than a month and i don't think the game had any updates in all this time. i don't remember seeing an update on download tab of steam.

edit:
i just started the game and there was no shader rebuild.
force close the game
restarted right away. and the nightmare started

Happens to me as well. On old Nvidia Driver 440.100 everything was fast, now on 455.38 is much slower.

AFAIK the Nvidia driver 455 added more support for more Vulkan extensions, as well as reworking some Vulkan components. Don't know about Radeon versions though.

Still having issue with Nvidia 455.45.01, 2080 Ti - I have disabled caching altogether.

Edit: disabling it does indeed introduce some stutter to some games (as expected) - it is noticeable...wasn't expecting this much!

Same here. Manjaro. Nvidia 455. 2080 ti.
I notice that with Dota 2 all the time, if I close the game and open again. It tries to rebuild the shader.
If I skip or disable the game stutters a lot.
it doesn't happen that much for other games as long as I don't close steam, or reset the computer.

Guess I spoke too soon. It was fine for a while but a couple updates (Steam and Kernel) and it's back to the same behavior. Guess I'm just leaving it off.

For me on GTA V and Shadow of the Tomb Raider shaders don't get rebuilt every time, but every reboot (or maybe restart of steam, I've not confirmed this).

this is starting to be very annoying. i wanted to start rise of tomb raider but.....
after wondering if my linux froze for more than 15 minutes (monitor turned off when computer had a moment :D) i finally was able to move my mouse.
don't get me wrong. i was able to move it for a 1-2 centimeters an than froze again for a few minutes.
so what happen when i was able to see some movements is that i got a network disconnect (i have pppoe) and i am still unable to do anything but moving the mouse from time to time.
after another 5-10 minutes the computer got a bit more responsive but that means less freezing of mouse cursor but still very hard click on anything.
after probably another 5 minutes everything was over
my chromium tabs have all crashed with "Aw, Snap!", steam is gone from tray and there is no tomb raider launcher anymore. even cinnamon desktop crashed on me at the end.

so i waited for nothing for more than 30 minutes unable to use my computer at all

good job Valve!

Happens to me as well. Arch Linux 5.9.9, GTX 1050(Mobile), driver: nvidia 455.38-8

Can confirm it happens on my system too.

Games - Fallout 4, Skyrim Special Edition, Dragon Age Inquisition, and Dark Souls II Scholar of the First Sin.

System - Kubuntu 20.10, nVidia GTX 1080, Ryzen 5 3600, 16GB DDR4, 1TB SSD.

nVidia Driver 455.45.01

Granted it only takes a couple of minutes on my system to create the cache but this is getting annoying, and generally not encouraging to new users of Linux want to try gaming, and this is the first thing they see.

Is there a solution in sight?

You can work around this problem by setting the environment variable __GL_SHADER_DISK_CACHE_SKIP_CLEANUP to 1, or alternatively increasing the value of __GL_SHADER_DISK_CACHE_SIZE

The new nvidia driver generates a lot bigger caches. If the cache size goes over __GL_SHADER_DISK_CACHE_SIZE the cache will be invalidated and regenerated on program start. Because the new driver version creates very big caches the size limit can be reached by only a single game, causing it to regenerate caches on every restart.

This was also reported on the nvidia forums:
https://forums.developer.nvidia.com/t/455-23-04-driver-produces-very-large-shader-caches-results-in-constant-cache-invalidation/156087/9

You can work around this problem by setting the environment variable __GL_SHADER_DISK_CACHE_SKIP_CLEANUP to 1, or alternatively increasing the value of __GL_SHADER_DISK_CACHE_SIZE

The new nvidia driver generates a lot bigger caches. If the cache size goes over __GL_SHADER_DISK_CACHE_SIZE the cache will be invalidated and regenerated on program start. Because the new driver version creates very big caches the size limit can be reached by only a single game, causing it to regenerate caches on every restart.

This was also reported on the nvidia forums:
https://forums.developer.nvidia.com/t/455-23-04-driver-produces-very-large-shader-caches-results-in-constant-cache-invalidation/156087/9

Thanks, I'll give that a try tonight and see if it resolves the issue.

FWIW, some people report that even with these variables set they still have problems. Does anybody also know what are the units for __GL_SHADER_DISK_CACHE_SIZE? Bytes? NVIDIA's docs only mention it's an integer, but not how it's interpreted.

My cache, for example, is about 5 GB, so if the units are bytes, I guess that 1000000000 (10^9), as reported by the Reddit user, might still be too low—and might explain why they continue to encounter the problem, despite using the variable. That doesn't explain why __GL_SHADER_DISK_CACHE_SKIP_CLEANUP doesn't help, but I also can't seem to find it in the documentation.

Separately from the NVIDIA big cache / out of memory / lock up issue, can users not on NVIDIA who are seeing shader processing every game launch attach a copy of their shader_log.txt file, found in ~/.steam/root/logs ?

@freundTech @kerberizer Setting:

export __GL_SHADER_DISK_CACHE_SKIP_CLEANUP=1
export __GL_SHADER_DISK_CACHE_SIZE=100000000000 [#100](/issue/ValveSoftware/steam-for-linux/100) gb max disk size of cache

Still doesn't work when complex/heavy games' shaders are cached (i.e. MH:W, DotA2, Jedi : Fallen Order, H:ZD, ...).
I guess we have to wait for Nvidia fix to get into mainline stable drivers (I'm on 455.45.01 with RTX 2080 Ti).

@Plagman I see shaders processing not every game start but every PC boot.
If my logs will be helpful, I put them below:

shader_log.txt
shader_log.previous.txt

Here is my hardware/software info: https://gist.github.com/AngryPenguinPL/8a0e97cc433bfbb1cf14f6b6034a0b4b

i think the export option is not good. it seems exporting a variable in a terminal windows is not available in another terminal window. and of course if you close the 1st terminal it will be gone.
probably declare it on boot to be available globally.
i'll have to try that to see if it works. but export it seems to work only in that terminal window. probably if you start steam from that terminal it should work

@logan001 Yes, you can put it in .profile or /etc/profile or alike for example to make it permanent and available globally.

@logan001 Yes, I've added it to ~/.profile and indeed it belongs to every process in my session. If I cat /proc/<steam pid>/environment | grep _GL_SHADER_DISK_CACHE... it does match.

I am experiencing the same issue on NVIDIA 450.80.02 940mx. Linux mint

@AngryPenguinPL it looks like it's getting legitimate frequent updates, albeit small, of the Dota shader database, which is why it's triggering rebuilds. However the rebuild time seems very long for an incremental 50k or 100k of new shader entries. What's the size of the mesa_shader_cache folder in the shadercache library folder for Dota 2? (570) Could anything be clearing out the Mesa cache?

seems that after processing all shaders for rise of tomb raider the folder had about 6.9 gb. i didn't play the game, i just let it process all shaders until the launcher window pop up. after i close this windows i lost 2gb+ in the shader folders.
i think there were 2 files in fozmediav1 folder that got remove after i close the game.

@Plagman recently Dota received small updates, which will trigger rebuild, yes, but just restarting Steam is enough for me to rebuild the cache again, especially if I allow Steam to rebuild cache in background, it will trigger immediately after Steam launch.

this shader crap it feels so random. i've started the game again (last time was yesterday) and now after shaders ware build and i've close the game, fozmediav1 folder kept its files but somehow then entire folder for RoTR went down again.
initially was 6.9gb. after closing the game went down to 4.7. after restarting the game and all the shaders then close the game the folder has 2.7 gb

Separately from the NVIDIA big cache / out of memory / lock up issue, can users not on NVIDIA who are seeing shader processing every game launch attach a copy of their shader_log.txt file, found in ~/.steam/root/logs ?

shader_log.txt
shader_log.previous.txt

sysinfo.txt

@Plagman

mesa_shader_cache: 793,6 MiB
but this whole dir
/steam/steamapps/shadercache/570: 2,9 GiB

Could anything be clearing out the Mesa cache?

No, nothing was clearing the Mesa cache.

t looks like it's getting legitimate frequent updates, albeit small, of the Dota shader database, which is why it's triggering rebuilds.

Well, there was one small Dota 2 update in last days but this happens every time the PC is turned on. Even when there was no update in Dota for a few days.

The updated shaders are deployed as they're being discovered, separately from game updates. Those were really small (50kb and 100kb for the last two updates you got) so they most likely didn't show up in the download manager. I will look at making incremental rebuilds faster, as building an extra 50kb of shaders should only take a few seconds.

Dota2. It used to be acceptable time until my last nvidia driver upgrade (ubuntu, driver version 455.45.01) Now it takes soooo much time. sad experience. allot of space used as well. Actually it used to work very well before this precompile of shaders. (nvidia 970)

this is annoying. why does it takes so much memory?
i just started steam and it started to process again the same freaking shaders for the same rise of tr. i got 4 fossilize_replay running each using 4+ Gb of RAM. thats just crazy. but the good part is that priority is very low and i can still use the computer when is set to background.
but still that is too much memory.

I have this same issue
asus zephyrus G15 on ubuntu 20.10 with nvidia 455 driver

There was a temporary "fix" for this for the Beta Steam Client on the 9th, basically disabling it for Nvidia owners.

Disabled shader processing on NVIDIA while driver issues are being looked into

https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/2918859890092727304

This problem exists on AMD too, I am using mesa-git on Fedora 33 with 5700xt . Shader process everytime I launch the game and it takes 20-30 minutes. If i skip it and relaunch the game for some reason (update or just restarting) it gives me the same window and it doesn't even allow to skip the second time.. So I have to restart steam in order to do that. Processing shaders is becoming a problem rather than a solution..

Edit : fixed in the latest steam update.

This problem exists on AMD too, I am using mesa-git on Fedora 33 with 5700xt . Shader process everytime I launch the game and it takes 20-30 minutes. If i skip it and relaunch the game for some reason (update or just restarting) it gives me the same window and it doesn't even allow to skip the second time.. So I have to restart steam in order to do that. Processing shaders is becoming a problem rather than a solution.

I'm on Fedora 33 with 5700xt also but I don't get the shader processing every time I start Dota 2. Sometimes I get "updating configuration" and sometimes i get "Processing Vulkan shaders". Processing the shaders does not take nearly as long for me as others have stated in this thread, mine usually takes up to a minute, maybe two sometimes. I remember it taking long once or twice, but that was at least 3 weeks ago.

@maetthew Yes it was fixed recently after a steam update..

This is fixed for me aswell but I am using the OpenGL version of Dota2, so its not pulling the vulkan shaders each time.
Maybe its still broken for vulkan users.

Looks like it is fixed here. Now I see fast shader processing only when I update mesa drivers or when game update is released. So fixed for me.

Mesa 20.3.2
Radeon RX 580 8gb
Tested in Dota on Vulkan

seems to be fixed for me too. i haven't seens a shader processing in at least a week if not 2.

Yesterday, I was playing Dota2 on OpenGL it worked fine. Now, after switching to Vulkan, its building shaders everytime and not even starting the game. How do I switch back to OpenGL ? DUhh!!

I am using Nvidia GTX 850M and Vulkan, and do not encounter the problem. It makes difference as when the error was still exist, everytime I started Dota 2, it rebuilt the cache entirely, which takes a very long time. Now it rebuilds only a portion, which takes just a few seconds to minutes. I'm surprised that this issue has not been closed yet. I think since the fix, the stored cache size decreases in size, cmiiw.

@rahbal you can add -gl to the command for Dota, this will force Dota to use OpenGL. What driver version did you use (Nvidia, Mesa, etc.)?

Yes, adding -gl works and its runs the game with OpenGL. But, Vulkan still has issues on my system. I am using MX Linux.

Driver is Nvidia 455.45.01

Haven't seen this for a few weeks, closing this as fixed.

I have the same problem, with GTA V, Fallout 4 and Red Dead Online (other games not tested).
Nvidia Driver 460.39 on Fedora 33.

It's still happening with Rise of the Tomb Raider. Nvidia 460.67 on Debian Bullseye. Not just "every day" but repeatedly when launching the same game twice in a row.

I am also experiencing this with GTA V (fossilize freezes, makes 2 zombies and the 2 active processes stay at a certain ram usage and nothing happens), Kisak Mesa 21.0.2 on Bionic. I managed to get it to go to 100% one time by constantly opening and closing the client + killing the frozen fossilize process, but when it tried to verify the cache next launch it gets stuck at 99%, tried deleting all shader caches to no effect.

edit. Seems resolved with Mesa 21.0.3, thanks

I'm having this same issue in Elite Dangerous. Every time I launch the game it spends 5-10 minutes "Processing Vulkan Shaders".

Arch linux, Threadripper 1950x, Radeon VII.

Can we get this issue reopened? This hasn't ever been resolved for me and I'm still having the issue.

On my ryzen 3 2200g with vega 8 seems to be good.

Can we get this issue reopened? This hasn't ever been resolved for me and I'm still having the issue.

which GPU do you have?

1080 ti

yep ok, I also have this problem, and I have 960 gtx.
So it seems it is problem with nvidia cards, right now.

I have a RX560 and starting Rise of the Tomb Raider takes forever to start. So it seems it's not only nvidia related. My CPU is a R5 2400g paired with 16GB RAM.

On NVIDIA it is only happening on specific versions of the NVIDIA prop driver. I did not see it on 460 series.

Ubuntu GamePack 20.04
nvidia GTX1070 8GB using 460 prop/tested drivers
64GB RAM

I have two games installed in steam (7 Days to Die and ARK: Survival Evolved) and both were consuming 100% CPU while fossilizing. Swap was untouched and RAM was barely touched.

Edit to add: I have turned off prechaching as per suggestions in posts above.

both were consuming 100% CPU while fossilizing.

Well at least they are fully utilizing the CPU

I have noticed that shader cache will validate everytime when I exit certain games.

Just an idea: this issue was closed as it was about a specific bug in NVidia drivers on Linux (which was fixed indeed). Maybe, we need a separate issue for those similar problems everyone seem to experience?

this issue was closed as it was about a specific bug in NVidia drivers on Linux (which was fixed indeed). Maybe, we need a separate issue for those similar problems everyone seem to experience?

What is the difference? From my perspective, it's the same issue and was never fixed. I've had shaders rebuilding every time I launch games since this ticket was opened and it has never been fixed.

I did experience this problem in the past (around the time this issue got created), then it got fixed (for me), but now it's back.
GTX 1080 Ti / Intel i7-4770K
Up-to-date Manjaro Linux
Nvidia 470.42.01 proprietary driver
Steam Beta branch

Hello this ticket needs 2 b reopened im still having this issue.
On a 3070

@kisak-valve reopen the issue.

Shaders still rebuild in the background every time steam/computer is restarted.

don't confirm the problem (fedora 34, steam via flatpak)

1080 Ti here too can confirm.

@Turmfalke2 can you reopen the issue please?

1050ti, same problem, mostly with rocket league

this issue was closed as it was about a specific bug in NVidia drivers on Linux (which was fixed indeed). Maybe, we need a separate issue for those similar problems everyone seem to experience?

What is the difference? From my perspective, it's the same issue and was never fixed. I've had shaders rebuilding every time I launch games since this ticket was opened and it has never been fixed.

This should be re-opened since it is a major drawback from using Steam under Linux!

This is happening to me with Skyrim Special Edition
Nvidia 470.63
GTX 1050 TI

I have the same issue since today on Arch with a ryzen 3700x and a rx 580.
Because of that, my cpu is always between 55 to 70°C idle with steam and some apps like mail client, web browser open

I'm having it happen with all games that use Vulkan, but steam is also reporting I have 0MB pre-cached files when the folder is actually containing 16.5GB of files.
Ubuntu 20.04.03
Nvidia 470.63
GTX 1650

Both Beta & Non-Beta report the amount wrong, but the beta re-downloads files & compiles them every time I run any game that uses Vulkan, recently it's taken to doing it after I've quit the game & not while waiting for the game to start.

Same here with GTA V, Intel Iris Xe graphics.

Interesting thing I noticed: When going to steam settings during compiling shaders, it is suddenly instantly done with precompiling and starts the game.

In my case, Steam redownloads all the cache again everytime I reboot. I'm fine with just few megabytes of download but for games like Monster Hunter World (with proton), it downloads about 1GB of shaders, which is a waste of time. Because of this Steam recompile shaders all the time, although some may just take a few seconds to minutes.

You could try the Steam BETA client, it has many changes, maybe that could help in your case. I download only little chunks nowadays.

I can't determine when Steam will download only little chunks though. It seems to me that sometimes it downloads 1GB sometimes few hundreds MB. I don't think there are updates to Monster Hunter World recently but it still download some shader contents nonetheless. But I'll try Steam beta.

I am on Ubuntu 20.04.2 and have RX 570. Everytime i run steam it downloads pre-chaching content for Dota 2. It was about 150MB-200MB until 3 days ago. Now its 850MB.
https://img.onl/4lOLC5

@Turmfalke2 I see this every single time I log into STEAM, without exception. Can we PLEASE have this re-opened? Like, before reboot, STEAM has already processed all shaders. Reboot computer, log back into STEAM, IMMEDIATELY processing shaders again. No change, just processing shaders again. It's really frustrating.

Ubuntu 20.04 RTX 3060 ti

@Turmfalke2 I see this every single time I log into STEAM, without exception. Can we PLEASE have this re-opened? Like, before reboot, STEAM has already processed all shaders. Reboot computer, log back into STEAM, IMMEDIATELY processing shaders again. No change, just processing shaders again. It's really frustrating.

Ubuntu 20.04 RTX 3060 ti

This issue was never fixed, not sure why it got closed.

At this point open an issue if you have problems regarding shader pre-caching, giving all required details. This issue isn't gonna be reopened so nothing said here is useful, it only spams the OP.

That open issue is different from what I experience. No reboots are necessary to repro the issue on my end.

Back to my message so? Stop flooding this closed issue, it has been asked multiple time to be reopened, it hasn't, it will not, gather the proper info, open an issue, and be done with this issue stop flooding people notification for useless comments (like this one, no need to comment about the irony of my reply).

If you don't want to receive notifications on this issue, I suggest that you unsubscribe from notifications using the button in the top-right.