I have the same issue. I'm on Arch and also use an AMD GPU (RX 580 SP) I've had this issue since i've joined the game recording beta and even now that is live it still persists. Also tried different versions of Proton and native games and same result as you. I'm surprised nothing has been done about it yet Valve please fix.

Edit: Forgot to mention that screenshoting works when you are on the steam overlay. Although that isn't really usefull since 99% of the time i want to screenshot the game and not the steam overlay.

Same, but completely different hardware. In my experience, this seem to be an issue with UE games. In those, screenshots either do not work at all, or only work sometimes, 1 out of 100 times, if key is mashed. What is weird is that screenshotting works flawlessly when the Steam Overlay is opened (though, naturally, that takes screenshot of overlay on top of the game).

At the same time i can open an non-UE game and screenshots work completely fine.

Chiming in that I'm seeing the same behavior here, i5-10400f / 6600xt / endeavouros kde, ever since screen recording first went into beta, though I am on the stable build nowadays.

As far as I've been able to figure out after playing with the main menus of Project Wingman (UE4) and Forza Horizon 5 (bespoke graphics engine "forzatech" afaict), refresh rates have something to do with it.
In FH5, with my monitor running at 144hz and the game vsynced down to 72, I could get maybe 1 in 15 key presses to register. Sometimes it would work first or second try, sometimes I'd have to mash for a good 3-4 seconds, but it would work eventually. Recently I switched my monitor to 120hz and am using similar in-game settings of 60, and the screenshot key has now stopped working entirely. Same for if I run the monitor at 60. However, if I disable the in-game vsync and frame limiters in the main menu, the screenshot key starts to work perfectly again.
In PW, it's a lot of the same story, except even with framerate completely unlocked I can still only get the screenshot to fire about 1/10 times. Much more than the zero times when vsync/frame cap is active though.
It's possibly worth noting that the main menu background in FH5 is a pre-rendered video while the background in PW is in-engine.

Just for curiosity, I tested FH5 with mangohud: the game settings itself unlocked, mangohud limiting fps to 60. No screenshot key. So it seems that if any kind of vsync or frame limit is active, the overlay can't manage to trigger a screenshot unless it's in the overlay itself.

Edit to add: pausing game recording does indeed seem to allow the screenshot key to work no matter what vsync settings are in place.

Press the Screenshot button a looooooooong time, then it work for me while Game Recording is on^^

Press the Screenshot button a looooooooong time, then it work for me while Game Recording is on^^

That works for some games, sometimes. Other times for other games mashing works instead (and long pressing doesn't do anything). And for some nothing works at all

Haven't tested in depth yet, but with the Jan 20th stable client update screenshots appear to be working again.