The game is most trying to use more VRAM than your card has. This will make it fallback to regular RAM which can be severely hurt performance.
EDIT:
I checked the apitrace and it does indeed use more than 4GB of VRAM.

The game is most trying to use more VRAM than your card has. This will make it fallback to regular RAM which can be severely hurt performance.
EDIT: I checked the apitrace and it does indeed use more than 4GB of VRAM.
You appear to be right. It's only a 4GB card, and mangohud shows about 3.6 GiB used....so pretty much fully used.
What's threw me off to never think about checking VRAM was how well WineD3D runs. I also stumbled upon VRAM usage differing depending on which quality mode the game is started in. The quality setting definitely handles the texture quality.
DXVK (game started in ultra quality):
Ultra Quality: 3.6 GiB VRAM, 6 fps
Very Low Quality: 3.7GiB VRAM, 10 fps
DXVK (game started in very low quality):
Ultra Quality: 3.6 GiB VRAM, 40 fps
Very Low Quality: 2.4 GiB VRAM, 40 fps
WineD3D:
Ultra Quality: 3.4GiB VRAM, 85 fps
Very Low Quality: 1.4GiB VRAM, 130 fps
In DXVK, you can most certainly see the texture resolution being scaled back as you switch from Ultra to Very Low quality, but it doesn't look like the VRAM usage goes down like in WineD3D
What's threw me off to never think about checking VRAM was how well WineD3D runs
RadeonSi probably has better memory management and moves some less critical stuff into RAM. Windows does that too but unfortunately Linux Vulkan drivers usually don't and DXVK can't do it either.
In DXVK, you can most certainly see the texture resolution being scaled back as you switch from Ultra to Very Low quality, but it doesn't look like the VRAM usage goes down like in WineD3D
Might be memory fragmentation. In my screenshot, the allocated memory and the used memory differ by 0.5GB.
Either way, there isnt really anything we can do about this unfortunately.
proton experimentalx1 2022-11
There is a huge performance discrepancy between DXVK and WineD3D that I seem to be observing. Graphical settings were at Ultra preset for both tests:
DXVK

WineD3D

Software information
Element TD 2 (Appid 1018830)
OS: Arch Linux
System information
Apitrace file(s)
https://cloud.ryebread.red/s/FY5bmRwn7Rm3cBZ
Log files
https://gist.github.com/ryester19/90accf846bf997adadaf51b70235450d