If you don't have enough memory, don't use settings that require more memory than you actually have.
DXVK does inherently have some memory overhead (mostly on system RAM) since it has to keep both the D3D and Vulkan representations of each object in memory, there's nothing to fix here. A large portion of that is going to be shader code, which has to be kept in memory at all times.
DXVK also doesn't have the most sophisticated memory manager in the world, but I don't plan on rewriting half of DXVK just to improve memory pressure situations (doing so on the DXVK level would be exceptionally hard), it does already use VK_EXT_memory_priority on drivers which support it.
@doitsujin the game does this with 8gb vram at 1440p too: if I enter a new area and then alt-tab 1-2 times then it'll consume all of the vram and the fps will drop below 5. This is relevant for #1148 and for many proton reports this is the source of the slow-downs and crashes: https://www.protondb.com/app/403640
@trialism, since then, I have modified my setup a bit. For this game and this game only I have switched Vulkan driver to AMDVLK (it's still Mesa for any other application) and this problem is gone! I don't know if this issue is to be reported rather to RADV team than here anymore.
I tried to play Dishonored 2 with DXVK, and problem occurs with memory allocation. I tested it out on both, Linux and Windows backend and problem source seems to be the same. I have 4GB of VRAM on my graphics card (usually enough for me), but in case of Dishonored 2, game started to allocate more than 4GB. On Linux game allocated memory in swap and killed performance, on Windows+DXVK application gets terminated because of insufficient memory. The same problem doesn't happen with native DX11.
I already tried to report that through comment on Proton issue, but it seems to be strictly DXVK related
https://github.com/ValveSoftware/Proton/issues/823#issuecomment-502182111
Problem recorded with OBS, settings very high with high textures at the end https://youtu.be/JBmaR1pT4tI?t=195
Software information
Dishonored 2, settings on high
System information
Log files
(Files from Windows, I can revert my rig to Linux if needed)
Dishonored2_d3d11.log
Dishonored2_dxgi.log