So I am digging deeper here and if my diagnosis is correct, textures are not such a big deal - although moving them off process' address space will still help.
However, I noticed something strange. Game creates a lot of Vertex Buffers with D3DUSAGE_WRITEONLY, and that still adds to process' address space usage. Why is that happening? @K0bin, any thoghts?
Also, does Sysmem counter in DXVK show the complete process' address space usage by DXVK? Thank you!
Edit: I spent time investigating memory allocations using WPA/WPR and I found that vertex buffers contribute to memory pressure as much as sysmem textures. In the second apitrace above, roughly 620mb of chunks is allocated for textures and 600mb for vertex buffers.
I was able to extend approach in the current PR, if anyone cares:
https://github.com/TheIronWolfModding/dxvk/tree/vr-merge - but this include changes specific to my branch.
https://github.com/TheIronWolfModding/dxvk/tree/b-mmf - closest to master and PR.
Following up on my ask here: https://github.com/doitsujin/dxvk/pull/2524#issuecomment-1089768392
Game (GTR2, but I am pretty sure whole family of games is affected by this) seems to load all textures into systemem and then transfer them as needed into the default pool. Sysmem textures are held around. Transfer does not happen much (only 1-2 UpdateTexture calls per frame) when perf actually matters. So transfers seem to happen mostly during the last step of the scene load.
I made some modifications to game behavior by releasing some sysmem textures and I can see memory usage reduction, but I it is hard to measure if that is the only problem leading to address space exhaustion. I think another contributor is DYNAMIC usage buffers, but hard to tell how much that is...
Please find APITrace and let me know if I can help in any way: https://1drv.ms/u/s!ApNCoFDyNnCJhMxAlKyIYbpafqsdVQ?e=xLzhVt
One more trace with higher detailed content: https://1drv.ms/u/s!ApNCoFDyNnCJhMxHumaK7x3uv98xgQ?e=C7d1zM