Does vsync cap the frame rate? Is this with recent xorg 1.20.7 and xf86-video-amdgpu 19.1?

Yes, yes and yes. Also happens with amdvlk-pro 19.50 with UseFlipHint,1 although the tearing seems more 'subtle'.

Ok, that's weird. The game's window is fullscreen and its size exactly matches the display's native resolution? Then I'm really out of ideas.

The window definitely appears to be proper fullscreen, but the log has this:

info:  D3D9: Setting display mode: 1920x1080@60
info:  Presenter: Actual swap chain properties:
info:    Format:       VK_FORMAT_B8G8R8A8_UNORM
info:    Present mode: VK_PRESENT_MODE_FIFO_KHR
info:    Buffer size:  1918x1078

so maybe the buffer size not being equal with the window size causes problems? FWIW, windowed mode with compositor enabled works around the problem. This is also the only out of many games i tested with DXVK/D9VK which has this problem.

Hmm, for me it works as intended (nVidia + KDE compositor enabled + fullscreen mode):

info:  D3D9: Setting display mode: 1920x1080@60
warn:  D3D9DeviceEx::SetRenderState: Unhandled render state D3DRS_MULTISAMPLEANTIALIAS
info:  Presenter: Actual swap chain properties:
info:    Format:       VK_FORMAT_B8G8R8A8_UNORM
info:    Present mode: VK_PRESENT_MODE_FIFO_KHR
info:    Buffer size:  1920x1080
info:    Image count:  3
info:    Exclusive FS: 0
info:  D3D9: Setting display mode: 1920x1080@60

It's not surprising that it works with a compositor enabled

@Oschowa

Though that buffer size difference seems suspicous, page flipping might not kick in due to it not matching your native resolution. Having a compositor (or TearFree) on top of that unfortuntely increases input latency, likely breaks FreeSync and might also introduce additional stuttering.

I've also found that setting a non native resolution, i.e. 960x540 on a 1080p display, or 1920x1080 on a 2160p display, also makes the problem disappear (Proton with fshack). In both of those cases, the buffer size matches the set resolution, but when switching to native again, the mismatch reappears.

DDDA_d3d9.txt

edit: i guess that makes it likely the issue is caused by proton's window handling interacting with my window manager (gnome/x11 3.34) or something and not a dxvk issue.

So, i did some more digging.
First of all, the same problem also happens when using the Openbox window manager, so not exclusive to Proton on Gnome.
Second, i found that always returning a RECT with right = 1920 and bottom = 1080 from wine's dlls/user32/win.c WIN_GetRectangles() function also works around the issue, so it seems like the buffer size mismatch is indeed to blame.
I'll close this, since it seems to be a wine issue and not related to dxvk.