protonscr

wineopenxr: xrEndFrame layer conversion uses layers[0] size for all layers, corrupting quads after a projection (XR_ERROR_POSE_INVALID)

protonclosed
ValveSoftware/Proton#10275 · opened 2026-10-17 by CruCo · updated 2026-10-17 · 1 comments · github
CCruCo 2026-10-17 github

Summary

wineopenxr corrupts composition layers in xrEndFrame when the first layer and later layers have different struct sizes. A typical case is one XrCompositionLayerProjection followed by two or more XrCompositionLayerQuad. The runtime receives garbage poses, and SteamVR rejects every such frame with XR_ERROR_POSE_INVALID.

Cause

In wineopenxr/openxr_thunks.c (generated by make_openxr, checked at experimental_11.0 7a94df49e4695fc101fd47d614166e69c43d9669), convert_XrCompositionLayerBaseHeader_pointer_array_win64_to_host chooses a single element size for all layers from in[0]->type:

switch(in[0]->type)
{
...
case XR_TYPE_COMPOSITION_LAYER_PROJECTION:
    out_elem_size = sizeof(XrCompositionLayerProjection);
    break;
case XR_TYPE_COMPOSITION_LAYER_QUAD:
    out_elem_size = sizeof(XrCompositionLayerQuad);
    break;
...
}
out = conversion_context_alloc(ctx, count * sizeof(*out));
for (i = 0; i < count; i++)
{
    if (in[i])
    {
        out[i] = conversion_context_alloc(ctx, out_elem_size);
        convert_XrCompositionLayerBaseHeader_win64_to_host(ctx, in[i], out[i]);
    }
    ...

conversion_context_alloc is a bump allocator (openxr_private.h). With a projection first, each quad (112 bytes on x64) is written into a 48-byte slot. The next layer's allocation starts inside the previous quad, so converting quad n+1 overwrites the pose/size of quad n. With a single trailing quad the overflow lands in unused buffer space, which is why simple apps work. The win32 variant and the generator code that emits polymorphic_elem_size(...) for these arrays have the same pattern.

Expected

Each layer is allocated with the size of its own type. For example, compute the size per element inside the loop from in[i]->type, or use the largest layer struct size.

Reproduction

  • Any Windows OpenXR app that submits [projection, quad, quad] in one xrEndFrame.
  • Observed with Vice City VR (reVC, D3D12 + OpenXR) and its wrist HUD panels:
    • Proton Experimental 11.0-20261014
    • SteamVR 2.17.10
    • PSVR2
    • RTX 3090
    • CachyOS
  • Every frame with more than one quad fails with XR_ERROR_POSE_INVALID. Logged layer data before xrEndFrame is valid (unit quaternions, sane sizes and rects).
  • Submitting a tiny quad before the projection (so layers[0] is a quad) makes all frames succeed and the quads render correctly. That confirms the conversion as the cause.

Mod-side report with logs: https://github.com/dubrovskiy-yevhen-stakelogic/vice-city-vr/issues/22

Kkisak-valve maintainer 2026-10-17 github

Hello @CruCo, we're tracking issues with game mods with the game that's being modified. Also, we're using one issue report per unofficially supported game title, so I've gone ahead and transferred this issue report to https://github.com/ValveSoftware/Proton/issues/291#issuecomment-6235815259.