protonscr

wineopenxr: xrEndFrame layer conversion uses layers[0] size for all layers, corrupting quads after a projection (XR_ERROR_POSE_INVALID)

protonclosed
ValveSoftware/Proton#10280 · opened 2026-10-17 by CruCo · updated 2026-10-17 · 1 comments · github
CCruCo 2026-10-17 github

Summary

wineopenxr corrupts composition layers in xrEndFrame when the first layer and later layers have different struct sizes. A frame [XrCompositionLayerProjection, XrCompositionLayerQuad, XrCompositionLayerQuad] reaches the runtime with corrupted quad poses, and SteamVR rejects every such frame with XR_ERROR_POSE_INVALID (-39). This affects any Windows OpenXR app, not one specific game. A standalone repro is below.

Cause

convert_XrCompositionLayerBaseHeader_pointer_array_win64_to_host in wineopenxr/openxr_thunks.c (generated by make_openxr via polymorphic_elem_size, checked at experimental_11.0 7a94df49e4695fc101fd47d614166e69c43d9669) picks one element size for all layers from in[0]->type:

switch(in[0]->type)
{
...
case XR_TYPE_COMPOSITION_LAYER_PROJECTION:
    out_elem_size = sizeof(XrCompositionLayerProjection);
    break;
...
}
for (i = 0; i < count; i++)
{
    if (in[i])
    {
        out[i] = conversion_context_alloc(ctx, out_elem_size);
        convert_XrCompositionLayerBaseHeader_win64_to_host(ctx, in[i], out[i]);
    }

conversion_context_alloc is a bump allocator. With a projection first, each quad (112 bytes on x64) is written into a 48-byte slot, so converting quad n+1 overwrites the pose/size of quad n. With a single trailing quad the overflow lands in unused space, which is why most apps work. The win32 variant has the same pattern.

Expected

Each layer is allocated with the size of its own type, for example by computing the size from in[i]->type inside the loop, or by using the largest layer struct size.

Standalone repro

A minimal D3D11 OpenXR app (about 270 lines, source below). It submits three layer layouts in turn for 45 seconds and counts the xrEndFrame results:

Mode Layers ok fail
A [projection, quad] 1348 0
B [projection, quad, quad] 0 1347 (-39, XR_ERROR_POSE_INVALID)
C [quad (tiny), projection, quad, quad] 1345 0

Mode C contains the same layers as B plus one tiny quad in front. It succeeds only because layers[0] is now a quad, so every converted slot is large enough. All layer data is valid: unit quaternions, sane sizes, full-swapchain rects, LOCAL and VIEW spaces.

Environment:

  • Proton Experimental 11.0-20261014
  • SteamVR 2.17.10
  • PSVR2
  • RTX 3090
  • CachyOS

Build: cl /EHsc /O2 /MT /std:c++17 xrquadtest.cpp /I <OpenXR-SDK>\include /link /LIBPATH:<OpenXR-SDK>\x64\lib (OpenXR SDK 1.1.58). Run it with SteamVR active and the headset on. It writes xrquadtest.log to the working directory.

I first reported this as #10275, which was moved into #291 because I found it with a mod. The repro above shows that it is independent of any game. Mod-side report: https://github.com/dubrovskiy-yevhen-stakelogic/vice-city-vr/issues/22

xrquadtest.cpp 
// Minimal OpenXR + D3D11 repro for the wineopenxr layer conversion bug.
// Cycles every 5 seconds through three layer lists and counts xrEndFrame results:
//   A: [projection, quad]               (one quad, expected to work)
//   B: [projection, quad, quad]         (two quads, fails under wineopenxr)
//   C: [quad(tiny), projection, quad, quad]  (workaround: quad first)
// Runs for 45 seconds, prints a summary and writes it to xrquadtest.log.
#define WIN32_LEAN_AND_MEAN
#include <windows.h>
#include <d3d11.h>
#include <dxgi1_2.h>
#define XR_USE_GRAPHICS_API_D3D11
#define XR_USE_PLATFORM_WIN32
#include <openxr/openxr.h>
#include <openxr/openxr_platform.h>
#include <stdio.h>
#include <stdarg.h>
#include <string.h>
#include <vector>

#pragma comment(lib, "d3d11.lib")
#pragma comment(lib, "dxgi.lib")
#pragma comment(lib, "openxr_loader.lib")

static FILE *gLog;
static void Log(const char *fmt, ...)
{
	va_list a;
	va_start(a, fmt); vprintf(fmt, a); va_end(a);
	if(gLog){ va_start(a, fmt); vfprintf(gLog, fmt, a); va_end(a); fflush(gLog); }
}

#define CHECK(call) do { XrResult r_ = (call); if(XR_FAILED(r_)){ \
	Log("FAILED %s -> %d\n", #call, (int)r_); return 1; } } while(0)

struct Swapchain
{
	XrSwapchain handle = XR_NULL_HANDLE;
	int width = 0, height = 0;
	std::vector<ID3D11RenderTargetView*> views;
};

static ID3D11Device *gDevice;
static ID3D11DeviceContext *gContext;

static bool CreateSwapchain(XrSession session, int64_t format, int width, int height, Swapchain &sc)
{
	XrSwapchainCreateInfo info = { XR_TYPE_SWAPCHAIN_CREATE_INFO };
	info.usageFlags = XR_SWAPCHAIN_USAGE_COLOR_ATTACHMENT_BIT;
	info.format = format;
	info.sampleCount = 1;
	info.width = width;
	info.height = height;
	info.faceCount = 1;
	info.arraySize = 1;
	info.mipCount = 1;
	if(XR_FAILED(xrCreateSwapchain(session, &info, &sc.handle))) return false;
	sc.width = width; sc.height = height;
	uint32_t count = 0;
	xrEnumerateSwapchainImages(sc.handle, 0, &count, nullptr);
	std::vector<XrSwapchainImageD3D11KHR> images(count, { XR_TYPE_SWAPCHAIN_IMAGE_D3D11_KHR });
	xrEnumerateSwapchainImages(sc.handle, count, &count, (XrSwapchainImageBaseHeader*)images.data());
	for(auto &img : images){
		D3D11_RENDER_TARGET_VIEW_DESC desc = {};
		desc.Format = (DXGI_FORMAT)format;
		desc.ViewDimension = D3D11_RTV_DIMENSION_TEXTURE2D;
		ID3D11RenderTargetView *rtv = nullptr;
		if(FAILED(gDevice->CreateRenderTargetView(img.texture, &desc, &rtv))) return false;
		sc.views.push_back(rtv);
	}
	return true;
}

static void Fill(Swapchain &sc, float r, float g, float b, float a)
{
	uint32_t index = 0;
	XrSwapchainImageAcquireInfo acquire = { XR_TYPE_SWAPCHAIN_IMAGE_ACQUIRE_INFO };
	xrAcquireSwapchainImage(sc.handle, &acquire, &index);
	XrSwapchainImageWaitInfo wait = { XR_TYPE_SWAPCHAIN_IMAGE_WAIT_INFO };
	wait.timeout = XR_INFINITE_DURATION;
	xrWaitSwapchainImage(sc.handle, &wait);
	const float colour[4] = { r, g, b, a };
	gContext->ClearRenderTargetView(sc.views[index], colour);
	XrSwapchainImageReleaseInfo release = { XR_TYPE_SWAPCHAIN_IMAGE_RELEASE_INFO };
	xrReleaseSwapchainImage(sc.handle, &release);
}

static XrCompositionLayerQuad MakeQuad(XrSpace space, Swapchain &sc, float x, float size)
{
	XrCompositionLayerQuad q = { XR_TYPE_COMPOSITION_LAYER_QUAD };
	q.layerFlags = XR_COMPOSITION_LAYER_BLEND_TEXTURE_SOURCE_ALPHA_BIT;
	q.space = space;
	q.eyeVisibility = XR_EYE_VISIBILITY_BOTH;
	q.subImage.swapchain = sc.handle;
	q.subImage.imageRect.extent = { sc.width, sc.height };
	q.pose.orientation.w = 1.0f;
	q.pose.position = { x, 0.0f, -1.0f };
	q.size = { size, size };
	return q;
}

int main()
{
	gLog = fopen("xrquadtest.log", "w");
	const char *extensions[] = { XR_KHR_D3D11_ENABLE_EXTENSION_NAME };
	XrInstanceCreateInfo ici = { XR_TYPE_INSTANCE_CREATE_INFO };
	strcpy(ici.applicationInfo.applicationName, "xrquadtest");
	ici.applicationInfo.apiVersion = XR_API_VERSION_1_0;
	ici.enabledExtensionCount = 1;
	ici.enabledExtensionNames = extensions;
	XrInstance instance;
	CHECK(xrCreateInstance(&ici, &instance));
	XrInstanceProperties ip = { XR_TYPE_INSTANCE_PROPERTIES };
	xrGetInstanceProperties(instance, &ip);
	Log("Runtime: %s %u.%u.%u\n", ip.runtimeName, XR_VERSION_MAJOR(ip.runtimeVersion),
		XR_VERSION_MINOR(ip.runtimeVersion), XR_VERSION_PATCH(ip.runtimeVersion));

	XrSystemGetInfo sgi = { XR_TYPE_SYSTEM_GET_INFO };
	sgi.formFactor = XR_FORM_FACTOR_HEAD_MOUNTED_DISPLAY;
	XrSystemId system;
	CHECK(xrGetSystem(instance, &sgi, &system));

	PFN_xrGetD3D11GraphicsRequirementsKHR getReq = nullptr;
	CHECK(xrGetInstanceProcAddr(instance, "xrGetD3D11GraphicsRequirementsKHR", (PFN_xrVoidFunction*)&getReq));
	XrGraphicsRequirementsD3D11KHR req = { XR_TYPE_GRAPHICS_REQUIREMENTS_D3D11_KHR };
	CHECK(getReq(instance, system, &req));

	IDXGIFactory1 *factory = nullptr;
	CreateDXGIFactory1(__uuidof(IDXGIFactory1), (void**)&factory);
	IDXGIAdapter1 *adapter = nullptr, *chosen = nullptr;
	for(UINT i = 0; factory->EnumAdapters1(i, &adapter) != DXGI_ERROR_NOT_FOUND; i++){
		DXGI_ADAPTER_DESC1 d; adapter->GetDesc1(&d);
		if(!memcmp(&d.AdapterLuid, &req.adapterLuid, sizeof(LUID))){ chosen = adapter; break; }
		adapter->Release();
	}
	D3D_FEATURE_LEVEL level = req.minFeatureLevel > D3D_FEATURE_LEVEL_11_0 ? req.minFeatureLevel : D3D_FEATURE_LEVEL_11_0;
	if(FAILED(D3D11CreateDevice(chosen, chosen ? D3D_DRIVER_TYPE_UNKNOWN : D3D_DRIVER_TYPE_HARDWARE,
		nullptr, 0, &level, 1, D3D11_SDK_VERSION, &gDevice, nullptr, &gContext))){
		Log("D3D11CreateDevice failed\n"); return 1;
	}

	XrGraphicsBindingD3D11KHR binding = { XR_TYPE_GRAPHICS_BINDING_D3D11_KHR };
	binding.device = gDevice;
	XrSessionCreateInfo sci = { XR_TYPE_SESSION_CREATE_INFO };
	sci.next = &binding;
	sci.systemId = system;
	XrSession session;
	CHECK(xrCreateSession(instance, &sci, &session));

	XrReferenceSpaceCreateInfo rsi = { XR_TYPE_REFERENCE_SPACE_CREATE_INFO };
	rsi.poseInReferenceSpace.orientation.w = 1.0f;
	rsi.referenceSpaceType = XR_REFERENCE_SPACE_TYPE_LOCAL;
	XrSpace local, view;
	CHECK(xrCreateReferenceSpace(session, &rsi, &local));
	rsi.referenceSpaceType = XR_REFERENCE_SPACE_TYPE_VIEW;
	CHECK(xrCreateReferenceSpace(session, &rsi, &view));

	uint32_t formatCount = 0;
	xrEnumerateSwapchainFormats(session, 0, &formatCount, nullptr);
	std::vector<int64_t> formats(formatCount);
	xrEnumerateSwapchainFormats(session, formatCount, &formatCount, formats.data());
	int64_t format = formats.empty() ? DXGI_FORMAT_R8G8B8A8_UNORM_SRGB : formats[0];
	for(int64_t preferred : { (int64_t)DXGI_FORMAT_R8G8B8A8_UNORM_SRGB, (int64_t)DXGI_FORMAT_B8G8R8A8_UNORM_SRGB, (int64_t)DXGI_FORMAT_R8G8B8A8_UNORM })
		for(int64_t f : formats) if(f == preferred){ format = f; goto picked; }
picked:
	Log("Swapchain format %lld\n", (long long)format);

	uint32_t viewCount = 0;
	xrEnumerateViewConfigurationViews(instance, system, XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO, 0, &viewCount, nullptr);
	std::vector<XrViewConfigurationView> configViews(viewCount, { XR_TYPE_VIEW_CONFIGURATION_VIEW });
	xrEnumerateViewConfigurationViews(instance, system, XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO, viewCount, &viewCount, configViews.data());
	std::vector<Swapchain> eyes(viewCount);
	for(uint32_t i = 0; i < viewCount; i++)
		if(!CreateSwapchain(session, format, configViews[i].recommendedImageRectWidth,
			configViews[i].recommendedImageRectHeight, eyes[i])){ Log("eye swapchain failed\n"); return 1; }
	Swapchain red, green;
	if(!CreateSwapchain(session, format, 256, 256, red) || !CreateSwapchain(session, format, 256, 256, green)){
		Log("quad swapchain failed\n"); return 1;
	}

	const char *names[] = { "A [projection, quad]", "B [projection, quad, quad]", "C [quad(tiny), projection, quad, quad]" };
	unsigned ok[3] = {}, fail[3] = {};
	XrResult lastFail[3] = {};
	bool running = false, quit = false;
	ULONGLONG start = 0;
	while(!quit){
		XrEventDataBuffer ev = { XR_TYPE_EVENT_DATA_BUFFER };
		while(xrPollEvent(instance, &ev) == XR_SUCCESS){
			if(ev.type == XR_TYPE_EVENT_DATA_SESSION_STATE_CHANGED){
				const XrSessionState state = ((XrEventDataSessionStateChanged*)&ev)->state;
				if(state == XR_SESSION_STATE_READY){
					XrSessionBeginInfo bi = { XR_TYPE_SESSION_BEGIN_INFO };
					bi.primaryViewConfigurationType = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO;
					CHECK(xrBeginSession(session, &bi));
					running = true;
					start = GetTickCount64();
				}else if(state == XR_SESSION_STATE_STOPPING){
					xrEndSession(session); running = false; quit = true;
				}else if(state == XR_SESSION_STATE_EXITING || state == XR_SESSION_STATE_LOSS_PENDING){
					quit = true;
				}
			}
			ev = { XR_TYPE_EVENT_DATA_BUFFER };
		}
		if(!running){ Sleep(10); continue; }
		const ULONGLONG elapsed = GetTickCount64()-start;
		if(elapsed > 45000){ xrRequestExitSession(session); running = false; quit = true; break; }
		const int mode = (int)((elapsed/5000)%3);

		XrFrameState fs = { XR_TYPE_FRAME_STATE };
		XrFrameWaitInfo fwi = { XR_TYPE_FRAME_WAIT_INFO };
		CHECK(xrWaitFrame(session, &fwi, &fs));
		XrFrameBeginInfo fbi = { XR_TYPE_FRAME_BEGIN_INFO };
		CHECK(xrBeginFrame(session, &fbi));

		XrViewLocateInfo vli = { XR_TYPE_VIEW_LOCATE_INFO };
		vli.viewConfigurationType = XR_VIEW_CONFIGURATION_TYPE_PRIMARY_STEREO;
		vli.displayTime = fs.predictedDisplayTime;
		vli.space = local;
		XrViewState vs = { XR_TYPE_VIEW_STATE };
		std::vector<XrView> views(viewCount, { XR_TYPE_VIEW });
		uint32_t located = 0;
		xrLocateViews(session, &vli, &vs, viewCount, &located, views.data());

		std::vector<XrCompositionLayerProjectionView> pviews(viewCount, { XR_TYPE_COMPOSITION_LAYER_PROJECTION_VIEW });
		for(uint32_t i = 0; i < viewCount; i++){
			Fill(eyes[i], 0.05f, 0.05f, 0.2f, 1.0f);
			pviews[i].pose = views[i].pose;
			pviews[i].fov = views[i].fov;
			pviews[i].subImage.swapchain = eyes[i].handle;
			pviews[i].subImage.imageRect.extent = { eyes[i].width, eyes[i].height };
		}
		Fill(red, 1.0f, 0.0f, 0.0f, 1.0f);
		Fill(green, 0.0f, 1.0f, 0.0f, 1.0f);

		XrCompositionLayerProjection proj = { XR_TYPE_COMPOSITION_LAYER_PROJECTION };
		proj.space = local;
		proj.viewCount = viewCount;
		proj.views = pviews.data();
		XrCompositionLayerQuad quadRed = MakeQuad(view, red, -0.15f, 0.2f);
		XrCompositionLayerQuad quadGreen = MakeQuad(view, green, 0.15f, 0.2f);
		XrCompositionLayerQuad guard = MakeQuad(view, red, 0.0f, 0.001f);

		const XrCompositionLayerBaseHeader *layers[4];
		uint32_t count = 0;
		if(mode == 2) layers[count++] = (XrCompositionLayerBaseHeader*)&guard;
		layers[count++] = (XrCompositionLayerBaseHeader*)&proj;
		layers[count++] = (XrCompositionLayerBaseHeader*)&quadRed;
		if(mode != 0) layers[count++] = (XrCompositionLayerBaseHeader*)&quadGreen;

		XrFrameEndInfo fei = { XR_TYPE_FRAME_END_INFO };
		fei.displayTime = fs.predictedDisplayTime;
		fei.environmentBlendMode = XR_ENVIRONMENT_BLEND_MODE_OPAQUE;
		fei.layerCount = fs.shouldRender ? count : 0;
		fei.layers = layers;
		const XrResult r = xrEndFrame(session, &fei);
		if(fs.shouldRender){
			if(XR_SUCCEEDED(r)) ok[mode]++; else { fail[mode]++; lastFail[mode] = r; }
		}
	}

	Log("\nResults (xrEndFrame):\n");
	for(int m = 0; m < 3; m++)
		Log("  %-42s ok=%u fail=%u lastError=%d\n", names[m], ok[m], fail[m], (int)lastFail[m]);
	xrDestroySession(session);
	xrDestroyInstance(instance);
	if(gLog) fclose(gLog);
	return 0;
}
Kkisak-valve maintainer 2026-10-17 github

@CruCo, rhetorically asking, how am I supposed to interpret this in any way other than you told your AI tool to create a testcase as an excuse to re-post your issue report? Your original feedback isn't less valid because it was put on the game's compatibility report. Please do not intentionally open duplicate issue reports.

Closing as a duplicate of #10275 / https://github.com/ValveSoftware/Proton/issues/291#issuecomment-6235815259.

Related threads

Proton versions

Upstream links