protonscr

Some special DLL load address cause DXVK crash on illegal instruction (UD2)

dxvkclosed
doitsujin/dxvk#4474 · opened 2024-11-17 by zhangboyang · updated 2024-12-02 · 15 comments · github
Zzhangboyang 2024-11-17 github

I'm using DXVK 2.5 on Win10 22H2. I found if DXVK's d3d9.dll loaded at some special addresses (e.g. 0x6FE90000), it will crash game with illegal instruction (UD2). Some other addresses (like 0x6EED0000) can cause game exit with code 3, and some addesses will operate normally. My quick analysis shows, the problem might related to https://github.com/doitsujin/dxvk/blob/56b7c8cd41ea17c0656f9485730fea5d55b3db62/src/d3d9/d3d9_common_texture.cpp#L404

I created a small sample program which can trigger this throw:

	if (FAILED(g_pd3dDevice->CreateTexture(256, 256, 9, D3DUSAGE_AUTOGENMIPMAP, D3DFMT_A4R4G4B4, D3DPOOL_DEFAULT, &g_pTexture, NULL))) {
		MessageBox(NULL, L"that's ok!", L"Textures.exe", MB_OK);
	}

Instead of returning an error code, dxvk's d3d9.dll crashed the application.

(1) exception string in lower left dump window:
1_throw
(2) continue, and it crashes on ud2
2_crash
(3) d3d9.dll is loaded at 0x6FE90000
3_modules

About how to create a d3d9.dll which always loads at 0x6FE90000

(1) use editbin.exe to relocate it to 0x6FE90000:

editbin /rebase:base=0x6FE90000 d3d9.dll

(2) use hex editor to add IMAGE_FILE_RELOCS_STRIPPED to FileHeader.Characteristics of d3d9.dll
IMAGE_FILE_RELOCS_STRIPPED

For your convenience, the EXE, DLL and log files is here:

bug.zip

System information

  • GPU: NVIDIA
  • Driver: 553.24.0
  • Wine version: Windows 10
  • DXVK version: 2.5
Zzhangboyang 2024-11-17 github

It's strange that the release DLL contains UD2 instruction, it might indicate compiler found undefined behavior in source code.(edit: this is not true.) I will investigate it tomorrow.

Zzhangboyang 2024-11-18 github

I compiled my own d3d9.dll (using git tag v2.5) on Ubuntu 24.04. However, I can't reproduce the buggy behavior using my own d3d9.dll (I tried a lot of different DLL base addresses). It doesn't contains '0F 0B 0F 0B' (two successive UD2 instruction) at all.

I found something problematic in release binary: strings d3d9.dll | grep GCC shows both GCC: (GNU) 12.2.0 and GCC: (GNU) 13.1.0. This probably indicates mixing compiled objects from different versions of GCC, which may cause problems. (edit: not related) The mingw toolchain on release machine may be broken.

The d3d9.dll in release package is stripped, so I can hardly confirm the situation. I'd like to suggest add non-stripped binary (or debug symbols) along with release package in future releases.

Zzhangboyang 2024-11-18 github

release v2.5.1 still have this issue.

BBlisto91 2024-11-19 github

I found something problematic in release binary: strings d3d9.dll | grep GCC shows both GCC: (GNU) 12.2.0 and GCC: (GNU) 13.1.0. This probably indicates mixing compiled objects from different versions of GCC, which may cause problems. The mingw toolchain on release machine may be broken.

I am not sure what this entails or if it even is a issue, but it seems to just be a general thing when compiling Windows files with mingw-w64-gcc on Arch. At least it looks the same in other similar projects that also compile on Arch.

Zzhangboyang 2024-11-19 github

I am not sure what this entails or if it even is a issue, but it seems to just be a general thing when compiling Windows files with mingw-w64-gcc on Arch. At least it looks the same in other similar projects that also compile on Arch.

C has stable ABI, but C++ ABI often changes across GCC versions, so mixing C++ objects compiled by different versions of GCC(G++) is dangerous. I will try to use Arch Linux to compile my own d3d9.dll and find out what's happening.

By the way, is this bug reproducible on your machine? You can use the EXE in zip bundle to test, or you can compile it yourself using the source code in zip bundle. The source code is based on Tut05_Textures of DirectX SDK (June 2010).

Qqinlili23333 2024-11-21 github

Seems to be toolchain issue of github actions.
I tested files generated from actions (https://github.com/doitsujin/dxvk/actions/runs/11891586570) has same behavior.
However, if I compile on my own device with MSVC, there is no issue. My compiled version is below.
d3d9.zip
image

I double checked the action workflow file and cannot find anything about fixed gcc version. https://github.com/doitsujin/dxvk/blob/master/.github/workflows/artifacts.yml
I think there should be sth wrong with github action runner image.

And I have checked it's NOT bring in with d13375f4ed2bcd2e63a6b585eaa38ac8663e83eb because the commit before this also has same issue (https://github.com/doitsujin/dxvk/actions/runs/11767191817)

In short term, switching to MSVC is a solution, although I really don't like the redist requirement introduced by MSVC.

BBlisto91 2024-11-21 github

By the way, is this bug reproducible on your machine? You can use the EXE in zip bundle to test, or you can compile it yourself using the source code in zip bundle. The source code is based on Tut05_Textures of DirectX SDK (June 2010).

I haven't checked yet. Will here later.

I double checked the action workflow file and cannot find anything about fixed gcc version.

The artifacts CI just use a regular Arch Linux docker image and compile with mingw-w64-gcc from the Arch repository.
https://github.com/misyltoad/arch-mingw-github-action

Zzhangboyang 2024-11-23 github

Thanks for testing @qinlili23333, and thanks for providing information about ArchLinux @Blisto91. I finally figured out what's happening. It's this bug: Exception handling broken for 32-bit Windows. This bug affected GCC 13.1.0 and it was fixed in GCC 13.2.0. Unfortunately, ArchLinux's mingw-w64-gcc is exactly 13.1.0, which is buggy. I have no idea of why ArchLinux has a lower version of mingw than Ubuntu LTS (13.2.0 on 24.04). It seems mingw is not actively maintained in ArchLinux? At least it received no updates for almost one year. @doitsujin I suggest migrate to another toolchain for future builds (at least temporarily until ArchLinux fixed their toolchain). It would be appreciated if providing both mingw builds and MSVC builds. MSVC redist dependency can be eliminated by linking to static C/C++ library. MSVC (and clang) also has a advantage: address sanitizer can be enabled.

By the way, if anybody interested, here is a dirty fix for this problem: just use your favorite hex editor, search for adjacent bytes and replace 0x25 with 0x48:
fix

MMartchus 2024-11-24 github

I have no idea of why ArchLinux has a lower version of mingw than Ubuntu LTS (13.2.0 on 24.04).

The maintainer had no interest in mingw-w64 anymore so the package was moved to the AUR. This was a good thing as it allowed community members to maintain the package there. Then the package was moved back to official repos but only for the sake of building WINE. For this staying at an old version seems sufficient but for other users this is problematic.

If you want to stay with Arch I can only self-promote my PKGBUILDs repo, see its README for details. The only caveat is that it uses the ucrt but at this point this is hopefully ok.

Qqinlili23333 2024-11-25 github

Since we have already tested MSVC build for a long time (https://github.com/doitsujin/dxvk/actions/workflows/test-build-windows.yml ), I think we should consider to migrate to MSVC build as release until mingw build is fixed, or we can provide both mingw and MSVC releases like some other projects.
Although we may need some additional tests about performance and compatibility of MSVC builds.

BBananaWorks07 2024-11-25 github

found this AUR package that might be helpful: https://aur.archlinux.org/packages/mingw-w64-gcc132

MMartchus 2024-11-25 github

Of course this package is now also a bit dated (as it is pinned to a certain version - not really the Arch spirit). That's why I mentioned my gcc PGKBUILD. When updating this PKGBUILD I also have a look at MSYS2 to see what they are doing (as there might be critical patches to apply). Note that my PKGBUILD only cannot be uploaded to the AUR because AUR packages must not have the same name as official packages. I also have newer versions of mingw-w64 itself and binutils which can also generally be beneficial.

BBlisto91 2024-11-27 github

@zhangboyang Is the issue supposed to show out of the box when using the files provided in your archive? It is running fine for me here through wine using the d3d9.dll you shipped.

Either way i have done some poking and the Arch mingw-w64 packages have moved maintainer as a result. gcc version 14.2.0 is currently sitting in staging getting some testing. Thank you for reporting the problem

Zzhangboyang 2024-11-30 github

@zhangboyang Is the issue supposed to show out of the box when using the files provided in your archive? It is running fine for me here through wine using the d3d9.dll you shipped.

Yes, it should crash immediately at startup. There might be some random factor cause it not crash on your machine. For example, your system may choose to relocate it to other address despite IMAGE_FILE_RELOCS_STRIPPED flag is set. You can use VMMap to verify the base address of d3d9.dll.
vmmap

Either way i have done some poking and the Arch mingw-w64 packages have moved maintainer as a result. gcc version 14.2.0 is currently sitting in staging getting some testing. Thank you for reporting the problem

That's great! :thumbsup:

BBlisto91 2024-12-02 github

The mingw-w64 update is now out in the regular Arch repo

DLLs