May be start from just -flto ?

As an additional note, it might be a good idea to explore including Link Time Optimization (LTO) alongside PGO. There will be a need to tell the compiler what is externally visible. Supposedly, the gold linker can be used to help with this, but that would need investigation.

Another idea is to to try concatenating all of the .cpp files and building them with -fwhole-program. This will require marking public functions with externally_visible, although it should generate a very well optimized binary.

@lieff You beat me to posting it. I edited the title to reflect the nature of this issue as encompassing more than just PGO.

Quite frankly, I suspect that concatenating all of the files into a single compilation unit and then using -fwhole-program would be better than LTO, but it is up to the person who volunteers to explore this to decide what to try.

Edit: Concatenating all of the files together and building them together is similar to Chromium's jumbo builds, although doing it to enable -fwhole-program would mean that it is for inter-procedural optimizations rather than reducing compile time:

https://chromium.googlesource.com/chromium/src/+/lkcr/docs/jumbo.md

Also, here is another thought. It would be interesting to try using LLVM/Clang with Google's Souper optimizer, especially with the other optimizations mentioned in place (i.e. PGO and WPO/LTO):

https://github.com/google/souper

There are other "superoptimizers" available that probably could be evaluated. They would make compile times skyrocket (taking days to months depending on how they are configured), but I hear that they can provide additional performance. I'd stick to the relatively low dangling fruit of PGO and LTO or a jumbo build with whole program optimization first though.

@ryao

it is up to the person who volunteers to explore this

why not you?

http://mesonbuild.com/Builtin-options.html#base-options
See b_lto, b_pgo and --unity

1. What benchmark can be run to generate profile data?

A bunch of unit tests covering all aspects for general optimization, or an concrete game you want optimize DXVK for.

Meson already have b_lto and b_pgo parameters, so it's build/packaging question, not really project related.

@pchome I am not sure if I have time. If I thought I had time to do it, I would have done it rather than posting about it. We'll see if I do, but I find it doubtful.

A bunch of unit tests covering all aspects for general optimization, or an concrete game you want optimize DXVK for.

The problem with games is that they rely on user input. I suspect that a game would be better than unit tests (although both could be run). We would need some way to start one from the commandline, have it run through a benchmark and then quit.

@lieff How builds work is project related for any project.

@doitsujin My replies are inline:

I can already tell you it won't.

That is unfortunate. Would you mind sharing how you profile? If I recall correctly, my usual profiling tricks don't give me much visibility into binaries running in Wine.

The shader compiler-related stutter happens inside the driver, and it is inherently slow due to differences in the D3D11 and Vulkan designs.

Would you name a few of the differences? I would like to know more. Are you referring to things like D3D binding slots vs vulkan descriptor sets?

Might still be worth looking at

I suggest leaving it to a volunteer and putting the help-wanted label on this. This sort of experiment is something a volunteer could do.

PGO only really helps optimize for one specific workload

That is what I thought until I saw that Firefox improved its Javascript performance in general with PGO.

LTO is notoriously broken

If it were up to me, I'd probably just dump all *.cpp files into a single file and then build with -fwhole-program. It is less fragile than LTO and should work just as well. The caveat about needing to mark public functions/variables with externally_visible does apply. Otherwise, breakage will occur when the symbols are optimized away. Another issue would be that it would reduce the information available in backtraces.

I'd probably just dump all *.cpp files into a single file

that's how meson unity builds (--unity on) work, but per module
http://mesonbuild.com/Unity-builds.html#unity-builds

@doitsujin I take it that your profiling shows that most of the time there is spent in the graphics driver. This might be asking the obvious question, but is there no way to parallelize that process?

For example, n shaders A[i] for i from 0 to n must be built, so m worker threads from j = 0 to m - 1 are created and they each do every A[i] where i % m == j. After they are all finished, the main thread just gathers all of the work from the worker threads. My feeling is that it is not that straightforward, but you piqued my curiosity.

Also, LTO won't work for winelib builds, because you need LTOed WINE, or particularly libwinecrt0.a.

I'm using LTO and PGO for my whole system wherever possible, and WINE is one of the unreached goals.

@pchome I was leaning toward thinking that a so called unity build with -fwhole-program would be better than LTO. As I said above, LTO is fragile. If you build everything as one compilation unit with -fwhole-program, you don't need LTO.

@doitsujin Nevermind about the parallelism. I need to do my own profiling. I have spent more time looking at this code than I really have at the moment, but I think I understand a few bits of it. In particular, the draw calls that you mentioned are likely in DxvkContext::commitGraphicsState. Given how much this piqued my interest, I'll probably profile the code at some point and learn where the time is being spent. Concurrent programming is always fun. ;)

Then note https://github.com/doitsujin/dxvk/commit/357277563563ca356323105c1121f0cf3a90041f#diff-b396766df186599eed8a7aff228ec9f7

@pchome That is good to know. I still think opportunities for interprocedural optimization from those (with -fwhole-program) could be a low dangling fruit for someone who has only minor programming knowledge to explore.

After reading what @doitsujin said and looking through the code, I found some more interesting avenues to explore. In particular, I am not seeing much threading and I see no use of machine prefetch hints in the code. I need to make time to profile to see where the bottlenecks are more clearly.

After some thought, I think I should close this. It is probably not a great use of people's time, although I did learn some interesting things from the discussion.

Why so? At least PGO is real, and quite easy to test.

The only thing we should do -- create a list of small tests (maybe wine's d3d11 tests, or some other d3d11 demos), and define a final benchmark to test results.

e.g.

#!/bin/sh
run_benchmark
buld_pofile
run_tests
use_profile
run_benchmark

Alright. I am reopening this.

@doitsujin One last thing as I could not help myself from eyeballing the code a bit more. Does your profiling indicate that the shuttering is from dxvkgraphicspipeline::DxvkGraphicsPipeline()? I see 5 ->createShaderModule() calls there that probably could run in parallel.

@doitsujin That is tricky. Couldn't you just cache the DXBC shaders and other things that DXVK receives from the game and turns into a pipeline? Then on subsequent runs, if one of the shaders from a previous session are loaded by a game (identified by a matching checksum), DXVK could load the rest from cache and pre-create the pipeline? That is just a rough idea, but some kind of driver independent cache seems like the only way around it.

@pchome

create a list of small tests
maybe wine's d3d11 tests

Ok, I able to build standalone dxgi test from wine sources, it's executing quickly and looks like it can be used for PGO needs.
0026:dxgi: 6386 tests executed (0 marked as todo, 270 failures), 5 skipped.

I going to do the same for other wine's dx10/dx11 related tests, and combine all together before sharing.

EDIT:
dxgi.test.txt
dxgi_dxgi.log.txt

https://github.com/pchome/wine-playground/tree/master/dx1x-tests

Note:

• winelib build
• unmodified WINE sources
• test-run.sh contains examples how to run
• only dxgi and d3d11 tests finished correctly
• d3d10.device is an RAM-consuming evil,
  failed with unimplemented function d3d10.dll.D3D10StateBlockMaskDifference
• d3d10.effect - effects not supported in DXVK
• d3d10_1 failed w/ exception and d3d10core failed w/ segfault

d3d11 test out: 0025:d3d11: 1154 tests executed (0 marked as todo, 201 failures), 1 skipped.

So dxgi and d3d11 tests could be used as is (for now), despite failures.
Others are requires a patching work.

The tests that failed probably merit their own issues.

Mostly no, missing interfaces, specific formats and probably WINE's internal stuff.

err:   D3D11: Cannot create texture:
  Format:  VK_FORMAT_E5B9G9R9_UFLOAT_PACK32
  Extent:  512x512x1
  Samples: 1
  Layers:  1
  Levels:  1
  Usage:   13

err:   DXGI: CheckInterfaceSupport: Unsupported interface
err:   db6f6ddb-ac77-4e88-8253-819df9bbf140

@doitsujin can check it by himself , if he'll want to.

EDIT: A lot of them (tests) are failing even for wine.
http://test.winehq.org/data/
http://test.winehq.org/data/64d9f309b7f74d4154e685c5d1d78c1b8335c0bc/index_Linux.html

I have a theory on why unity builds took longer. For large projects, the headers can be substantially more complex than the files themselves. Furthermore, you can have many files to compile such that even with -j$(nproc), each core must process a large number of them. The time savings from unity builds comes from parsing the headers only once for all of those files. If all of the additional time spent parsing all of the files that would be handled on other cores is less than the savings from not parsing the headers once for each file on a single core, you save time. If not, you do not save time.

I believe that DXVK’s headers are not complex enough to save time with unity builds. However, there should be opportunities for strong interprocedural optimizations from unity builds if DXVK is adapted to support -fwhole-program as part of them. This means marking functions externally accessible with the always_visible attribute according to the compiler documentation. This idea needs testing to see if it makes a difference.

@ryao
I might be misunderstanding you, but if not: "unity builds" was cancelled due to performance degradation, and not "build time".

Not tested it since it was dropped, so for all i know it might not be an issue anymore?

@SveSop I misremembered what I read when I was thinking about it then. Anyway, I would not expect unity builds to be a performance win unless -fwhole-program Is used. That needs function annotations with the always_visible attribute. If performance still degrades performance with -fwhole-program, then some attention probably needs to be given to the compiler’s optimization stages to see what it is doing wrong and how we can toggle switches to get it to do things correctly.

This was a quick stab at producing d3d11.dll and dxgi.dll files built with -fwhole-program for evaluation purposes:

cd /path/to/dxvk
meson --unity on --cross-file build-win32.txt --prefix /tmp/dxvk-win32-whole-program build.w32
cd build.w32
meson configure -Dbuildtype=release
ninja

cat << END > dxgi.dll.c
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/dxgi/src@dxgi@@dxgi@sha/dxgi-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

i686-w64-mingw32-g++ -fwhole-program -std=c++1z -O2 -g -o src/dxgi/dxgi.dll ../src/dxgi/dxgi.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/dxgi/dxgi.def -Wl,--start-group -Wl,--out-implib=src/dxgi/libdxgi.dll.a dxgi.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w32/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w32/src/dxgi/src@dxgi@@dxgi@sha/ ../lib32/vulkan-1.lib  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

cat << END > d3d11.dll.c
#include "src/d3d11/src@d3d11@@d3d11@sha/d3d11-unity.cpp"
#include "src/dxbc/src@dxbc@@dxbc@sta/dxbc-unity.cpp"
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

i686-w64-mingw32-g++ -std=c++1z -O2 -g  -o src/d3d11/d3d11.dll ../src/d3d11/d3d11.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/d3d11/d3d11.def -Wl,--start-group -Wl,--out-implib=src/d3d11/libd3d11.dll.a d3d11.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w32/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w32/src/dxgi/src@dxgi@@dxgi@sha/ ../lib32/vulkan-1.lib -ldxgi /home/richard/devel/dxvk/lib32/vulkan-1.lib -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

ninja install

Contrary to my belief, setting always_visible was unnecessary. This was confirmed by quick examination of The Export Tables (interpreted .edata section contents) via i686-w64-mingw32-objdump, which showed that the same symbols were being exported, plus a quick runtime test.

I was able to replace the d3d11.dll and dxgi.dll files provided with proton and it ran without an problem. I am certain that I did replace the correct binary because DXVK_HUD=version is showing a changed version number.

It also might be of interest that the binaries built this way are smaller after stripping.

Before:

richard@desktop ~/devel/dxvk $ ls -l /tmp/dxvk-win32-v0.72-39-g20c89c3/{d3d11.dll,dxgi.dll}
-rwxr-xr-x 1 richard richard 2353664 Sep 23 00:26 /tmp/dxvk-win32-v0.72-39-g20c89c3/d3d11.dll
-rwxr-xr-x 1 richard richard 1845248 Sep 23 00:26 /tmp/dxvk-win32-v0.72-39-g20c89c3/dxgi.dll

After:

richard@desktop ~/devel/dxvk $ cp ./build.w32/src/d3d11/d3d11.dll ./build.w32/src/dxgi/dxgi.dll /tmp/
richard@desktop ~/devel/dxvk $ i686-w64-mingw32-strip /tmp/dxgi.dll /tmp/d3d11.dll 
richard@desktop ~/devel/dxvk $ ls -l  /tmp/dxgi.dll /tmp/d3d11.dll 
-rwxr-xr-x 1 richard richard 1959936 Sep 24 12:50 /tmp/d3d11.dll
-rwxr-xr-x 1 richard richard 1414144 Sep 24 12:50 /tmp/dxgi.dll

I am sharing this in case someone else who has more time wants to test this to see if it helps. However, I suspect that doing this for release builds might be worthwhile for the smaller binary sizes, even if performance does not improve, as long as performance does not become worse. The build system would need to be fixed to avoid the horrible hack that I did to make the PoC though.

Do "Before" version was built using same flags, except -fwhole-program ?

-Dbuildtype=release will add -O3 flag, so binaries expected to be bigger.

@pchome The before build was built like this:

meson --cross-file build-win32.txt --prefix /tmp/dxvk-win32
cd build.w32
meson configure -Dbuildtype=release
ninja
ninja install

I manually moved the files and stripped them afterward.

I didn't capture the CFLAGS being used for the build, so I didn't check. The size difference had been unexpected and was included in my comment at the last moment. A quick grep of the sources didn't show me any CFLAGS and I am not familiar with meson. However, I just tested -O3 builds out of curiosity:

richard@desktop ~/devel/dxvk $ ls -l  /tmp/dxgi.dll /tmp/d3d11.dll
-rwxr-xr-x 1 richard richard 2098176 Sep 24 13:13 /tmp/d3d11.dll
-rwxr-xr-x 1 richard richard 1437184 Sep 24 13:13 /tmp/dxgi.dll

They are still smaller.

@doitsujin Why is your official release built with 2 different compilers?

richard@desktop /tmp $ strings dxvk-0.80/x32/d3d11.dll | grep GCC: | sort -u
GCC: (GNU) 4.9.2
GCC: (GNU) 8.1.0

I just checked configure phase, and unity files available on this stage : build.64/src/d3d11/src@d3d11@@d3d11.dll@sha/.

So it's possible to integrate -fwhole-program into build process, by skipping some modules compilation.
I'll check this later.

@pchome It is not quite that simple because there are internal libraries being built. I had to work around that by making a manual unity file combining all of the unity files for those libraries to make it work.

I had to work around that by making a manual unity file combining all of the unity files for those libraries to make it work.

You can use generator
https://github.com/doitsujin/dxvk/blob/master/meson.build#L54
https://github.com/doitsujin/dxvk/blob/master/src/dxgi/meson.build#L19

or (maybe) pass all *_src variables to shared_library().

Also, it may be worth to ask https://github.com/mesonbuild/meson for such (-fwhole-program+unity) feature.

whole-program.patch.txt

A hack for -fwhole-program (for testing purpose)

• rename sha1/sha1.c to sha1/sha1.cpp
• d3d10 disabled
• -Dwhole_program=true by default, so --unity on required
• winelib compiles fine w/o -fwhole-program, otherwise produces errors like dxgi.spec:1: function 'CreateDXGIFactory' not defined
• not tested at all, except compilation

@pchome Were you able to reproduce the smaller binaries?

I have no MinGW installed, and as I said I can't use -fwhole-program with this patch for winelib build.
W/o -fwhole-program almost (different version) equal sized files was generated, compared to those currently installed in system.

This produces two huge d3d11.dll-unity.cpp and dxgi.dll-unity.cpp, but automatically.

In my observations -flto have advantage over unity builds (not much, but still). This because of exported (non-static) symbols must have an ABI in unity build. Compiler can't change it because he do not know if someone wants to call this symbol from resulting object. With -flto compiler makes decision when actually links application, so he can change ABI if he wants. Not sure if we can bypass it with -fvisibility=hidden.

@lieff -fwhole-program tells the toolchain to assume that no one will ever want to call those externally.

Here are instructions for a 64-bit build of the proof of concept that I posted earlier:

cd /path/to/dxvk
meson --unity on --cross-file build-win64.txt --prefix /tmp/dxvk-win64-whole-program build.w64
cd build.w64
meson configure -Dbuildtype=release
ninja

cat << END > dxgi.dll.c
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/dxgi/src@dxgi@@dxgi@sha/dxgi-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

x86_64-w64-mingw32-g++ -fwhole-program -std=c++1z -O2 -g -o src/dxgi/dxgi.dll ../src/dxgi/dxgi.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/dxgi/dxgi.def -Wl,--start-group -Wl,--out-implib=src/dxgi/libdxgi.dll.a dxgi.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w64/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w64/src/dxgi/src@dxgi@@dxgi@sha/ ../lib/vulkan-1.lib  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

cat << END > d3d11.dll.c
#include "src/d3d11/src@d3d11@@d3d11@sha/d3d11-unity.cpp"
#include "src/dxbc/src@dxbc@@dxbc@sta/dxbc-unity.cpp"
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

x86_64-w64-mingw32-g++ -std=c++1z -O2 -g  -o src/d3d11/d3d11.dll ../src/d3d11/d3d11.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/d3d11/d3d11.def -Wl,--start-group -Wl,--out-implib=src/d3d11/libd3d11.dll.a d3d11.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w64/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w64/src/dxgi/src@dxgi@@dxgi@sha/ ../lib/vulkan-1.lib -ldxgi /home/richard/devel/dxvk/lib32/vulkan-1.lib -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

ninja install

I noticed a small mistake in how I was doing the -fwhole-program build of d3d11.dll where src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp was built separately. That kept the binaries from being as well optimized / small as they could have been. I have updated the build instructions and size data with the corrections.

@ryao Yes, it should turn all to static. But it's my old observation that it does not work for C++ member functions (may be because C++ methods can't be static in terms of С++). May be now newer gcc behaves differently, I've look at it long time ago and I've not checked clang.

@ryao

I noticed a small mistake ...

You also should use --unity on for both, so we could see the real -fwhole-program difference/advantage.

Edit: And -O3

@lieff I am building with GCC 8.2.0. I have yet to try Clang

@pchome I am using --unity on for both.

Also, I have proof of concept builds available for those interested in testing them:

wget http://dev.gentoo.org/~ryao/dist/dxvk-win64-v0.80-whole-program.txz{,.sig}
gpg --verify dxvk-win64-v0.80-whole-program.txz{.sig,}

wget http://dev.gentoo.org/~ryao/dist/dxvk-win32-v0.80-whole-program.txz{,.sig}
gpg --verify dxvk-win32-v0.80-whole-program.txz{.sig,}

I already found a volunteer to help test, so there is no need for more, but I'm posting them for the various contributors here to be able to evaluate. You can get my PGP key from github:

https://github.com/ryao.gpg

I included the d3d10 dlls, but they aren't built with -fwhole-program, so there is really nothing special about them beyond being unity builds built with GCC 8.2.0. These binaries have been stripped to save space on the webserver. Also, despite my instructions showing -O2, all of the binaries were built with -O3 to match the release builds. I left the instructions with -O2 to avoid making the history confusing.

@pchome I just realized that you meant before/after. That will need to wait a few days because I have spent all of my spare time on this and then some, but I will be happy to provide numbers for them when I have some more time.

I'm still want to do my own build/test, no luck.
Here is final options passed to compiler by winegcc:

COLLECT_GCC_OPTIONS='-fdiagnostics-color=always' '-fshort-wchar' '-D' 'WINE_UNICODE_NATIVE' \
'-D' '_REENTRANT' '-D' 'WIN64' '-D' '_WIN64' '-D' '__WIN64' '-D' '__WIN64__' '-D' 'WIN32' \
'-D' '_WIN32' '-D' '__WIN32' '-D' '__WIN32__' '-D' '__WINNT' '-D' '__WINNT__' \
'-D' '__stdcall=__attribute__((ms_abi))' '-D' '__cdecl=__attribute__((ms_abi))' \
'-D' '_stdcall=__attribute__((ms_abi))' '-D' '_cdecl=__attribute__((ms_abi))' \
'-D' '__fastcall=__attribute__((ms_abi))' '-D' '_fastcall=__attribute__((ms_abi))' \
'-D' '__declspec(x)=__declspec_##x' '-D' '__declspec_align(x)=__attribute__((aligned(x)))' \
'-D' '__declspec_allocate(x)=__attribute__((section(x)))' \
'-D' '__declspec_deprecated=__attribute__((deprecated))' \
'-D' '__declspec_dllimport=__attribute__((dllimport))' \
'-D' '__declspec_dllexport=__attribute__((dllexport))' \
'-D' '__declspec_naked=__attribute__((naked))' \
'-D' '__declspec_noinline=__attribute__((noinline))' \
'-D' '__declspec_noreturn=__attribute__((noreturn))' \
'-D' '__declspec_nothrow=__attribute__((nothrow))' \
'-D' '__declspec_novtable=__attribute__(())' \
'-D' '__declspec_selectany=__attribute__((weak))' \
'-D' '__declspec_thread=__thread' \
'-D' '__int8=char' '-D' '__int16=short' '-D' '__int32=int' '-D' '__int64=long' '-D' '__WINE__' \
'-c' '-o' 'src/dxgi/src@dxgi@@dxgi.dll@sha/meson-generated_dxgi.dll-unity.cpp.o' \
'-I' 'src/dxgi/src@dxgi@@dxgi.dll@sha' '-I' 'src/dxgi' '-I' '../../dxvk/src/dxgi' \
'-I' '../../dxvk/./include' '-I' 'src/dxvk' '-I' '../../dxvk/src/dxvk' '-I' '.' \
'-pipe' '-D' '_FILE_OFFSET_BITS=64' '-Wall' \
'-Winvalid-pch' '-Wnon-virtual-dtor' '-std=c++17' '-O3' '-D' 'NOMINMAX' \
'-fwhole-program' \
'-fPIC' '-pthread' '-m64' '-Wno-attributes' '-march=native' \
'-O3' '-fgraphite-identity' '-floop-nest-optimize' \
'-MD' '-MQ' 'src/dxgi/src@dxgi@@dxgi.dll@sha/meson-generated_dxgi.dll-unity.cpp.o' \
'-MF' 'src/dxgi/src@dxgi@@dxgi.dll@sha/meson-generated_dxgi.dll-unity.cpp.o.d' \
'-v' '-isystem' '/usr/include/wine-vanilla-3.16/wine/windows' '-shared-libgcc'

Maybe there is some attributes I should change, before -fwhole-program use?

My variant uses compile and link steps, not sure if it can be changed for winelib build.
Because different jobs should be done by winegcc/winebuild itself, before something actually compiled/linked.

Is it emitting errors? I would need to see at least some of them to be able to guess what is wrong.

I want winelib builds too, although my winelib builds don’t work for me yet, so I need to resolve that before I even try.

using -v to explain all commands:

wineg++ -v  -o src/dxgi/dxgi.dll.so ../../dxvk/src/dxgi/dxgi.spec 'src/dxgi/src@dxgi@@dxgi.dll@sha/meson-generated_dxgi.dll-unity.cpp.o' -Wl,--no-undefined -Wl,--as-needed -Wl,-O1 -shared -fPIC -Wl,--start-group -Wl,-soname,dxgi.dll.so -lwinevulkan -Wl,--end-group -pthread -m64 -mwindows 
winebuild -v -fno-asynchronous-unwind-tables --cc-cmd=x86_64-pc-linux-gnu-gcc -m64 --ld-cmd=x86_64-pc-linux-gnu-ld -m64 -D_REENTRANT -fPIC --dll -o dxgi.dll-4y5sAd.spec.o -E ../../dxvk/src/dxgi/dxgi.spec -L/usr/lib64/wine-vanilla-3.16/wine -L/usr/lib64/wine-vanilla-3.16 -- src/dxgi/src@dxgi@@dxgi.dll@sha/meson-generated_dxgi.dll-unity.cpp.o /usr/lib64/wine-vanilla-3.16/wine/libwinevulkan.def /usr/lib64/wine-vanilla-3.16/wine/libshell32.def /usr/lib64/wine-vanilla-3.16/wine/libcomdlg32.def /usr/lib64/wine-vanilla-3.16/wine/libgdi32.def /usr/lib64/wine-vanilla-3.16/wine/libadvapi32.def /usr/lib64/wine-vanilla-3.16/wine/libuser32.def /usr/lib64/wine-vanilla-3.16/wine/libwinecrt0.a /usr/lib64/wine-vanilla-3.16/wine/libkernel32.def /usr/lib64/wine-vanilla-3.16/wine/libntdll.def 
x86_64-pc-linux-gnu-gcc -m64 -xassembler -c -m64 -o dxgi.dqLQXl.o dxgi.YIr7Yg.s
x86_64-pc-linux-gnu-ld -m elf_x86_64 -r -o dxgi.qyfoFq.o dxgi.dqLQXl.o src/dxgi/src@dxgi@@dxgi.dll@sha/meson-generated_dxgi.dll-unity.cpp.o /usr/lib64/wine-vanilla-3.16/wine/libwinecrt0.a
../../dxvk/src/dxgi/dxgi.spec:1: function 'CreateDXGIFactory' not defined
../../dxvk/src/dxgi/dxgi.spec:2: function 'CreateDXGIFactory1' not defined
../../dxvk/src/dxgi/dxgi.spec:3: function 'CreateDXGIFactory2' not defined
winegcc: winebuild failed

@pchome I see what is happening. You are not passing the spec file to the compiler, so it thinks that it is alright to make everything static. This is the reason why I expected annotations using the always_inline attribute to be necessary. You can try passing the spec file to the compiler and hope that it will realize that those symbols must be visible. In my case, I did both at once, so it “magically” understood and took care of things.

By the way, are the release binaries actually built with -march=native, -fgraphite-identity and -floop-nest-optimize or did you decide to include those? -march=native binaries cannot be redistributed without risking them not working for others. -fgraphite-identity and -floop-nest-optimize are part of graphite, which has open bugs where it generates broken code. I believe that is why they are not part of -O3.

By the way, I want winelib builds too

I would need to see some output too, to answer.

Have you tried my dxvk-9999 ebuilds?
and WINEPREFIX=/path/to/pfx dxvk-setup-9999

are the release binaries actually built with -march=native

I'm using same flags for testing as for my whole system, so I can compare with system dxvk.
And it's not for redistribution, it's my local builds.

@pchome I have not had a chance to try out your ebuilds yet, so no. As for output, they build fine, but crash when I try using them, so there is no build system output to show in terms of error messages. I spent the time that I had to look into that on making the proof of concept. I used more than I had in free time on the proof of concept, so I will need to look into winelib builds some other time.

As for your system cflags, that makes sense. Seeing them after your earlier comment about -O3 made me ask myself if I had missed some additional CFLAGS such that the proof of concept deviated from the official build’s flags in some non-trivial way other than -fwhole-program.

@pchome

Maybe there is some attributes I should change, before -fwhole-program use?

Yep, as addition to my patch

if get_option('whole_program')
  add_project_arguments('-fwhole-program', language : 'cpp')
  add_project_arguments('-fvisibility=hidden', language : 'cpp')
  add_project_arguments('-U__declspec_dllexport', language : 'cpp')
  add_project_arguments('-D__declspec_dllexport=__attribute__((visibility("default"))) __attribute__((used))', language : 'cpp')
endif

(stripped)

Still not tested if it really produces valid libraries, but at least it compiles.

whole-program.winelib.patch.txt

Copy/paste friendly example

cd /tmp
git clone https://github.com/doitsujin/dxvk dxvk-tst
cd dxvk-tst
wget https://github.com/doitsujin/dxvk/files/2412911/whole-program.patch.txt
patch -p1 < whole-program.patch.txt
mv src/util/sha1/sha1.c src/util/sha1/sha1.cpp

# winelib part
wget https://github.com/doitsujin/dxvk/files/2413478/whole-program.winelib.patch.txt
patch -p1 < whole-program.winelib.patch.txt

meson --cross-file build-wine64.txt \
  --buildtype "release" \
  --prefix "/tmp" \
  --unity on \
  --strip \
  /tmp/dxvk-build.64

cd /tmp/dxvk-build.64
ninja -j4 install
ls -l /tmp/lib64

I can't see any difference using my favourite "not-a-benchmark" glxgears d3d11-triangle.exe.

# export VSYNC=
#    0 - Force off
#    1 - Mailbox mode. Vsync with uncapped framerate.
#    2 - Traditional vsync with framerate capped to refresh rate.
#    3 - Adaptive vsync with tearing at low framerates.
export VSYNC=1

strangle d3d11-triangle.exe

~3000fps for all 3 variants (yes, compilation using whole-program hack produces valid libraries)

@pchome That is not quite the news I had hoped to hear, but it is the news that I thought would be alright. Even if performance remains the same (because we are heavily dependent on the driver), smaller binaries are useful. It also means that we do not have the issue of the old unity builds where performance reportedly dropped.

We also could try a few more techniques. Whole program optimization should work well in conjunction with PGO. It could turn out that all of the low dangling fruit here has been picked, but it would be nice to confirm that. Getting smaller binaries out of this investigation means that we still win. :)

Even if performance remains the same

That is for simple test, someone with powerful GPU should test this using e.g. Unigine Superposition test or an real game, to be able observe small fps difference. I can't definitely detect changes with 15-20fps.

That makes Unigine Superposition sound like a good benchmark to use as part of building a PGO build. ;)

I am happy that -fwhole-program seems to prevent this problem with unity builds:

We already had Unity builds at some point, but for some strange reason they ended up being significantly slower than regular builds.

Also, whole program optimization makes implementing a PGO proof of concept really easy. It should be adapting what this says to the whole program optimization proof of concept:

https://gist.github.com/daniel-j-h/c4b109bff0b717fc9b24

That makes Unigine Superposition sound like a good benchmark to use as part of building a PGO build.

I'm not sure.
Anyway, it's a bad benchmark to test CPU optimizations. As well as d3d11-triangle.exe, they are both GPU bound for me and shows no noticeable difference w/ or w/o dxvk optimization.

Here are some numbers from Unigine Superposition test on my side:

• all builds uses --unity on meson option
• glitches - sometimes stage stuttering for a moment and fps showing false higher numbers

We need an CPU bound test to properly check such optimizations. Also I believe there is no much places where DXVK can be optimized using CPU optimizations.

The instrumentation should be collecting frequency statistics on branching, so anything that exercises the code decently should be alright for generating a profile. Firefox’s own boost from PGO ran through some things that exercised the code and then received a boost in JavaScript performance in general. I am hopeful that we can just pick something that is easy to run and get something out that experiences a boost whenever CPU bound. There is more to it than that, but most real world runs are probably going to give better information than GCC’s guesswork does with no data at all.

I believe there is no much places where DXVK can be optimized using CPU optimizations.

If it turns out that all of the low dangling fruit has been picked here, then that is okay. I just wanted to see if that was the case after seeing how compiler options had influenced FPS in the past. This is real time software, so it is more sensitive to unnecessary overhead.

Do you have profiling data to show that there is not much room for improvement? Perhaps a flame graph?

http://www.brendangregg.com/flamegraphs.html

That said, we did learn that we can get smaller binaries from WPO, which is nice.

Do you have profiling data to show that there is not much room for improvement? Perhaps a flame graph?

No, I need to reconfigure some parts of my system for this tools to work.
And I have no plans yet to do this.

It will take a while for me to do complete testing to understand the impact of various options, but for what it is worth, a quick read of disassembly of generated binaries built with -O3 -fwhole-program -fweb -march=ivybridge -mtune=ivybridge show fairly extensive use of AVX instructions. Here is an sample of a couple of functions to show what I mean:

https://paste.pound-python.org/show/9tUvJeDO22SGUwKlyJ9p/

This is from a version that I built for myself. I plan to do testing of just -O3 -fwhole-program vs -O3 built the normal way to try isolate the benefits of -fwhole-program. Also, I still need to make a build that I can profile so that I can find the “hot” functions to know which functions’ disassembly I should compare to see if there is any difference in the quality of the assembly being emitted. If the hot functions’ assembly does not improve, then the compiler flag changes would have very little impact beyond making binaries smaller.

I've tried to use PGO, but with no success. Maybe WINE should be built with debug symbols and -fprofile-generate.
Also tried to use AutoFDO, and finally I can at least reproduce optimization benefit on sample program.

Since all tools Intel-specific, I faced a problem: -b, :pp and even BR_INST_RETIRED:TAKEN event not supported on my system :) Found an alternative in BKDG (EventSelect 0C4h Retired Taken Branch Instructions), and used this as -e cpu/event=0xc4,umask=0x0,name=br_inst_retired_taken/, but finally ended up with just -e branch-instructions.

How it work for me:

• # echo "-1" > /proc/sys/kernel/perf_event_paranoid
  I want to use perf and tools under regular user

• $ gcc -Ofast -g3 sort.c -o sort
  main program can be optimized, but should contain debug info (-g or -g1 levels not enough for my system)

• $ perf record -e branch-instructions ./sort
  collect perf.data

• $ create_gcov --binary=./sort --profile=perf.data --gcov=sort.gcov -gcov_version=1 -use_lbr=false
  yep, no lbr :angry:

• $ dump_gcov -gcov_version=1 sort.gcov
  for check if profile generated correctly, do not required:

bubble_sort total:17407 head:0
  2: 0
  3: 0
  4: 9780
  4.1: 0
  5: 243
  7: 0
  8: 2
  9: 5039
  13: 2343
sort_array total:5171 head:0
  0: 455
  3: 209
  3.3: 2164
  4.3: 256
  7: 0
  1: printf total:2085
    2: 2085
  6: bubble_sort total:2
    2: 0
    4: 0
    5: 2
    7: 0
    8: 0
    9: 0

• $ gcc -O3 -fauto-profile=sort.gcov sort.c -o sort_autofdo

$ ./sort
Bubble sorting array of 30000 elements
1925 ms

$ ./sort_autofdo 
Bubble sorting array of 30000 elements
1689 ms

Going to try this on DXVK later, I hope it will work.

As a minor correction, -fweb was redundant because it is turned on by default. There has been a time nearly 10 years ago where I recall that had not been the case, but today, it is on by default, so there is no point in specifying it.

It is possible to look at what optimization passes GCC is enabling by using gcc -O3 -Q --help=optimizers. Turning them all on is actually not a good idea (because not all are beneficial or even meant for making things quicker), but examining things can give some candidates for additional optimizations. In specific, the following look interesting for reasons of smaller code / better performance:

-fdelete-dead-exceptions
-fgcse-las
-fgcse-sm
-fipa-pta
-fmodulo-sched
-fmodulo-sched-allow-regmoves

Linaro explicitly mentions using -fmodulo-sched-allow-regmoves to speed up Android code:

https://www.linaro.org/blog/compiler-flags-used-to-speed-up-linaro-android-2011-10-and-future-optimizations/

It is also possible to look at GCC's tunables with gcc -Q --help=params. Some are a trade-off between compilation time and the quality of the generated code. In particular, the documentation states all of these could increase compile time:

max-crossjump-edges
max-delay-slot-insn-search
max-delay-slot-live-search
max-modulo-backtrack-attempts
max-tail-merge-iterations
dse-max-object-size
max-reload-search-insns
max-cselib-memory-locations
max-sched-ready-insns
max-tail-merge-iterations
loop-invariant-max-bbs-in-loop

https://gcc.gnu.org/onlinedocs/gcc-8.2.0/gcc/Optimize-Options.html#Optimize-Options

Consequently, all of them are possible candidates for improving the quality of generated code. We would need to look at the assembly output after changing these to see if they can make any difference.

We need a way of measuring whether any of this makes a difference beyond measuring binary sizes and doing disassembly. Perhaps we can profile time spent in DXVK's code to see how much time is spent relative to everything else, although that will require some thought. I plan to try testing before/after, but that won't necessarily identify improvements that reduce CPU time (and help in CPU bounded situations) unless I happen to find something that is CPU bound on my machine to test.

Here are the updated binary sizes on 33408a8a74f3eb3af9d971d857503d9c322433b2 built with GCC 8.2.0 without -fwhole-program and without unity:

richard@desktop ~ $ ls -l  /tmp/dxvk-win64/bin/{d3d11.dll,dxgi.dll}
-rwxr-xr-x 1 richard richard 2014208 Sep 28 12:57 /tmp/dxvk-win64/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1600512 Sep 28 12:57 /tmp/dxvk-win64/bin/dxgi.dll
richard@desktop ~ $ ls -l  /tmp/dxvk-win32/bin/{d3d11.dll,dxgi.dll}
-rwxr-xr-x 1 richard richard 2389504 Sep 28 12:57 /tmp/dxvk-win32/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1882624 Sep 28 12:57 /tmp/dxvk-win32/bin/dxgi.dll

The same, except with unity turned on:

richard@desktop ~ $ ls -l  /tmp/dxvk-win64/bin/{d3d11.dll,dxgi.dll}
-rwxr-xr-x 1 richard richard 2807296 Sep 28 12:53 /tmp/dxvk-win64/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 2128384 Sep 28 12:53 /tmp/dxvk-win64/bin/dxgi.dll
richard@desktop ~ $ ls -l  /tmp/dxvk-win32/bin/{d3d11.dll,dxgi.dll}
-rwxr-xr-x 1 richard richard 2481152 Sep 28 12:52 /tmp/dxvk-win32/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1904640 Sep 28 12:52 /tmp/dxvk-win32/bin/dxgi.dll

And unity + -fwhole-program + the hack to make them truly use 1 file per dll:

richard@desktop ~ $ ls -l /tmp/dxvk-win64-whole-program/bin/d3d11.dll /tmp/dxvk-win64-whole-program/bin/dxgi.dll
-rwxr-xr-x 1 richard richard 1666048 Sep 28 13:07 /tmp/dxvk-win64-whole-program/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1406464 Sep 28 13:07 /tmp/dxvk-win64-whole-program/bin/dxgi.dll
richard@desktop ~ $ ls -l  /tmp/dxvk-win32-whole-program/bin/d3d11.dll /tmp/dxvk-win32-whole-program/bin/dxgi.dll 
-rwxr-xr-x 1 richard richard 1760768 Sep 28 12:49 /tmp/dxvk-win32-whole-program/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1445888 Sep 28 12:49 /tmp/dxvk-win32-whole-program/bin/dxgi.dll

It is interesting how the unity builds generate bigger binaries, but turning on -fwhole-program suddenly makes them smaller.

There was a mistake in my earlier instructions. It seems that I had left out -fwhole-program on d3d11.dll. This made d3d11.dll unnecessarily large. Here are updated instructions for 64-bit:

meson --unity on --strip --cross-file build-win64.txt --prefix /tmp/dxvk-win64-whole-program build.w64
cd build.w64
meson configure -Dbuildtype=release
ninja

cat << END > dxgi.dll.c
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/dxgi/src@dxgi@@dxgi@sha/dxgi-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

x86_64-w64-mingw32-g++ -fwhole-program -std=c++1z -O3 -o src/dxgi/dxgi.dll ../src/dxgi/dxgi.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/dxgi/dxgi.def -Wl,--start-group -Wl,--out-implib=src/dxgi/libdxgi.dll.a dxgi.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w64/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w64/src/dxgi/src@dxgi@@dxgi@sha/ ../lib/vulkan-1.lib  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

cat << END > d3d11.dll.c
#include "src/d3d11/src@d3d11@@d3d11@sha/d3d11-unity.cpp"
#include "src/dxbc/src@dxbc@@dxbc@sta/dxbc-unity.cpp"
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

x86_64-w64-mingw32-g++ -fwhole-program -std=c++1z -O3 -o src/d3d11/d3d11.dll ../src/d3d11/d3d11.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/d3d11/d3d11.def -Wl,--start-group -Wl,--out-implib=src/d3d11/libd3d11.dll.a d3d11.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w64/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w64/src/dxgi/src@dxgi@@dxgi@sha/ ../lib/vulkan-1.lib -ldxgi /home/richard/devel/dxvk/lib32/vulkan-1.lib -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

And for 32-bit:

meson --unity on --strip --cross-file build-win32.txt --prefix /tmp/dxvk-win32-whole-program build.w32
cd build.w32
meson configure -Dbuildtype=release
ninja

cat << END > dxgi.dll.c
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/dxgi/src@dxgi@@dxgi@sha/dxgi-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

i686-w64-mingw32-g++ -fwhole-program -std=c++1z -O3  -o src/dxgi/dxgi.dll ../src/dxgi/dxgi.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/dxgi/dxgi.def -Wl,--start-group -Wl,--out-implib=src/dxgi/libdxgi.dll.a dxgi.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w32/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w32/src/dxgi/src@dxgi@@dxgi@sha/ ../lib32/vulkan-1.lib  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

cat << END > d3d11.dll.c
#include "src/d3d11/src@d3d11@@d3d11@sha/d3d11-unity.cpp"
#include "src/dxbc/src@dxbc@@dxbc@sta/dxbc-unity.cpp"
#include "src/dxvk/src@dxvk@@dxvk@sta/dxvk-unity.cpp"
#include "src/util/src@util@@util@sta/util-unity.cpp"
#include "src/spirv/src@spirv@@spirv@sta/spirv-unity.cpp"
#include "../src/util/sha1/sha1.c"
END

i686-w64-mingw32-g++ -fwhole-program -std=c++1z -O3 -o src/d3d11/d3d11.dll ../src/d3d11/d3d11.def -Wl,--no-undefined -Wl,--as-needed -shared ../src/d3d11/d3d11.def -Wl,--start-group -Wl,--out-implib=src/d3d11/libd3d11.dll.a d3d11.dll.c -I ../include -I ../src/dxvk -I ../src/dxgi -I . -I ../build.w32/src/dxvk/src@dxvk@@dxvk@sta/ -I ../build.w32/src/dxgi/src@dxgi@@dxgi@sha/ ../lib32/vulkan-1.lib -ldxgi /home/richard/devel/dxvk/lib32/vulkan-1.lib -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 -Wl,--end-group -static -static-libgcc -static-libstdc++ -Wl,--add-stdcall-alias,--enable-stdcall-fixup

ninja install

Out of curiosity, I tried adding -fdelete-dead-exceptions -fgcse-las -fgcse-sm -fipa-pta -fmodulo-sched -fmodulo-sched-allow-regmoves to the -fwhole-program build:

richard@desktop ~ $ ls -l /tmp/dxvk-win64-whole-program/bin/d3d11.dll /tmp/dxvk-win64-whole-program/bin/dxgi.dll
-rwxr-xr-x 1 richard richard 1666048 Sep 28 13:15 /tmp/dxvk-win64-whole-program/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1406464 Sep 28 13:15 /tmp/dxvk-win64-whole-program/bin/dxgi.dll
richard@desktop ~ $ ls -l /tmp/dxvk-win32-whole-program/bin/d3d11.dll /tmp/dxvk-win32-whole-program/bin/dxgi.dll
-rwxr-xr-x 1 richard richard 2094592 Sep 28 13:05 /tmp/dxvk-win32-whole-program/bin/d3d11.dll
-rwxr-xr-x 1 richard richard 1640448 Sep 28 13:05 /tmp/dxvk-win32-whole-program/bin/dxgi.dll

32-bit significantly increased (perhaps from fewer registers?) while 64-bit remained became the slightest bit smaller.

One last thing for a while. It occurs to me that the 64-bit binaries have an advantage over the 32-bit binaries because MMX, SSE and SSE2 are guaranteed to be available on 64-bit. Anything that does not have those would have to be older than a Pentium 4. It seems unlikely that anyone would run DXVK on anything older than P4, so it might be a good idea to tell g++ to use those instructions on 32-bit.

The first of hopefully more commits to come from this investigation was just merged to master:

https://github.com/doitsujin/dxvk/commit/6fb09cb9fceaf9890ec83ded507ddae22c3b6a40

It is interesting how the unity builds generate bigger binaries

more aggressive optimizations

32-bit significantly increased (perhaps from fewer registers?) while 64-bit remained became the slightest bit smaller.

different code generated for __WIN32__ and __WIN64__ directives

@pchome I suspect that the optimizations are not aggressive enough without -fwhole-program, which shrinks it nicely. Also, the difference in code size for 32-bit could be in part because SSE and SSE2 were not being used. That is fixed now, but I have yet to test to see if it closes the gap between 32-bit and 64-bit code sizes.

That is not to say that making them the same size is important. However, seeing 32-bit binaries be significantly larger than 64-bit binaries is strange because usually they are a little smaller.

because SSE and SSE2 were not being used

$ gcc -m32 -O3 -Q --target-help | grep -i '\-msse'
$ gcc -march=native -m32 -O3 -Q --target-help | grep -i '\-msse'

They seems always used for me.

For comparison (where they disabled):
$ gcc -march=pentium -m32 -O3 -Q --target-help | grep -i '\-msse'

@pchome Not for the official DXVK release builds or for the DXVK bundled with Proton. The same goes for the builds for which I posted size numbers. SSE and SSE2 were omitted from the 32-bit binaries.

I've smoothed rough corners in my whole-program patch a bit, and going to include it into my ebuild repository using experimental USE flag. Still rough hack, but it's OK for compiler. Tested only on winelib builds.

For those wanted to test: dxvk-0.80-whole-program-support2.patch.txt

Use $ meson configure -Dwhole_program=true in your build directory, or path -Dwhole_program=true along with --unity on option on configuration stage.
No other actions required (-fwhole-program will be added to project flags).

Note:

• patch was created using patched sources, so I bundled them too, and it can be applied to current DXVK sources w/o conflicts
  • it contains patch from https://github.com/doitsujin/dxvk/pull/604#issuecomment-416283000 , to fix tests for winelib
  • it contains patch #618 was based on, to not install anything into bin/ by default
• @doitsujin , please check d3d10 part, maybe you'll want something like this to be upstreamed

@pchome If you designed your patch to not change behavior unless told to use wpo, you can just apply it unconditionally. Conditional patching is highly discouraged in Gentoo ebuilds because it makes them less maintainable. You could also call the USE flag wpo.

I built DXVK using AutoFDO (https://github.com/doitsujin/dxvk/issues/646#issuecomment-425420081).
But test program fails w/ this libs.

Also there is new warning during build: [-Wmaybe-uninitialized] for src/dxbc/dxbc_compiler.cpp:5034:31
And using unity build along with -fwhole-program or -g3 during optimized compilation causes ICEs (during IPA pass: cp -- looks like -fipa-cp).

Not sure is it me doing something wrong, or compiler/code issue.

# AutoFDO
so_path=/tmp/lib64
data_path=/var/tmp/dxvk.profile

# Record profile data
perf record -e branch-instructions -o $data_path/dxvk.data  d3d11-triangle.exe

# Create afdo profile
create_gcov --binary=$so_path/d3d11.dll.so --profile=$data_path/dxvk.data --gcov=$data_path/d3d11.dll.so.afdo -gcov_version=1 -use_lbr=false
create_gcov --binary=$so_path/dxgi.dll.so --profile=$data_path/dxvk.data --gcov=$data_path/dxgi.dll.so.afdo -gcov_version=1 -use_lbr=false

profile_merger $data_path/d3d11.dll.so.afdo $data_path/dxgi.dll.so.afdo -output_file=$data_path/dxvk.afdo -gcov_version=1 -use_lbr=false

# Use -fauto-profile=/var/tmp/dxvk.profile/dxvk.afdo

whole-program patch
going to include it into my ebuild repository

https://github.com/pchome/dxvk-gentoo-overlay/commit/476a1439cd63e4905d734a9353d3ed3988687731

Can anyone share perf.data generated on Intel system for DXVK?

Using something like:
perf record -b -e br_inst_retired.near_taken:pp -- wine d3d11-app.exe (I suppose)

or using pmu-tools and example (create_gcov and profile_* : google/autofdo)

@ryao

Do you have profiling data to show that there is not much room for improvement? Perhaps a flame graph?

perf-dxvk.svg.gz
From profile data collected for AFDO, not perf -ag.

@pchome What is AFDO? Also, that shows a significant amount of time is spent in __clock_gettime(). This can be optimized, although not by the compiler. I can look into it later this week.

@pchome I imagine that you are using your own wine build. Try applying this patch:

https://github.com/ValveSoftware/wine/commit/781fbb086ed84cc46923eab5930edddf156b3fbc

It might make that faster. We would need to try to profile the native build in wine to see what it does, but hopefully, it does the same thing.

@ryao

What is AFDO?

AutoFDO, see above https://github.com/doitsujin/dxvk/issues/646#issuecomment-425667821

I imagine that you are using your own wine build. Try applying this patch:

It's wine-staging-3.16[custom-cflags] (mean system *FLAGS, except LTO), patched w/ rebased esync+proton patches. So it's "custom Proton-3.16" build, already contains this patch.

@pchome In that case, we need to consider reimplementing how time is looked up, perhaps by using the rdtsc instruction while implementing a fallback for processors where using it is a bad idea.

Also, I spotted a function that seems to take a little too long relative to the code, so I am going to disassemble it to see what it is doing. Maybe we can do better there with better optimization flags, maybe we could do better there by changing the code or maybe that code really does more than I realize. C++ is abstract enough that it is hard to do a ballpark estimate of how expensive code is just by eyeballing a few lines.

Lastly, a program that draws triangles really is not the best choice to profile. It would be better to profile a unigine benchmark or overwatch. Also, I hope that you are using a sampling rate of 99Hz or 997Hz. Round number sample rates of 100Hz or 1000Hz can end up failing to spot expensive things that run at the same sample rate, which can happen.

On second thought, there is a better way of doing time in this sort of application. instead of polling time, we could make this interrupt driven by having a volatile flag variable that controls whether we flush that we set on a timer. Then the flush code would check that flag. When it is set, it would unset it and do a flush. This should be less expensive than polling in this function. Maybe this could work for that:

http://www.cplusplus.com/reference/future/future/

I would need to study the code, but perhaps we could make the flush itself timer driven so that we do not need that check and can just rely on the timer.

rdtsc

Note: profile may contain calls from perf itself, and rdtsc very likely one of them.
Edit: Or, probably it called from DXVK_HUD, which is no subject for optimization.

Also, I want to profile at least anything small, to prove profiling can be done on wine applications (*.exe/*.dll).

Also, I hope that you are using a sampling rate of 99Hz or 997Hz.

Yes, I planned to use -F99 on something "big", but I'll try it on triangles, thanks.

Also, I believe my hardware is not fully suitable for profiling multithreaded applications (no -b, :pp, LBR, PEBS, ... support), at least this explains why profiling easily reproducible on ./sort example and constantly failing on DXVK.
But if someone can generate such profile on Intel system -- it can be reused on non-Intels (I suppose).

Meanwhile I just built DXVK using AutoFDO, but w/o any optimizations:
-O0 and -fwhole-program/"unity" build disabled. Other than -O0 optimization levels produce failing *.sos.

Edit: I see no reason to measure it's performance on this stage.

@pchome Would you try out this patch and see if it makes a difference in your flame graph?

https://github.com/doitsujin/dxvk/commit/b9b86b432792634cd14ab6dbc3d12c23e26be7c7

1. Why don't you generate profile data on your side? (It was easier than I thought, to setup and configure all required parts.)
2. https://github.com/doitsujin/dxvk/blob/master/src/util/thread.h#L14

@pchome

1. I have not found time to fix my local winlib builds. I am spending more time on this than I should as it is, so fixing that is going to wait for while.
2. I fixed this to use the dxvk::thread class. I did a 64-bit winelib build test, but as you know, I can't do winelib runtime tests at the moment. If you are willing to try it out, here is the revised version:

https://github.com/doitsujin/dxvk/commit/35ca9bc0fc222fabe9a86686a5909ce278ba1eb2

I thought about it some more. I am not certain that a timer is even necessary. If the GPU is about to run out of things to do, then giving it more seems like a good idea. I don't see the point of having a timeout. We could try deleting this code and seeing how that works.

I did a 64-bit winelib build test, but as you know, I can't do winelib runtime tests at the moment.

Why?
Once you built it:

#!/bin/sh
mkdir -p $(pwd)/dxvk-test

export WINEPREFIX="$(pwd)/dxvk-test"
export WINEDLLOVERRIDES="d3dcompiler_47,d3d11,dxgi=n"
export DXVK_HUD=version,devinfo,fps

cp /tmp/lib64/dxgi.dll.so  dxvk-test/drive_c/windows/system32/dxgi.dll
cp /tmp/lib64/d3d11.dll.so  dxvk-test/drive_c/windows/system32/d3d11.dll

wine d3d11-triangle.exe

That's all.

@pchome I am using proton. I don't know why it does not work. It is on my todo list. I doubt that I will have time to do anything else here until later in the week.

Last time I tried, the winelib builds didn't work on Proton because Proton's Wine version was not recent enough and winelib DXVK couldn't use native Vulkan calls.

You can have different WINE version for tests and measures. What the problem? I don't get it.
I have four of them at the same time.

If problem somewhere in your distro, then

• You can grab winegcc, winebuild, etc. from any wine-3.15+ .deb, .rpm ...
  And put them into your $PATH using different names, or use full path in build-wine32.txt
• You can grab whole WINE from PlayOnLinux site (I hope they fixed vulkan in their builds), or same .deb, .rpm ...
  And use script

#!/bin/sh

tst_wine="/var/tmp/wine-3.17"

WINEPREFIX="/path/to/pfx"

WINE="${tst_wine}/bin/wine"
WINESERVER="${tst_wine}/bin/wineserver"
WINELOADER="${tst_wine}/bin/wine"
WINEDLLPATH="${tst_wine}/lib64/wine"

"${WINE}" d3d11-triangle.exe

I am juggling several different things. Some things just get assigned a lower priority than others. I did a prototype of that patch a few days early, so do we really need to discuss how I have yet to allocate more time to try again with winelib?

do we really need to discuss how I have yet to allocate more time to try again with winelib?

You spending more time randomly sticking a code you can't even test.

You think it's ok to ask people to spend their time for testing random changes.
No one asking you to setup winelib build, profile your MinGW build -- I don't see any difference.

Since this issue is not my blog, I'm reporting my experience and expect participants can reproduce it.
So yes, if you have something not working to "Explore advanced toolchain optimizations (e.g. PGO, LTO, whole program optimization, etcetera)" -- I'll try to help.

My interest here is AutoFDO, PGO, LTO and -fwhole-program, but not DXVK optimization process.
So if I can't get feedbacks here, then I'll find better place to continue my investigations.

The previous code had a call to an external symbol. The compiler cannot optimize across function calls to external symbols. The patched code is therefore easier for the compiler to optimize.

Before I put this down, I made another version of the patch that does away with the added atomics, which makes the code even more compiler optimization friendly. I’ll push that version when I am back at my Gentoo development workstation.

For what it is worth, doing away with atomics also improves compiler optimization opportunities, but it can be absurdly difficult to do right. The amount of code that you see changed to do that is disproportionally less than the amount of effort.

During the course of this, we will probably find other source code changes that can make the compiler’s optimization passes do a better job. We really ought to make them in addition to experimenting with compiler optimizations.

Also, I tested all versions that I pushed, but only as native binaries because I am not setup for doing tests with winelib builds yet. Just as you are not setup to build native binaries and can only test winelib builds, I am not setup for winelib builds (at least doing usable ones) and I can only test the native binaries. It would take time to setup something else. I estimate it would take several hours to debug why my winelib builds do not work and I don’t have time for that. It is on my todo list. It will be done eventually.

It might only look like a few minutes to you (and perhaps it is), but I have been sucked into rabbit holes that should take only a few minutes in the past. Writing the patch was a small rabbit hole. I am not willing to jump into another one without adequate time to dedicate to it.

Lastly, your triangles test is a horrible way of profiling for optimization opportunities, whether they are opportunities for better compilation or something else. Getting the time does not involve a system call thanks to vdso (and possibly also a patch done to proton to stop using CLOCK_MONOTONIC_RAW). There is very little that we can learn from profiling it because it does not exercise the code in a realistic way. Proper profiling requires running something that exercises the graphics API in a realistic way, such as a unigine benchmark.

I had a chat with one of the LLVM experts from BNL in person yesterday about this. The only thing he could add that we were not already doing was to try building with Clang. I had hoped that he could suggest some compiler options known to generate better code at the expense of absurdly long compilation times, but he is not familiar with those.

I heard about suoeroptimization from a researcher from the Institute for Advanced Computational Science that does work in compiler optimization. With super optimization, builds can take days, weeks or even months, but the performance of the generated code is better than from normal optimization. Someone might want to look into testing every tool that comes up when googling “gcc superoptimizer” or related queries.

I am withholding names because I do not think the people involved would want to receive emails from random people on this topic. The researcher that I mentioned receives so many emails that she only responds to emails with specific keywords in the subject, so random people who email her are unlikely to receive a response.

I have come to the conclusion that all of the low dangling fruit has been found. Further improvements will come from making code changes according to profiling.

Also, I tested that patch. It makes no improvement in any game, even in the fairly unrealistic Rise of Nations: Extended Edition title screen.