TIMESTAMP_DISJOINT queries are kind of supported in that DXVK reports the timer frequency to the application and reports the timers as not disjoint. Can you point me to the code that you think is not working correctly?
all query related code seems to be in perftest/perftest/directx.cpp
(https://github.com/sebbbi/perftest/blob/master/perftest/directx.cpp)
(building the project is very easy (has no deps.) in Visual Studio)
can help if I attach prebuilt binary here?
// Queries
for (auto &&q : queries)
{
D3D11_QUERY_DESC desc;
ZeroMemory(&desc, sizeof(desc));
desc.Query = D3D11_QUERY_TIMESTAMP_DISJOINT;
device->CreateQuery(&desc, &q.disjoint);
desc.Query = D3D11_QUERY_TIMESTAMP;
device->CreateQuery(&desc, &q.start);
device->CreateQuery(&desc, &q.end);
}
QueryHandle DirectXDevice::startPerformanceQuery(const std::string& name)
{
PerformanceQuery& query = queries[queryCounter % queries.size()];
query.name = name;
deviceContext->Begin(query.disjoint);
deviceContext->End(query.start); // NOTE: timestamp queries don't use Begin(), only End()
QueryHandle out {queryCounter};
queryCounter++;
return out;
}
void DirectXDevice::endPerformanceQuery(QueryHandle queryHandle)
{
PerformanceQuery& query = queries[queryHandle.queryIndex % queries.size()];
deviceContext->End(query.end); // NOTE: timestamp queries don't use Begin(), only End()
deviceContext->End(query.disjoint);
}
void DirectXDevice::processPerformanceResults(const std::function<void(float, std::string&)>& functor)
{
while(true)
{
PerformanceQuery& query = queries[queryProcessCounter % queries.size()];
D3D11_QUERY_DATA_TIMESTAMP_DISJOINT disjoint;
bool succDisjoint = deviceContext->GetData(query.disjoint, &disjoint, sizeof(disjoint), 0) == S_OK;
UINT64 start = 0;
UINT64 end = 0;
bool succStart = deviceContext->GetData(query.start, &start, sizeof(start), 0) == S_OK;
bool succEnd = deviceContext->GetData(query.end, &end, sizeof(end), 0) == S_OK;
// Wait until all queries are ready
if (!succDisjoint || !succStart || !succEnd)
break;
if (!disjoint.Disjoint)
{
UINT64 d = end - start;
float delta = (float(d) / float(disjoint.Frequency)) * 1000.0f;
// Call functor to process results
functor(delta, query.name);
}
queryProcessCounter++;
}
}
can help if I attach prebuilt binary here?
Please do, since I currently do not have a working Visual Studio setup.
I think I have a working build using meson, winegcc and a bunch of hacks.
Load R8 invariant: 6.066ms
Load R8 linear: 6.800ms
Load R8 random: 7.204ms
Load RG8 invariant: 6.156ms
Load RG8 linear: 6.802ms
Load RG8 random: 7.191ms
Load RGBA8 invariant: 6.076ms
Load RGBA8 linear: 6.888ms
Load RGBA8 random: 7.186ms
Load R16f invariant: 6.066ms
Load R16f linear: 6.891ms
Load R16f random: 7.109ms
Load RG16f invariant: 6.156ms
Load RG16f linear: 6.894ms
Load RG16f random: 7.100ms
Load RGBA16f invariant: 6.171ms
Load RGBA16f linear: 6.800ms
Load RGBA16f random: 7.191ms
Load R32f invariant: 6.156ms
Load R32f linear: 6.801ms
Load R32f random: 7.201ms
Load RG32f invariant: 6.064ms
Load RG32f linear: 6.888ms
Load RG32f random: 7.188ms
Load RGBA32f invariant: 11.164ms
Load RGBA32f linear: 11.264ms
Load RGBA32f random: 11.144ms
Load1 raw32 invariant: 16.737ms
Load1 raw32 linear: 15.747ms
Load1 raw32 random: 16.193ms
Load2 raw32 invariant: 17.373ms
Load2 raw32 linear: 16.874ms
Load2 raw32 random: 17.320ms
Load3 raw32 invariant: 18.477ms
Load3 raw32 linear: 18.303ms
Load3 raw32 random: 18.284ms
Load4 raw32 invariant: 22.178ms
Load4 raw32 linear: 22.387ms
Load4 raw32 random: 22.571ms
Load2 raw32 unaligned invariant: 17.441ms
Load2 raw32 unaligned linear: 17.733ms
Load2 raw32 unaligned random: 17.072ms
Load4 raw32 unaligned invariant: 22.476ms
Load4 raw32 unaligned linear: 22.351ms
Load4 raw32 unaligned random: 22.231ms
Tex2D load R8 invariant: 5.823ms
Tex2D load R8 linear: 6.821ms
Tex2D load R8 random: 6.812ms
Tex2D load RG8 invariant: 5.822ms
Tex2D load RG8 linear: 6.824ms
Tex2D load RG8 random: 6.819ms
Tex2D load RGBA8 invariant: 6.781ms
Tex2D load RGBA8 linear: 7.460ms
Tex2D load RGBA8 random: 10.002ms
Tex2D load R16F invariant: 5.729ms
Tex2D load R16F linear: 6.915ms
Tex2D load R16F random: 6.719ms
Tex2D load RG16F invariant: 5.820ms
Tex2D load RG16F linear: 6.909ms
Tex2D load RG16F random: 9.913ms
Tex2D load RGBA16F invariant: 6.873ms
Tex2D load RGBA16F linear: 7.367ms
Tex2D load RGBA16F random: 11.543ms
Tex2D load R32F invariant: 5.817ms
Tex2D load R32F linear: 6.822ms
Tex2D load R32F random: 9.995ms
Tex2D load RG32F invariant: 5.732ms
Tex2D load RG32F linear: 6.914ms
Tex2D load RG32F random: 11.530ms
Tex2D load RGBA32F invariant: 11.140ms
Tex2D load RGBA32F linear: 11.263ms
Tex2D load RGBA32F random: 13.875ms
It's running in loop. Possible there is overhead with wined3d, and while I'm trying to wakeup my desktop (100% GPU load, no chances for even mouse).
@pchome yes it goes in a loop..
@doitsujin
glad to help, here it goes:
perftest.zip
Here is quick winelib variant: perftest-wine.tar.gz
Shaders compiled using modified DXVK hlsl-compiler, patches included.
How do I use this? It doesn't seem to do anything on my system:
PerfTest
To select adapter, use: PerfTest.exe [ADAPTER_INDEX]
Adapters found:
0: AMD RADV POLARIS10 (LLVM 6.0.1)
1: AMD Radeon (TM) RX 480 Graphics
Using adapter 0
: 0.000ms
: 0.000ms
: 0.000ms
: 0.000ms
: 0.000ms
: 0.000ms
Actually, is this the bug I'm supposed to fix? It seems to work with wined3d.
@oscarbg in general I'd prefer if you could focus on describing the actual problem rather than your assumption what might be causing it, since I'm not sure what exactly I'm supposed to see on my end.
Fixed as of 45f61cbae5b644a0daa75df0930f523bcf69c0e7.
@doitsujin sorry for not so informative problem description.. I will try to not make assumputions in my next bug reports.. and thanks for fixing!
@doitsujin so tested on a NV Titan V using latest 399.17 and results are almost the same, but found some perf. issue (up to 5x slower execution of shader Load R8 invariant) that I don't know if it's because of some (unavoidable) overhead of DXVK DXBC->SPIR-V opcodes translation of because NV VK driver for Volta might have some enhancement possible..
can share your thoughts?
Here is Nvidia running your code with Vulkan backend using DXVK..
seems almost equal but invariant improvements in Volta aren't present running under DXVK+Volta..
seems either DXVK or NV Vulkan drv has improvements to do in that regard:
DX:
Load R8 invariant: 0.098ms
Load RG8 invariant: 0.153ms
Load RGBA8 invariant: 0.301ms
Vulkan with DXVK:
Load R8 invariant: 0.511ms
Load RG8 invariant: 0.502ms
Load RGBA8 invariant: 0.497ms
full post report here:
https://github.com/sebbbi/perftest/issues/5#issuecomment-419755749
I don't think the invariant results on the D3D11 side are valid, or of any practical relevance. There's nothing wrong with how DXVK reads typed buffers.
Invariant results are definitely valid and of practical relevance too.
These tests were made originally to test how well AMD GCN compiler generates scalar code. AMD has a separate scalar unit with scalar register set. AMD compiler automatically tracks whether data inputs are invariant (uniform) and if so, the compiler emits scalar load instruction instead of 64-wide vector load instruction. Scalar loads have significantly higher throughput, lower latency and result is stored to SGPR instead of 64-wide VGPR (saving registers = higher occupancy). It appears that Nvidia might have introduced something similar to Volta/Turing. For some reason this optimization is not yet enabled in their low level compiler (with SPIR-V input). I would assume that future drivers will fix this.
This is relevant because scalar optimizations are very common in optimized AAA console game code. Nvidia definitely also wants to run code like this faster.
Well, looks like I misinterpreted the DXBC shader code. Looking at your HLSL sources, I understand what it's trying to do now.
That said, apart from Nvidia's Dx11 driver, nobody seems to implement such an optimization in practice. There's no difference between invariant and linear when running the test on my RX 480 on Windows. The only thing that the AMD driver does is partial loop unrolling, which doesn't happen on Vulkan.
AMD implements scalar load optimizations for all GCN generations (GCN1-GCN5), including all consoles. However typed loads can't be optimized, since AMDs scalar unit doesn't have type conversion hardware. Invariant (uniform address) load instructions from raw (ByteAddressBuffer) and structured buffers are optimized. This results in around 4x performance increase.
AMD also has another runtime optimization (no compiler intervention required) which is applied to all 1d loads (typed and raw). If addresses of 16 consecutive lanes are perfectly linear, they have fast coalesced load path. This is 4x faster issue rate. If you compare scalar 1d invariant load to linear 1d load, you see them performing roughly the same, because both of these optimizations bring around the same perf boost in this special case (raw 1d load). However the coalesced vector load still has much higher latency than scalar load (which is not visible in this particular throughput oriented test case).
You can see the scalar load performance improvement in the raw Load2/Load3/Load4 cases on AMD hardware.
I think the bigger issue in this DXVK test case is that raw buffer performance gets linearly worse when using wider load instructions (Load, Load2, Load3, Load4).
It seems to be a common practice to translate HLSL ByteAddressBuffer Load2/Load3/Load4 as a series of SPIR-V 1d load instructions: https://github.com/Microsoft/DirectXShaderCompiler/blob/master/docs/SPIR-V.rst#byteaddressbuffer
This approach relies on the low level IHV compiler to regroup these narrow loads to one wide load. Otherwise it will simply issue multiple narrow loads, causing linear drop of issue rate with ByteAddressBuffer wide loads.
DXVK currently implements raw and structured buffers using typed loads from an R32_UINT texel buffer view. This isn't optimal, and on AMD hardware I could probably remap these to storage buffers, but since there doesn't seem to be any alignment requirement for buffer SRVs and UAVs in D3D11, that's not going to work on Nvidia where minStorageBufferOffsetAlignment is 32 bytes, except for structured buffers where the stride is a multiple of 32 bytes, or on drivers where the maximum number of per-stage storage buffer descriptors is too low (ANV).
FWIW I very recently tested a different optimization for structured buffers, where it would use R32G32B32A32 or R32G32 views where possible, but it didn't result in any measurable performance gain in practice.
That said, i don't think this is worth optimizing. Games simply don't use raw/structured buffers outside compute shaders in practice, most games don't spend a lot of time running compute shaders either, and the only hardware that would benefit from it already runs DXVK much better than the competition anyway. Considering the amount of work required, the potential rewards are too low.
Nothing extracted yet.
Hi,
I just keep asking about "low hanging fruit" features I think you don't care at all but anyway let me keep asking..
this time about D3D11 time queries support..
reason is I was reading @sebbbi twitter about:
"Nvidia driver seems to generate awful code for DirectX ByteAddressBuffers: https://github.com/sebbbi/perftest . Never got an answer from Nvidia why, but I assume it’s because their wide raw load instructions (vec2, vec4) require alignment. At least in CUDA that’s true."
so I check with Nvidia Titan V:
https://github.com/sebbbi/perftest/issues/5
really wanted to see some microbenchmarks using this tool, but with Nvidia Vulkan driver so using DXVK..
but it returns 0.0ms..
seeing code I see using things like:
D3D11_QUERY_DATA_TIMESTAMP_DISJOINT
so I'm assuming this support is missing..
I don't even know if Vulkan has something similar to implement it easily in DXVK..
thanks anyway..