Now I've seen for myself how problematic DX9 > DXVK is in Lineage 2 when I tried to make an AO. There are so many pitfalls and compatibility issues. I'm thinking about trying DX9 > DX11 > DXVK, too, to see if it improves the inject.

Try with current master https://github.com/doitsujin/dxvk/actions/runs/24126385952
Improvements have been made in regards to stuttering with fixed function.

Try with current master https://github.com/doitsujin/dxvk/actions/runs/24126385952 Improvements have been made in regards to stuttering with fixed function.

Thank you. I have Lineage 2 with +- stable FPS. I wrote more about the difficulty of working with the wrapper and this game. I'm currently trying to make a DXVK fork with AMD CACAO for this game.

What issues are you having? And on what setup

What issues are you having? And on what setup

Thank you, I don't need anything from you for my task. I've implemented everything I need in the local fork. However, there are some points regarding any AO mods for DX9. Below is a brief summary from the agent and examples of vide code, but I hope you get the general idea.

DXVK D3D9 Backend Requirements for Ambient Occlusion Mods

Context: This document describes the general API requirements for integrating SSAO/AO effects (AMD CACAO, NVIDIA HBAO+, RTGI, MXAO, etc.) into D3D9 games through DXVK. These requirements are game-agnostic and apply to any compute-based post-processing AO effect.

1. Present Hook — Compute Pipeline Injection Point

Location: src/d3d9/d3d9_swapchain.cpp → D3D9SwapChainEx::Present()

Requirement: A mechanism to execute compute dispatch before the present blit.

Why needed: All SSAO/AO effects are post-processing compute pipelines that must execute before the final image is presented to the swapchain.

Proposed API:

// In d3d9_swapchain.h
using PresentHookFn = void(*)(DxvkContext* ctx,
                               uint32_t width,
                               uint32_t height,
                               Rc<DxvkImage> backBuffer);

void RegisterPresentHook(PresentHookFn hook);

// In d3d9_swapchain.cpp::Present()
if (m_presentHook) {
    m_presentHook(ctx,
                  swapImage->info().extent.width,
                  swapImage->info().extent.height,
                  backBuffer);
}

Implementation notes:

• The hook must execute before the blitter present call
• The hook receives a valid DxvkContext* for emitting compute commands
• Multiple hooks should be supported (chained execution)
• The hook must complete before vkQueuePresentKHR is called

2. Depth Buffer Access — Sampled Image Support

Location: src/d3d9/d3d9_common_texture.cpp → depth-stencil surface creation

Requirement: Ability to sample the depth buffer from compute shaders.

Why needed: All SSAO/AO algorithms require reading the depth buffer to compute occlusion factors.

Current limitation: Depth-stencil surfaces are created with only VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT. Compute shaders cannot sample them.

Required change:

// In d3d9_common_texture.cpp
// Always add SAMPLED_BIT for depth-stencil surfaces:
if (isDS) {
    imageInfo.usage |= VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT
                    |  VK_IMAGE_USAGE_SAMPLED_BIT; // <-- for AO sampling
}

Additional requirements:

• Correct format mapping: D3DFMT_D24S8 → VK_FORMAT_D32_SFLOAT_S8_UINT with sampled support
• API to get VkImage handle from IDirect3DSurface9 (depth-stencil surface)
• Guarantee sufficient depth precision (D16 is inadequate for SSAO; prefer D24 or D32)

Proposed API:

// In d3d9_surface.h
VkImage GetDepthBufferImage() const;
VkFormat GetDepthBufferFormat() const;
uint32_t GetDepthBufferSampleCount() const;

3. Projection Matrix Access — Camera Matrix Retrieval

Location: src/d3d9/d3d9_device.cpp → transform state and constant buffer handling

Requirement: Ability to obtain the exact projection matrix for unprojecting depth values to view space.

Why needed: Without an accurate projection matrix, it is impossible to convert depth buffer values to view-space positions, which is required for any screen-space AO technique.

Critical issue: Many D3D9 games (UE2, UE3, Source Engine, etc.) use vertex shaders where the projection matrix is stored in vertex shader constant buffers, NOT in SetTransform(D3DTS_PROJECTION).

Required changes:

// In d3d9_device.cpp
// Intercept VS constants to detect projection matrix pattern
HRESULT SetVertexShaderConstantF(UINT StartRegister,
                                  const float* pConstantData,
                                  UINT Vector4fCount) {
    // Search for projection matrix pattern in constants
    if (IsProjectionMatrixPattern(pConstantData, Vector4fCount)) {
        m_cachedProjMatrix = ExtractProjectionMatrix(pConstantData);
        m_hasValidProj = true;
    }
    return ...;
}

Projection matrix detection heuristics:

• 4x4 matrix with specific structure (perspective projection):
  • m[0][0] = 1/(aspect * tan(fov/2))
  • m[1][1] = 1/tan(fov/2)
  • m[2][2] = far/(far-near) or similar depth encoding
  • m[3][2] = -near*far/(far-near) or -near
• Search all vertex shader constant registers (c0-c255)
• Cache the first valid non-identity projection matrix found

Proposed API:

// In d3d9_device.h
const Matrix4& GetProjectionMatrix() const;
bool HasValidProjectionMatrix() const;

4. AO Compositing in Blitter — Flexible Blending Formulas

Location: src/dxvk/shaders/dxvk_present_common.glsl → composite_image()

Requirement: Flexible AO compositing formula in the present blitter shader.

Why needed: Different games require different compositing approaches:

• Games without baked lighting: simple multiplicative blending
• Games with baked lighting (UE2, UE3): luma-masked blending to avoid double-darkening
• Creative/RTGI effects: additive blending for bounced light

Current limitation: The blitter uses a fixed formula: color.rgb *= max(0.0, 1.0 - ao). This causes severe double-darkening in games with pre-baked ambient occlusion in lightmaps.

Required change:

// In dxvk_present_common.glsl
layout(constant_id = 8) const bool c_composite_ao = false;
layout(constant_id = 9) const bool c_ao_debug_view = false;
layout(constant_id = 10) const int c_ao_blend_mode = 0;
  // 0 = multiplicative (default)
  // 1 = luma-masked (for baked lighting)
  // 2 = additive (for RTGI/bounced light)

vec4 composite_image(vec4 color) {
    ivec2 coord = ivec2(gl_FragCoord.xy);

    if (c_composite_ao) {
        float ao = texelFetch(s_ao, coord, 0).r;
        if (c_ao_debug_view) {
            color.rgb = vec3(ao);  // Debug: show raw AO
        } else if (c_ao_blend_mode == 0) {
            // Multiplicative: standard AO
            color.rgb *= max(0.0, 1.0 - ao);
        } else if (c_ao_blend_mode == 1) {
            // Luma-masked: avoid double-darkening baked lighting
            float luma = dot(color.rgb, vec3(0.2126, 0.5870, 0.1140));
            float mask = smoothstep(0.05, 0.30, luma);
            color.rgb *= (1.0 - ao * mask);
        } else if (c_ao_blend_mode == 2) {
            // Additive: for RTGI/bounced light effects
            color.rgb -= ao * 0.5;
        }
    }

    // ... rest of composite (HUD, cursor, etc.)
    return color;
}

Configuration:

• Blend mode should be configurable via dxvk.conf
• Default should be mode 0 (multiplicative) for backward compatibility
• Luma-masked mode (1) should be recommended for UE2/UE3 games

5. Descriptor Set Binding — AO Buffer to Blitter

Location: src/dxvk/dxvk_swapchain_blitter.cpp → descriptor set setup

Requirement: Mechanism to pass the AO buffer from the compute pipeline to the present blitter.

Why needed: The blitter must read the AO buffer to composite it onto the final frame.

Required change:

// In dxvk_swapchain_blitter.h
void SetAoView(Rc<DxvkImageView> aoView, bool debugView);

// In dxvk_swapchain_blitter.cpp
// Add binding 4 to the blitter descriptor set layout:
// binding 4: sampled image (AO buffer)

// In present():
if (m_aoView) {
    descriptors[4] = m_aoView->getDescriptor();
    ctx->track(m_aoView->image(), DxvkAccess::Read);
}

Implementation notes:

• AO buffer must be in VK_IMAGE_LAYOUT_GENERAL or VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL
• Proper synchronization: compute write → fragment read barrier must be implicit via DXVK's state tracker
• The AO view should be cleared/reset when AO is disabled

Summary Table

Total estimated effort: ~125 lines of code across 6 files.

What Is NOT Required

• ❌ Changes to the Vulkan API or drivers
• ❌ Support for new Vulkan extensions
• ❌ Changes to D3D11 or D3D12 backends
• ❌ Changes to WSI or presentation code
• ❌ Changes to the DXVK state tracker core
• ❌ Changes to command buffer management

All changes are localized to the D3D9 backend and present blitter. They are backward-compatible and do not affect existing functionality.

Reference Implementation

A working implementation of all 5 requirements is available in a private development fork with integrated AMD FidelityFX CACAO SSAO pipeline.

Key files in the implementation:

• src/d3d9/d3d9_swapchain.cpp — Present hook integration
• src/d3d9/d3d9_cacao.cpp — Full CACAO compute pipeline (26 SPIR-V shaders from FFX SDK)
• src/d3d9/d3d9_device.cpp — Projection matrix caching from SetTransform + VS constants
• src/dxvk/shaders/dxvk_present_common.glsl — AO compositing with luma-masked blending
• src/d3d9/d3d9_common_texture.cpp — SAMPLED_BIT for depth-stencil surfaces

Tested on:

• Lineage 2 (Unreal Engine 2, D3D9)
• AMD RADV (RDNA2)

Conclusion

For AO mod support in DXVK's D3D9 backend, 5 changes are needed (3 critical, 2 important). All changes are backward-compatible, localized to the D3D9 backend and present blitter, and do not require any Vulkan API changes or driver modifications.

Without these changes, compute-based SSAO/AO mods are impossible in DXVK for D3D9 games. With these changes, any AO technique (CACAO, HBAO+, RTGI, MXAO, etc.) can be integrated as an external mod without modifying DXVK's core code.