Edit: CLICK TO GO TO THE SOLUTION.

@doitsujin The game has a famous issue on Linux where DLSS will allocate memory for CUDA within the game's existing memory, thus overwriting some game memory.

Someone created a patch for libcuda which somehow fixes that. I dug up the patch and particularly this interesting comment:

https://github.com/Open-Wine-Components/umu-protonfixes/blob/cf8d5a2ef83d09f84ba4fa475b01b889d97f6300/util.py#L565-L574

It says that if the game is launched with the Vulkan extensions VK_NVX_binary_import and/or VK_NVX_image_view_handle then the game crashes on startup.

It also mentions that DXVK contained a patch that automatically re-creates the Vulkan device without those extensions to fix the issue.

Perhaps that's the patch which has been removed in DXVK 2.7?

However... I am running the game with WINE_HIDE_NVIDIA_GPU=1 which is supposed to already fix this issue by hiding DLSS from the game. Also, I am not seeing a crash. I'm seeing the inability to create a Vulkan rendering device at all. Hmm.

@doitsujin Yeah, the vulkan extensions would still be visible, but since the GPU masquerades as AMD then the game wouldn't try to use DLSS/CUDA's APIs.

I'm not getting a crash, so it doesn't seem like libcuda is being loaded. The symptom of that old bug was always that the game would start up but would quickly crash due to a libcuda memory allocation error. It doesn't seem to be what is happening here.

The symptom I'm seeing is a failure to create the rendering output device before the game window appears at all:

err: D3D11InternalCreateDevice: Failed to create D3D11 device.

https://github.com/jp7677/dxvk-nvapi/issues/174#issuecomment-2227462795
https://github.com/starcitizen-lug/knowledge-base/issues/67

Your error message is getting triggered when u use wine staging without patched cuda.

this is a known bug. the patch is working pretty well. guess thats on nvidia

when u need to skate around EAC u can 1) patch cuda patch 2) and then mount mount it over your real cuda. without EAC LD_PRELOAD works too.

TL;DR if u want to play SC on nvidia with EAC u need a custome wine runner NON STAGING since staging addresses to much memory.

option 1) use wine non staging
option 2) patch cuda and use staging

don't wanna do any ads here but i guess can figure out where to get help with that

I wanted to see if libcuda is truly loaded, because I still don't believe that it is responsible since the game is NOT trying to do any CUDA/DLSS calls. Sure, the library is loaded, but it's not USED.

I ran strace to trace syscalls, follow forked child processes, track all file open/connect calls:

strace -f -e trace=open,openat,connect flatpak run com.usebottles.bottles 2>&1 | grep -i libcuda

We can see that the Electron-based launcher itself (not the game) causes libcuda to be loaded. This is the console output when the launcher opened:

[pid 91433] openat(AT_FDCWD, "/app/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 91433] openat(AT_FDCWD, "/app/lib32/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 91433] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.12-staging-tkg-amd64/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 91433] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.12-staging-tkg-amd64/lib/wine/x86_64-unix/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 91433] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.12-staging-tkg-amd64/lib/wine/i386-unix/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 91433] openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/GL/nvidia-575-64-03/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = 213

I then wanted to see if the game loads it too.

But when I click "Play", EasyAntiCheat detects the strace debugger and refuses to start the game. So the renderer is never initialized. Therefore I can't see whether the game itself loads libcuda. But even if it loads it, it doesn't mean the game uses it. It should not be using it at all, since the GPU is hiding as a AMD Radeon card.

Here's the summary so far:

• Star Citizen requires wine-staging with TKG patch to enable the EasyAntiCheat bridge. Without it, the anticheat doesn't work and refuses to launch the game at all.
• libcuda.so.1 is being loaded into every process that uses 3D, regardless of whether the process calls CUDA/DLSS functions at all. I am guessing that NVIDIA's own 3D driver is responsible for always forcing this to be loaded.
• The game is being run with WINE_HIDE_NVIDIA_GPU=1 which means the GPU is hiding as an AMD Radeon, meaning that the game is not using CUDA/DLSS at all.
• The issue is that DXVK 2.7 fails to initialize the Vulkan rendering pipeline.

Perhaps libcuda is interfering with the Vulkan rendering pipeline creation regardless of whether the game uses it or not?

It's not possible to patch libcuda on Bottles (a Flatpak app) to test this theory.

Alright. The issue is caused by libcuda.so. Even though the game doesn't call any CUDA/DLSS functions, the mere fact that the library is auto-loaded by NVIDIA's driver causes the entire Vulkan stack to crash out and fail to initialize.

See the "patch.py" at the end of this post, to patch your runner.

Here's how I figured it out and solved it:

1. Seeing the previous strace made me realize that there's a higher load-order directory in the list. We can see that it tries a bunch of library locations and then finally gives up and tries loading libcuda from the NVIDIA driver's default path:

[pid 98484] openat(AT_FDCWD, "/app/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98484] openat(AT_FDCWD, "/app/lib32/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98484] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98484] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/wine/x86_64-unix/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98484] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/wine/i386-unix/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98484] openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/GL/nvidia-575-64-03/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = 209

1. Looking at the list, we can't modify most of them since they are in paths that cannot be mounted on Flatpaks. But luckily, there's one path that is part of the runner itself and therefore easy to modify outside the Flatpak (Bottles):

~/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/wine/x86_64-unix/

1. Next, we need to actually patch the library file. I wrote some code based on the protonfixes version of the libcuda patch. It finds libcuda on the host (there's no concern about version mismatches since CUDA is separate from the driver version), then patches it to generate libcuda.so.1 in the same directory the patch script is run from. So just execute it anywhere.

I also made it automatically install the patched binary for every kron4ek runner for the Bottles Flatpak, since I'll be using this patch method for as long as necessary and don't want to deal with copying things manually.

1. Finally, copy the libcuda.so.1 library file into the runner's special library directory. It depends on the runner, but for kron4ek, that's the following path. But note that my patch.py automatically installs the patches into this directory already.

~/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/wine/x86_64-unix/

1. I then ran one more trace to verify that libcuda is now loading the patched version instead of the default NVIDIA driver version:

[pid 98834] openat(AT_FDCWD, "/app/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98834] openat(AT_FDCWD, "/app/lib32/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98834] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/libcuda.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
[pid 98834] openat(AT_FDCWD, "/home/johnny/.var/app/com.usebottles.bottles/data/bottles/runners/kron4ek-wine-10.10-staging-tkg-amd64/lib/wine/x86_64-unix/libcuda.so.1", O_RDONLY|O_CLOEXEC) = 207

1. The game now loads properly.

So with this much verified, what do you think is the appropriate path forward, @doitsujin? I guess we close this issue and push the issue to NVIDIA. Or maybe leave the issue open until NVIDIA fixes it?

I am guessing that you have good connections with the developers at NVIDIA. What method do you suggest for reporting this bug to them and actually getting it fixed? It seems like a very minor tweak of their code.

I still don't understand what the actual issue with libcuda is. Apparently it tries to allocate some CUDA memory inside the game's own address space? And apparently the libcuda patch expands the maximum address range so that CUDA can place itself in higher memory. That's my guess, but it could be wrong because there's not really any clear information on this bug or what the patch actually does.

patch.py:

#!/usr/bin/env python

import os
import re
import shutil
import subprocess
from pathlib import Path

SCRIPT_DIR = Path(__file__).resolve().parent


# Ported from Protonfixes:
# https://github.com/Open-Wine-Components/umu-protonfixes/blob/cf8d5a2ef83d09f84ba4fa475b01b889d97f6300/util.py#L507
def patch_libcuda() -> Path:
    """Patches libcuda to work around games that crash when initializing libcuda and are using DLSS.

    We will replace specific bytes in the original libcuda.so binary to increase the allowed memory allocation area.

    The patched library is overwritten at every launch and is placed in .cache

    Returns true if the library was patched correctly. Otherwise returns false
    """

    # Use shutil.which to find ldconfig binary
    ldconfig_path = shutil.which("ldconfig")
    if not ldconfig_path:
        print("ldconfig not found in PATH.")
        raise RuntimeError("cannot find ldconfig")

    # Use subprocess.run with capture_output and explicit encoding handling
    try:
        result = subprocess.run([ldconfig_path, "-p"], capture_output=True, check=True)
        # Decode the output using utf-8 with fallback to locale preferred encoding
        try:
            output = result.stdout.decode("utf-8")
        except UnicodeDecodeError:
            import locale

            encoding = locale.getpreferredencoding(False)
            output = result.stdout.decode(encoding, errors="replace")
    except subprocess.CalledProcessError as e:
        print(f"Error running ldconfig: {e}")
        raise e

    libcuda_path = None
    for line in output.splitlines():
        if "libcuda.so" in line and "x86-64" in line:
            # Parse the line to extract the path
            parts = line.strip().split(" => ")
            if len(parts) == 2:
                path = parts[1].strip()
                if os.path.exists(path):
                    libcuda_path = os.path.abspath(path)
                    break

    if not libcuda_path:
        print("libcuda.so not found as a 64-bit library in ldconfig output.")
        raise RuntimeError("cannot find libcuda at expected path")

    print(f"Found 64-bit libcuda.so at: {libcuda_path}")
    from tempfile import mkdtemp

    patched_library = Path(mkdtemp()) / os.path.basename(libcuda_path)
    try:
        binary_data = Path(libcuda_path).read_bytes()
    except OSError as e:
        print(f"Unable to read libcuda.so: {e}")
        raise e

    # Replace specific bytes in the original libcuda.so binary to increase the allowed memory allocation area.
    # Context (see original comment here: https://github.com/jp7677/dxvk-nvapi/issues/174#issuecomment-2227462795):
    # There is an issue with memory allocation in libcuda.so when creating a Vulkan device with the
    # VK_NVX_binary_import and/or VK_NVX_image_view_handle extensions. libcuda tries to allocate memory in a
    # specific area that is already used by the game, leading to allocation failures.
    # DXVK and VKD3D work around this by recreating the device without these extensions, but doing so disables
    # DLSS (Deep Learning Super Sampling) functionality.
    # By modifying libcuda.so to increase the allowed memory allocation area, we can prevent these allocation
    # failures without disabling the extensions, thus enabling DLSS to work properly.
    # The hex replacement changes the memory allocation constraints within libcuda.so.

    hex_data = binary_data.hex()
    hex_data = hex_data.replace("000000f8ff000000", "000000f8ffff0000")
    patched_binary_data = bytes.fromhex(hex_data)

    try:
        patched_library.write_bytes(patched_binary_data)

        # Set permissions to rwxr-xr-x (755)
        patched_library.chmod(0o755)
        print(f"Permissions set to rwxr-xr-x for {patched_library}")
    except OSError as e:
        print(f"Unable to write patched libcuda.so to {patched_library}: {e}")
        raise e

    print(f"Patched libcuda.so saved to: {patched_library}")

    return patched_library


def copy(source: Path, target_dir: Path) -> Path:
    target_file = target_dir / source.name
    if target_file.is_file():
        target_file.unlink()
    print(f"Copying to: {target_file}")
    shutil.copy2(source, target_file)

    return target_file


# Generate patched library.
patched_library = patch_libcuda()
copy(patched_library, SCRIPT_DIR)


# Patch all kron4ek runners for Bottles.
bottles_runners = Path.home() / ".var/app/com.usebottles.bottles/data/bottles/runners"
if bottles_runners.is_dir():
    for runner_base in bottles_runners.glob("kron4ek*"):
        if not re.search(r"-amd64$", runner_base.name):
            print(f"Skipping runner, not amd64 variant: {runner_base}")
            continue

        libs_subdir = runner_base / "lib/wine/x86_64-unix"
        if not libs_subdir.is_dir():
            print(f"Skipping runner, missing library dir: {libs_subdir}")
            continue

        copy(patched_library, libs_subdir)

Summary of usage:

• Create a directory anywhere, and place patch.py inside it.
• Run python patch.py.
• It will automatically find and patch libcuda, and will place the result in its own directory.
• If you're using the Bottles Flatpak with the Kron4ek runner (currently the best runner while we're all waiting for GE-Proton support with UMU later), it will automatically install the patch into every Kron4ek runner.
• If you aren't using that runner, or if you're using an unofficial non-Flatpak version of Bottles, then you need to manually install the patched libcuda into all of your Wine/Proton runners which are affected by this NVIDIA bug. Look at this message in detail and try to find your own runner's appropriate "lib" directory.
• Hopefully NVIDIA will fix this bug soon, since DXVK 2.7+ is no longer automatically masking their bug with a workaround. I look forward to a day when this patch isn't necessary.

@doitsujin I let NVIDIA know about the libCUDA driver issue and how it affects DXVK 2.7+ here:

https://forums.developer.nvidia.com/t/580-release-feedback-discussion/341205/225

It received this response from an NVIDIA developer, with very interesting new information:

I tracked down the history of this and it looks like it’s being tracked in internal bug number 4745620. It sounds like the problem is not exactly a bug, but a limitation in which virtual address ranges are needed by CUDA and an incompatibility between how CUDA needs to run and how DXVK is using memory. It’s being worked on but it doesn’t sound like it’s simple to solve.

The patch sounds pretty iffy to me. If it works, it’s likely purely by luck depending on which address ranges the game ends up using.

That response is here:

https://forums.developer.nvidia.com/t/580-release-feedback-discussion/341205/228

Note that bug number 4745620 is very old, so it's been known for a very long time, but a mix of difficulty in solving it and the "workarounds" (that auto-disabled DLSS if it crashes) in both DXVK and VKD3D is probably why it hasn't been solved. For comparison, in August this year they mentioned filing another internal bug for something else and got ID 5436037, so there have been 690 417 new bugs filed internally since they first heard about the libCUDA bug.

Hopefully it gets high priority since DXVK no longer contains a workaround. :) It's nasty that GE-Proton, UMU and others have resorted to patching libcuda.so.