Previous comments of the developer:

Thanks for trying. Once you click continue / start game a Python process should briefly start in the background to cache some things. This python interpreter is included with the the game (python.exe somewhere deep in the steamapps/JOY OF PROGRAMMING folder). Could that be the problem that the WinApi command StartProcess I'm using from c++ to start this python process crashes proton?

In this thread.

Thank you @Patola and thank you @kisak-valve for tracking this issue. I'm the developer of the game. I recently got a Steam Deck and tried running the game myself under Proton (experimental and 9). It seems the included Python interpreter is working just fine. On the deck the game does not crash, but rather freezes / screen turns black once I select a level to play and I can only terminate it afterwards. This is the last few lines of the Proton log before the game freezes:

2840.390:0120:0124:fixme:win:GetPointerDevices (0000000000B9EF34 0000000000000000): partial stub
2840.390:0120:0124:fixme:system:NtUserQueryDisplayConfig flags 0x2, paths_count 0xb9ef00, paths 0x1304006be300, modes_count 0xb9ef34, modes 0x130400561d80, topology_id (nil) semi-stub
2840.390:0120:0124:fixme:system:NtUserDisplayConfigGetDeviceInfo Unimplemented packet type 11.
2840.390:0120:0124:fixme:ui:uisettings2_get_TextScaleFactor iface 00000000337527B8, value 0000000000B9EE10 stub!
2840.409:03b8:0618:warn:threadname:NtSetInformationThread Thread renamed to L"ThreadPoolForegroundWorker"
2840.409:03b8:0614:warn:threadname:NtSetInformationThread Thread renamed to L"ThreadPoolForegroundWorker"
2840.409:03b8:0610:warn:threadname:NtSetInformationThread Thread renamed to L"ThreadPoolForegroundWorker"
2840.412:03b8:061c:warn:threadname:NtSetInformationThread Thread renamed to L"ThreadPoolForegroundWorker"
2840.515:0104:0118:fixme:uiautomation:msaa_provider_GetPatternProvider Unimplemented patternId 10002
2840.515:0104:0118:fixme:uiautomation:base_hwnd_provider_GetPatternProvider 0000000000B159C0, 10002, 000000000193F8A0: stub
2840.608:0104:0118:fixme:uiautomation:msaa_provider_GetPatternProvider Unimplemented patternId 10002
2840.608:0104:0118:fixme:uiautomation:base_hwnd_provider_GetPatternProvider 0000000000B15FB0, 10002, 000000000193F8A0: stub
2844.566:03b8:062c:warn:threadname:NtSetInformationThread Thread renamed to L"ThreadPoolForegroundWorker"
2855.848:0120:0264:fixme:process:CreateProcessInternalW Unsupported attribute 0x20007.
wine: using kernel write watches, use_kernel_writewatch 2.
wine: overriding CPU configuration, 8 logical CPUs, host CPUs 0,1,2,3,4,5,6,7.
2855.962:0630:0634:trace:seh:sigsys_handler SIGSYS, rax 0xffff, rip 0x600000000005.
2855.962:0630:0634:trace:seh:install_bpf Seccomp filters already installed.
pid 9526 != 9525, skipping destruction (fork without exec?)

I don't have any significant Linux experience and never worked with Proton, so I'm not really sure what to look for to try and fix this issue.

@kisak-valve I found the issue - a little inconsistency between Windows and Proton: My game spawns a separate Python process and saves the handle. This all works as expected. Sometimes I forcibly close this Python process using
TerminateProcess( _In_ HANDLE hProcess, _In_ UINT uExitCode );
In my calling code, I did not check whether hProcess is valid or not and it was indeed sometimes called with a NULL handle. This did not cause any issues under Windows. In Proton however this crashes the whole game. I fixed this for my game by checking for a valid handle and it now seems to work. I will do some further tests and push an experimental branch to Steam this week.

@maschere Thank you for the detailed description! We are going to look into things on our end and see if we need to change something to better match Windows behavior. I see that the old version of the game before your fix is still live on the main branch but the beta branch already has the fixed version. If it is possible for you to temporarily leave the old version live on some beta branch for us to test with so that we can improve Proton/Wine, that would be very greatly appreciated :)

@maschere Hi! I am a Proton developer and I had a look from Proton side.

TerminateProcess(NULL, ) cannot do that, it doesn't do anything. NULL handle is explicitly checked and treated as invalid in TerminateProcess implementation and such a call just sets last error and returns FALSE (like on Windows).

I looked what is actually going on there and it looks like the problem is that the game tries to kill the whole tree starting from pid 0. It recursively traverses the process tree using CreateToolhelp32Snapshot(), apparently skipping the processes which do not have the specified parent process ID and descending recursively to kill subprocesses of the found process which could be successfully open with OpenProcess. So what happens with Wine when this process gets pid 0 as initial parent PID is it traverses the whole tree, kills everything (like, explorer.exe, own CEF subprocesses and finally itself, not through 0 pid but correct open pid and a handle opened from it by OpenProcess). Why that doesn't happen on Windows is that the whole system process tree (also present in CreatdToolhelp32Snapshot data) is different. All the processes which are direct chidlren of process with pid 0 are system processes, OpenProcess() for their pid fails (due to insufficient permissions) and the traversal doesn't reach any process which can be actually killed.

So the difference in Proton here is that we currently don't recreate the full Windows system process structure, e. g., our steam.exe is direct decendant of pid 0 (while the game is decendant of steam.exe), all of those run as the same user and accessible to open (similarly it is possible to open steam.exe on Windows but it is not a direct child of process 0).

So the change from the Proton side to make it work like on Windows would be to introduce some (not so much needed otherwise) additional limited access system processes, which we ideally would like to avoid for solving this very specific problem (also given it is said to be already fixed in the next version).

There are few other potential issues with this process which I spotted while looking into it though, which issues might be at the same time potential issues on Windows:

1. During that traversal the game calls CreateToolhelp32Snapshot many times for each level of recursion. And doesn't seem to ever CloseHandle() on those returned snapshot handles. Those handles from CreateToolhelp32Snapshot are actually file mapping handles, not the most cheap resource, leaking so lot (assuming such a subprocess kill scan is performed periodically) might lead to problems on long game runs;
2. Besides python processes, this process also kills Chromium CEF render and etc. subprocesses (running EpicWebHelper.exe) which are also children of the game process, created as a part of embedded Chromium operation (GPU process, utility etc.). Not sure if that is intended, but if not maybe something best to avoid even if that works now somehow (e. g., through Chromium restarting its processes);
3. Using PID tree captured by CreateToolhelp32Snapshot to kill all the descendants is not entirely reliable. One potential issue is that the parent PID in the process info is the one which was the parent in creation. If that parent process terminates (e. g., normally by itself) its children will still have that stale parent ID they had on creation. So, e. g,, if a child process of the game has exited but there were some grandchildren those won't be found and killed that way. There is also an opposite potential problem. I think I spotted the game opening some processes (created earlier) using their pid with OpenProcess and terminating at the time when the process is already exited (that happens on game start, before we reach that total exit on Proton). If there are no open handles to the process the process ID can be reused for some new processes, so opening the process by pid later has the potential of reaching and killing entirely unrelated process, would it be one of the newer game's child processes or unrelated process started by the user.

If the above looks concerning to some extent, I can suggest an easier and perfectly reliable way to manage that using jobs (intended specifically for such usage patterns). Those are created CreateJobObject(). Then a process being created can be added to job at once using PROC_THREAD_ATTRIBUTE_JOB_LIST extended process creation attribute. Or, alternatively, added to job after creation using AssignProcessToJobObject(). In the latter case it is best to create the process suspended and resume after assigning to job to guarantee it doesn't create any subprocesses before assigned to job. All the children of the child processes will be created in the same job. So to stop all those child and grandchild processes in job there is TerminateJobObject(). All that is supported in Wine / Proton (apart from proper job resources accounting and limits but it is not needed for this purpose).

@gofman Thanks for the detailed reply. I made the old version of the game available under a private steam branch called "dev_internal" with the highly secure password "devpw12characterslong".
Also I discovered another inconsistency in the game under proton concerning TCP socket data transfer. The game uses a non-blocking TCP socket to exchange data between the game process and the python process used to script in-game entities. This works fine in almost all levels, except for those were I am transferring image data from the game to the python process. This never seems to reach it, but works fine under windows. The image capture process itself within the game seems to work fine as I can see the captured image in-game, but cannot retrieve it in python. Apart from that I have not been able to track down the issue further or figure out a workaround.

Also I discovered another inconsistency in the game under proton concerning TCP socket data transfer. The game uses a non-blocking TCP socket to exchange data between the game process and the python process used to script in-game entities. This works fine in almost all levels, except for those were I am transferring image data from the game to the python process. This never seems to reach it, but works fine under windows. The image capture process itself within the game seems to work fine as I can see the captured image in-game, but cannot retrieve it in python. Apart from that I have not been able to track down the issue further or figure out a workaround.

Could you please give exact reproduction instructions if possible? Or, if that requires some time to get to the level, maybe some private save file or any way I can debug it?

Sure thing @gofman .
Quickest way would be to start the "Level Editor" from the main menu (after clicking "Play Game" from the first person view). Then open the "Code" window and click "Construct" after you pasted the following Python snippet in there:

from pyjop import *

SimEnv.connect()
editor = LevelEditor.first()
### END INIT CODE ###


### CONSTRUCTION CODE - Add all code to setup the level (select map, spawn entities) here ###
editor.select_map(SpawnableMaps.MinimalisticIndoor)
editor.spawn_entity(SpawnableEntities.SmartCamera, "cam", location=(0,0,1))
sleep(1)
cam = SmartCamera.first()
print(cam.get_camera_frame())


### EOF CODE - DO NOT CHANGE ###
editor.run_editor_level()
### EOF ###

Here is the expected output as it is happening in windows

Under Proton (note that I only tested this on SteamDeck), the print output on the lower right (red rectangle) will be black. However the camera capture itself as seen in the green rectangle will still work (indicating it is not a rendering issue).

@maschere

Thanks for the info, this is most helpful.

Indeed, the problem is in TCP non-blocking socket transfer between main process and a python process. The socket on the main process is not explicitly set as non-blocking but is initialized as such after accept() because the listening socket is non-blocking. What is specific about sending (probably) that image is that send() is called with larger length than for the other objects (~250k). On Proton / Linux with the set up SO_SNDBUF it currently always ends up in short write (i. e., the actually written by send() and returned length is smaller than requested; while tweaking SO_SNDBUF would not be a solution here, it may still end up with short write if there is some congestion on delivering the data). The game actually tries to handle short write and calls send() again advancing send buffer pointer to the number of bytes returned by previous (short) send(). The problem is that it doesn't decrement the amount of bytes to send (e. g, while advancing buffer start, calls next sends with exact same length as the initial one). That results in send() attempting to read past the allocated buffer and failing with WSAEFAULT (that doesn't cause access violation because the data access happens on the kernel side and results in the error without sending anything). The loop which does this repeated sending and has this issue starts at address 0x141d8e400 in the main executable (for the default game branch as of now). But exactly the same behaviour is also observed in the python process, I didn't look where exactly that happens there.

Why that works on Windows is because Windows actually never does short send() (regardless of what is said in [1]). It may return -1 and set WSAGetLastError() to WSAEWOULDBLOCK (i. e., send nothing) or queue the full requested length for sending (of virtually unlimited size, I've tested that up to 2GB) and return the requested length. So we actually have a compatibility issue in Proton. I sent a patch Wine upstream [2] for that, but this is a bit tricky and has potential of breaking a lot if I missed something, so I am not inclined to rush that into Proton and prefer to wait for some review comments first.

Meanwhile, maybe that can be fixed in game? That would probably be more robust regardless (by properly reducing the length on those repeated sends)? I don't know though if that is inside the game code itself or inside some library used, if the latter then probably would be ideal to let the library developers know.

1. https://learn.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-send , "Return value" section: "If no error occurs, send returns the total number of bytes sent, which can be less than the number requested to be sent in the len parameter."
2. https://gitlab.winehq.org/wine/wine/-/merge_requests/6490

@maschere That short send() issue should be hopefully fixed in the current Proton Experimental ([bleeding-edge] branch).

@gofman thanks, that is great news. Is this branch already available on SteamDeck? I just tried it with "Proton Experimental" there, but it was the same behaviour as before.

Thank you, I can confirm it's now working as expected and identical to the windows version.