How to build Proton with VAAPI support?

protonclosed

ValveSoftware/Proton#7306 · opened 2023-12-02 by wallcarpet40 · updated 2024-12-28 · 2 comments · github

Wwallcarpet40 2023-12-02 github

Hello,

I would like to build Proton with VAAPI support, so that any VR-videoplayer running on Proton would use my GPU for decoding instead of my CPU.
Player in guestion is Whirligig, but should be applicable for other VR-players as well.

Proton, Proton Experimental and GE-Proton all use CPU for decoding.
Proton-TKG can use GPU for decoding, but fails to display the VR player in the headset and only outputs to the desktop.

What I have tried so far:

I have appended these lines to Makefile.in:

FFMPEG_CONFIGURE_ARGS := \
	--enable-vaapi \
	--enable-gpl \
	--enable-nonfree \

GSTREAMER_MESON_ARGS := \
	-Dvaapi=enabled \
	-Dgpl=enabled \

I have successfully compiled a custom Proton version.
I've used Protontricks to configure Whirligig's Proton Prefix to use VAAPI backend in the Staging tab of Winecfg.

I can start Whirligig with my custom Proton, but still only CPU decoding, only a Proton-TKG version will use GPU.

I have looked at the build log and I see some mentions of vaapi, but I'm not sure, if it has actually enabled VAAPI support for Gstreamer (I assume it's needed for Gstreamer).

Any help would be appreciated.

CPU: AMD 2600X
GPU: AMD Radeon 5700XT
OS: Arcolinux with latest 6.6 kernel
VR Headset: Valve Index

Wwallcarpet40 2023-12-03 github

At least some progress has been made. I was able to compile gstreamer-vaapi with Proton by adding these lines to Makefile.in :

##
## gstreamer-vaapi
##

GST_VAAPI_DEPENDS = gst_orc gstreamer gst_base gst_bad gst_ugly gst_good

$(eval $(call rules-source,gst_vaapi,$(SRCDIR)/gstreamer/subprojects/gstreamer-vaapi))
$(eval $(call rules-meson,gst_vaapi,32))
$(eval $(call rules-meson,gst_vaapi,64))

I can start Whirligig with the new custom Proton and I can even see some VA-API text in the steam steam-451650.log file:

0:00:15.366664606 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: VA-API version 1.20.0
0:00:15.366692186 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: Trying to open /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
0:00:15.367685651 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: Found init function __vaDriverInit_1_20
0:00:15.377258745 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: va_openDriver() returns 0
0:00:15.377291614 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:122:vaapi_initialize: VA-API version 1.20
0:00:15.380636938 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: VA-API version 1.20.0
0:00:15.380665488 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: Trying to open /usr/lib/pressure-vessel/overrides/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
0:00:15.380740838 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: Found init function __vaDriverInit_1_20
0:00:15.386527679 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:95:gst_vaapi_log: va_openDriver() returns 0
0:00:15.386548619 18266 0x7f7b48001ea0 INFO                   vaapi gstvaapiutils.c:122:vaapi_initialize: VA-API version 1.20
0:00:15.386555519 18266 0x7f7b48001ea0 INFO            vaapidisplay gstvaapidisplay.c:983:gst_vaapi_display_create:<vaapidisplaydrm0> new display addr=0x7f7b6c0c51d0
0:00:15.386565809 18266 0x7f7b48001ea0 INFO            vaapidisplay gstvaapidisplay.c:795:ensure_vendor_string:<vaapidisplaydrm0> vendor: Mesa Gallium driver 23.2.1-arch1.2 for AMD Radeon RX 5700 XT (navi10, LLVM 16.0.6, DRM 3.54, 6.6.3-arch1-1)
0:00:15.386576379 18266 0x7f7b48001ea0 INFO            vaapidisplay gstvaapidisplay.c:834:set_driver_quirks:<vaapidisplaydrm0> Matched driver string "Mesa Gallium driver 23.2.1-arch1.2 for AMD Radeon RX 5700 XT (navi10, LLVM 16.0.6, DRM 3.54, 6.6.3-arch1-1)", setting quirks (0x1)

I still get either a crash or it does not appear to be using the GPU for decoding, depending on what format of media I'm trying to watch.

The crash seems to be with these lines:

wl_drm@10: error 0: authenticate failed
VA error: wayland: Wayland roundtrip error: Protocol error (errno 71)

Gist for the log:

https://gist.github.com/wallcarpet40/15c6cc5b870458a88d26bb1c98a4b009

edit: Switching to X11 seems to make this Wayland errno 71 go away and GPU decoding seems to be working, at least on some file formats (looking with Mission Center and observing GPU decoder being used. )
Videopath in Whirligig needs to be "Direct Show". VideoLAN will not use GPU and Media Foundation crashes.

Still something weird about this

Rroadapathy 2024-12-28 github

Ummm. I would like to see FFmpeg updated to 6.x or 7.x too. I tried dropping into the project the new FFmpeg but it doesn't build. To see if that's even possibly, I replaced FFmpeg with a version just slightly newer. 4.4.3 is what Proton uses and it's very old but when I tried dropping in 4.4.6, no luck. There's probably a good reason why the project uses a very old FFmpeg but I can't see that. Great project though. Thank you!

Proton versions

proton experimentalx1 2023-12