protonscr

[BUG] [Regression] Overlay Wobble/Jitter/Artifacting introduced in 1.15.X

steamvrclosed bug
ValveSoftware/SteamVR-for-Linux#395 · opened 2020-10-07 by mcoffin · updated 2023-10-21 · 72 comments · github
Mmcoffin 2020-10-07 github

Description

Starting with SteamVR 1.15.X, overlays (via IVROverlay from a game client, and the "system" overlay displayed when you hit the "system" button on the index controllers), wobble, as if they are attached to the HMD with a rubber band. If you move around enough, there will also be significant artifacting ocurring in the overlays (though it's a little harder to induce with the system overlays compared to the IVROverlay instances created by ACC).

This only occurs when asynchronous reprojection is enabled.

Reproduction Steps

Steps to reproduce the behavior:

  1. Launch SteamVR 1.15.2
  2. Open system overlay
  3. Move your head around

Expected Behavior

The overlay should stay static in space, and not have artifacts, as it did in SteamVR 1.14.16.

System Information

  • Distribution: Arch Linux (amd-staging-drm-next kernel - f24cd554aacf4e989a0796c7d30d758115512732 rebased off of v5.9-rc6 from Linus' tree with rebased futex patches from mcoffin/[email protected], but this behavior is observed on stable channel kernels as well)
  • Exact kernel (though likely non-factor since reproducible on stable channel as well) - mcoffin/linux@7d4ad53110a23391a790e8baf22dd48f8a1f0602
  • SteamVR version: 1.15.2 (working: 1.14.16)
  • Steam client version: Built: Sep 3 2020 at 21:18:20
  • Opted into Steam client beta?: No
  • Graphics driver version: Mesa 20.2.0 (git-663d464366) (ACO), AMD Radeon RX 5700 XT (NAVI10, DRM 3.40.0, 5.9.0-rc6-1-amd-staging-drm-next-git-00457-g3637df487bdb, LLVM 11.0.0). Mesa versions down to 20.1.4 were also tried, with no change in behavior. Regression still occurs between 1.14.X and 1.15.X.
  • Gist for SteamVR System Information: https://gist.github.com/mcoffin/bc4460030a4414e8929c78e252a80766
  • CPU: AMD Threadripper 3960X

Additional Context

A cpu profile with sysprof shows a significant amount of time being spent in clock_gettime compared to "working" versions. (~10% of samples for ACC with it open!). This could indicate a busy-wait somewhere that's misbehaving?

Mesa 20.2.0 released a whole bunch of RADV features for timeline syncobj's, and I chased down that path for a while, but observing the same behavior with previous mesa releases ruled the timing of this release out as the issue.

Under normal operation, I'm running a pretty hefty GPU overclock, and setting the min frequency to quite close to the max to achieve livable performance, but disabling this (clean boot with no writing to sysfs), did not affect the observed behavior in any way.

Screenshots

Unfortunately, overlays are only displayed in the HMD for me, so I cannot capture this behavior, though I can look in to alternative solutions if upstream developers cannot readily reproduce the issue.

In-progress debugging

  • [ ] Hook up a 2070 super from my GF's computer to see if the problem is isolated to amdgpu/mesa interactions with SteamVR
  • [x] Downgrade mesa pre-timeline-syncobj implementation (result: no change)
  • [x] Use stable kernels (result: no change)
Mmcoffin 2020-10-08 github

New note: This only occurs when asynchronous reprojection is enabled.

ZZamundaaa 2020-10-09 github

Same here.

System Information (please complete the following information):

  • Distribution: Manjaro KDE
  • SteamVR version: 1.15.2
  • Steam client version: 1602115886
  • Opted into Steam client beta?: yes
  • Graphics driver version: Mesa 20.1.8 (LLVM 10.0.1)
  • SteamVR System Information: SteamVR-2020-10-09-PM_03_08_42.txt
WWebFreak001 2020-11-03 github

can reproduce:

Distribution: Arch Linux (5.9.2-zen1-1-zen)
SteamVR version: tested 1.15.2 - 1.15.5 (worked in 1.14.16)
Steam client version: Built: Oct 28 2020 at 23:35:02
Opted into Steam client beta?: Yes
Graphics driver version: Mesa 20.2.1 on AMD Radeon RX 5700 XT (NAVI10, DRM 3.39.0, 5.9.2-zen1-1-zen, LLVM 10.0.1)
Gist for System Information: https://gist.github.com/WebFreak001/8fe9ab2ad916efc11a42c483fb6b44b5
SteamVR System Information: https://gist.github.com/WebFreak001/35b7e824996ec98d763bb6978a74f2c7
CPU: AMD Ryzen 7 1700X

Dduckbytes 2020-11-18 github

I have this problem too and now that the beta has been pushed into the main release branch it can no longer be avoided by opting out of the beta.

Distribution Arch Linux
Linux 5.9.8-zen1-1-zen #1 ZEN SMP PREEMPT Tue, 10 Nov 2020 22:44:06 +0000 x86_64 GNU/Linux
mesa 20.2.2-2
AMD 5700XT
AMD 3700X

Edit: the problem also effects xrdesktop windows.

Hhex3562 2020-11-19 github

Can reproduce

SteamVR version: 1.15.10
Distribution: Arch Linux
Linux 5.9.8-zen1-1-zen #1 ZEN SMP PREEMPT Tue, 10 Nov 2020 22:44:06 +0000 x86_64 GNU/Linux
Mesa 21.0.0-devel
AMD Radeon RX 580 Series (POLARIS10, DRM 3.39.0, 5.9.8-zen1-1-zen, LLVM 11.0.0)
AMD Ryzen 5 3600

Llostgoat 2020-11-23 github

There is a linux_1.14 branch available in SteamVR as a temporary solution to this problem.

Ssupernovajm 2020-11-23 github

Can reproduce, but not just the overlay. Timing-sensitive games like Beat Saber felt significantly more choppy, which the fps graph confirmed.

SteamVR version 1.15.10
Distribution: Kubuntu 20.04
Linux 5.4.0-54-generic
mesa 20.3.0-rc2
AMD Radeon RX 5700
AMD Ryzen 5 3600

Kkedodrill 2020-12-05 github

There is a linux_1.14 branch available in SteamVR as a temporary solution to this problem.

@lostgoat Could I ask what are the details for the system that Valve uses for testing SteamVR on Linux? It has been awhile since there has been a Linux update, specifically for performance. Are there things specifically preventing Valve from getting SteamVR working with good performance (close to / matching Windows) on Linux?

Aanadon 2021-01-12 github

I'm hitting something that may or may not be related.

https://gitlab.freedesktop.org/mesa/mesa/-/issues/4044

WWebFreak001 2021-01-12 github

@anadon what you are experiencing might rather be related to #377 (which for me happened because of different vulkan driver)

check the solutions I posted there to see if that fixes it for you

Aanadon 2021-01-12 github

@WebFreak001 Nope, different pattern in the specific details for visual corruption, and much different characteristics around how problems emerge. I'm guessing for yours there is an implementation error in Vulkan in your driver. I think for mine it has to do with the memory allocater and/or use before initialization and/or erroneous overwrite of allocated memory. My preview also differs from what I see, where your steam preview matches what you see. I've already fixed a conflicting drivers issues that was amdgpu-PRO related a few weeks ago, which I needed to do to get VR working at all.

SsteffenWi 2021-01-21 github

Would like to confirm that doing this fixes the 'wobbly'-ness as stated initially by @mcoffin

SteamVR version 1.15.19
Distribution: Arch Linux
Kernel 5.10.8
Mesa 20.3.3
AMD Radeon RX 5700 XT
AMD Ryzen 5 2600

Oominitay 2021-01-21 github

Has been stated at the top of the thread. Unfortunately, we are best of sticking around with 1.14 until Valve fixes this.

SsteffenWi 2021-01-22 github

Has been stated at the top of the thread. Unfortunately, we are best of sticking around with 1.14 until Valve fixes this.

Yes, though while mcoffin wrote what caused it, it wasn't explained how to turn it off. The source that I found and linked is also bit old and I was not certain that it would still work.

CChristophHaag 2021-02-06 github

Because it has not been mentioned: This affects OpenXR quad layers too.

Hhappysmash27 2021-02-26 github

I get this issue too. Initially I created a separate issue for it, but it turns out it's the same as this one.

  • Distribution: Gentoo Linux, kernel 5.9.14-gentoo with the newest Mesa drivers pulled from Git and LXDE.
  • SteamVR version: 1.16.6
  • Steam client version: Dec 20 2020, 23:07:25
  • Opted into Steam client beta?: For SteamVR, yes. If you mean Steam itself, no.
  • Graphics driver version: [run nvidia-settings or vulkaninfo | grep driverInfo:
    Mesa 21.0.0-devel (git-ecac89b732) (ACO)
  • Gist for SteamVR System Information: https://gist.github.com/happysmash27/b0365a9032d5a2ca2c68517c990394b0

I also recorded the issue through the lenses of my HMD: https://happysmash27.me/Upload/Screenshots/Videos/Bugs/SteamVR/VID_20210204_050848.webm

Mmcoffin 2021-03-10 github

Just to chime in on this one guys, in my experience of only about 6months and ~1300hrs of usage, I've seen that the asynchronous reprojection implementation in SteamVR on Linux is mostly unusable, especially when compared to the stability of the performance achievable with some manual resolution tuning and just keeping it disabled.

While I originally disabled it to fix this issues, I've seen many other performance benefits to just keeping it off, and I'd recommend other performance-sensitive folks who's hardware isn't going to fall behind do the same for now. There's little reason to stay on 1.14 just to save the ability to use a feature that doesn't quite cut it to begin with.

Oominitay 2021-03-10 github

Just to chime in on this one guys, in my experience of only about 6months and ~1300hrs of usage, I've seen that the asynchronous reprojection implementation in SteamVR on Linux is mostly unusable, especially when compared to the stability of the performance achievable with some manual resolution tuning and just keeping it disabled.

While I originally disabled it to fix this issues, I've seen many other performance benefits to just keeping it off, and I'd recommend other performance-sensitive folks who's hardware isn't going to fall behind do the same for now. There's little reason to stay on 1.14 just to save the ability to use a feature that doesn't quite cut it to begin with.

I'll try that out then, since my hardware is on the low end of VR users (RX 580 8GB, R5 1600X). Thank you for telling us all!

ZZamundaaa 2021-03-10 github

Async Reprojection causes problems because of #269. If you have low end hardware then you're likely gonna be throttling most of time anyways, then disabling it isn't a good idea (I can't have it off in Boneworks or Blade&Sorcery for example, with a 5700XT. They just stutter too much without it).
For stuff like Beat Saber that always runs at max refresh rate though it does improve the experience a good bit.

Dduckbytes 2021-03-10 github

I too experience a lot of headache inducing stuttering with a 5700XT if async is disabled. I'm not so keen to reduce the resolution because it makes it difficult to read text. Are there other tweaks that can make disabling async usable or is reducing resolution the only way?

Mmcoffin 2021-03-10 github

I'm currently running a 6900XT, but I spent the last year on the 5700XT @duckbytes and @Zamundaaa. I found a few tricks that DRASTICALLY improved the experience.

Even though it may seem like you're throttling, sometimes, what's happening is that during the downtime between frames, the 5700XT will downclock to a lower power state, causing just enough lag on the next frame to temporarily trigger throttling before it clocks back up, does a few frames quickly, and the whole process repeats.

By setting the minimum sclk speed to something relatively high, I was able to completely stabilize my experience, even in ACC, which is a notoriously GPU-intensive game, especially in VR. I was able to run at 90Hz with no issue.

If you want to give it a whirl, here were the settings I was using all year. You may have to tweak some things due to silicon quality and cooler quality (I had a custom liquid loop, so lots of thermal mass to absorb spikey power consumption).

sclk_min: 1950
sclk_max: 2210
mclk_min: 700
mclk_max: 903 # my memory sucks. you might be able to do better than this
power_limit: 300000000 # This means 300W - at 300W often other limits are reached before the card hits this limit, which keeps the clocks as stable as they can be so long as your cooler can handle it
voltage_curve:
  - 750mV @ 800MHz
  - 912mV @ 1450MHz
  - 1230mV @ 2210MHz

Note that the voltage curve can still have points on the curve below the actual minimum sclk frequency, which makes it easier to adapt.

In some titles, I had similar problems on the CPU side, but I'm not as familiar as I didn't write the overclocking support on that side, so for those titles I just settled for setting max all-core frequencies, which worked out... okayish on my 3960X

sudo cpupower frequency-set -g performance

Some tools I made throughout this process (in addition to writing the reclocking support for amdgpu):

  • fanctl - powerful rule/curve based userspace fan controller. Think of it as an actually usable replacement for fancontrol from lm_sensors. Example config that has been rock solid for me: fanctl.yml.
  • amdgpu-smu-od - tool I use to automatically set GPU configuration from YAML files instead of having to do it manually with sysfs all the time. I'll attach an example config file. superclock.yml.

If you still have issues, please reach out. I spent tons of time on this over the last year, and good performance is possible. The biggest gains I saw were from drastically increasing the minimum shader clock (to prevent the downclocking), so I'd start there if I were you.

Hhappysmash27 2021-03-11 github

Which would be a good way to monitor my GPU clock to see if I may also be having a similar issue on my RX 480?

Hhappysmash27 2021-03-11 github

Never mind; radeon-profile (https://github.com/marazmista/radeon-profile) seems to work pretty well, and was in my distribution's repositories.

Dduckbytes 2021-03-12 github

Thanks @mcoffin for the ideas. I want to try out these configurations but I'm hesitant because I don't know enough about overclocking to know if these are suitable for my GPU or aren't going to cause damage. Should I look up frequencies for my particular model or reduce some of the values to account for only having air cooling? The specific card I have is MSI Radeon RX 5700 XT MECH OC 8GB.

I did find some options in radeon-profile that let me fix the frequency under Overclock > Manual frequency control. I can set that to only use the highest frequency (2100, 875 on there by default) by unchecking the other options. Is that more or less doing the same thing and would help with the downclocking bug you mention? Or is there more to it?

Thanks again!

Mmcoffin 2021-03-13 github

I want to try out these configurations but I'm hesitant because I don't know enough about overclocking to know if these are suitable for my GPU or aren't going to cause damage

  1. You're (probably) not going to damage your GPU by overclocking, you'd likely just make it crash. The potential for damage only really comes from running really hot for extended periods of time. As long as you're monitoring Tjunction (and with that MSI card, which has a known memory cooling issue, Tmem) and making sure they're somewhat sane, you should be fine.
  2. You don't have to push the clocks as far as I did to solve the issue. Keeping the default setting for your card for the the slck max and setting the min sclk value to something like sclk_max * 0.8 should cut it.

I don't use radeon-profile as the differences between the way various generations of cards manage power states and clocks is so widely variant. The biggest changes were from the 3/4/500 cards to the 5000-series (polaris -> navi).

Here is a tool I just now wrote for you @duckbytes. If you run it, it won't touch the voltages or anything, just set the minimum sclk to a percentage of the maximum sclk value (default: 85%). the -p option can be used to also set a power limit (in watts). Check the readme (or run amd-vr-clocks.sh -h) for more info, but here's an example usage. The -r flag tells the script to reset the card's clock speeds to whatever was default in the card's VBIOS at boot time.

This script should be pretty safe as it won't go beyond what the card is already allowed to boost to (except for increasing power limit if you choose to do so). If you're gonna set a higher power limit, just make sure you keep an eye on the temperatures when you do to make sure it's sane. You don't really want Tjunction to be over 90C that often (though I think AMD says that it's fine to run the 5700XT with Tjmax all the way up to ~98C). With my liquid cooling and a 300W power limit, my Tjunction stays around 73C, but ymmv with the default air cooler (especially on that MSI card).

To summarize

  1. try it without the -p flag. See how that goes. This is super duper uber safe.
  2. Try it with the -p flag to set a higher power limit, but keep an eye on temps. power limit will still obey card VBIOS OD limits, so this is pretty safe so long as you take notice if it's running insanely hot (unlikely)
  3. You can also override the card's default maximum power limit by using custom powerplay tables to overwrite the VBIOS-provided powerplay table. This is advanced, and this is where you get dangerous, so I didn't include instructions for this, but if you want to take the plunge, then - amdgpu-smu-od + upp + just the registry files from the igor's lab link in this article to get the base64 of some modded powerplay tables (or create one yourself with upp after modifying your dumped pptable from /sys/class/drm/cardX/device/pp_table).
# set min sclk to 85% of max sclk, and 210W power limit
./amd-vr-clocks.sh -p 210
# reset clocks to whatever is in the card's VBIOS, and set power limit to 180W
./amd-vr-clocks.sh -r -p 180
# set min sclk to 85% of max sclk on device at /sys/class/drm/card1/device, without touching power limit
./amd-vr-clocks.sh -d /sys/class/drm/card1/device
# reset clocks on device at /sys/class/drm/card1/device without touching power limit
./amd-vr-clocks.sh -d /sys/class/drm/card1/device -r

EDIT: forgot the link - mcoffin/amd-vr-clocks

@duckbytes let me know how that goes for you, eh?

Usage

Usage:  [-d /path/to/device] [-i DEVICE_INDEX] [-v] [-y] [-p POWER_LIMIT_IN_WATTS] [-s SCALE_PERCENT] [-r] [-h]

Flags:
	-d - device_path (default: /sys/class/drm/card0/device)
	-i - device index
	-v - increase output verbosity
	-y - do not ask for confirmation
	-p - power limit to set (default: none)
	-s - percentage to scale sclk_max when deriving sclk_min (default: 85)
	-r - reset clocks to original settings instead of setting VR mode
	-h - print this help text and exit
Dduckbytes 2021-03-14 github

@mcoffin thanks so much for making the tool for us. Really good of you.

It works great for me after adding amdgpu.ppfeaturemask=0xffffffff to my boot options.

I tried out some worlds in VRChat. It is really decently smooth at the moment. Heavier worlds are still too much without async reprojection, but I can comfortably be in more chilled worlds. I can use xrdesktop again now too since it was failing with SteamVR 1.14 but not the latest version.

Thanks again for the help.

Jjp7677 2021-03-14 github

I'm on an Nvidia GTX 1080. I figured that having Ferals Gamemode being active/enabled while running SteamVR made a lot of difference on my Fedora system here. No idea though what the effects are when running an AMD card, but you might give it a shot too (if you haven't already).

WWebFreak001 2021-04-27 github

it seems like in latest SteamVR beta (1.17.6) and stable (1.16.10) there is no async reprojection setting anymore, however the wobble issue still persists.

For workaround see #227

SsteffenWi 2021-04-28 github

You don't really want Tjunction to be over 90C that often (though I think AMD says that it's fine to run the 5700XT with Tjmax all the way up to ~98C). With my liquid cooling and a 300W power limit, my Tjunction stays around 73C, but ymmv with the default air cooler (especially on that MSI card).

@mcoffin just for your information, I have a MSI 5700XT GAMING X and junction temperature is around 100°C during most games and in some cases sits 'comfortably' at 105°C. The maximum I've seen is 112°C, though only for a second or so. AMD themselves have stated that junction sensor reporting 110°C is "within specs" in this blogpost.

@WebFreak001 yeah the setting has been gone for a while, I think this is the fourth time that someone, including myself, linked that workaround in here. I wish Valve would just fix this wobbly-mess so that this issue can be closed.

Mmcoffin 2021-06-15 github

@mcoffin just for your information, I have a MSI 5700XT GAMING X and junction temperature is around 100°C during most games and in some cases sits 'comfortably' at 105°C. The maximum I've seen is 112°C, though only for a second or so. AMD themselves have stated that junction sensor reporting 110°C is "within specs" in this blogpost.

@steffenWi - Check out the fan curve controller I wrote to get those under control! - fanctl. Been working well for me on my 6900XT (not water cooled like my 5700XT) since I got it.

Ttingvarsson 2021-07-07 github

Now also visible with Nvidia since Async support in 1.18.6 (and r470 driver).
system info

"Goes away" by removing CAP_SYS_NICE+ep or completely disabling async with:

{
   "steamvr" : {
      "enableLinuxVulkanAsync" : false
   }
}

PS; what's the actual drawback of not having the CAP? implicit disabling Async and/or other impacts?

Ppodiki 2021-07-10 github

I've also noticed this on AMD (6700 XT, Mesa 21.1.4/git, SteamVR beta) and can confirm either disabling async or reversing setcap (as in #417) or just not fixing setcap never happening (as per #234) prevents the wobble/jitter.

Mmcoffin 2021-07-13 github

either disabling async or reversing setcap (as in #417) or just not fixing setcap never happening (as per #234) prevents the wobble/jitter.

This is very interesting that this resolves the issue. Do you have any info that you've dug into as to why that would resolve the problem? I'll have to try this out later. Even though I don't use async anyways since it performs pretty poorly in general, if this really does resolve the issue, then it would be very interesting to hear about why that resolves it, as it doesn't seem to make sense to me.

One would think that setting the niceness of a few of the SteamVR processes to a lower value would only help performance rather than hurt it, except if that application is eating up so much CPU time that other processes can't get any time. In my case, that definitely isn't the case running on a 24-core 48-thread CPU, so maybe there's something strange at play with the cross-process synchronization around memory access for the buffers used for the overlays while async reprojection is in use.

Like I said, I don't use async, but I am interested to hear more from you about how you found out that that would work.

For others here, @podiki is stating that if SteamVR is unable to set it's processes as higher priority than standard (by setting the niceness values lower), then it resolves this overlay jittering. SteamVR, on initial setup, sets the file capabilities of SteamVR to allow it to change the niceness of other processes without being a superuser. Apparently, removing this capability causes the niceness-setting to soft-fail, but SteamVR will continue anyways, and apparently that resolves this problem. It's interesting, though, that giving SteamVR processes a higher priority would cause a performance degradation.

If you're just a new Linux SteamVR user seeing this issue, as of July 2021 (time of writing), I would still recommend disabling asynchronous reprojection (setting "Use legacy reprojection mode" to true), and just manually tuning the resolution of each game on a case-by-case basis to get the best image quality you can, while still not experiencing any performance issues. YMMV on lower-end hardware, but if you have graphics hardware that you know can handle the task at hand, that's definitely the best way forward for the time being.

Mmcoffin 2021-07-13 github

what's the actual drawback of not having the CAP? implicit disabling Async and/or other impacts?

@tingvarsson That file capability allows SteamVR to set the niceness value of some of the SteamVR processes lower than the default, effectively giving them a higher "priority" for the CPU scheduler. Normally, you would have to be a privileged user to do this, but setting that file capability allows SteamVR to perform that action without being run as a privileged user.

The drawback to that is that SteamVR won't run at a higher priority. It is very strange though, that running at what is effectively a lower priority would cause an uplift in performance there, so I think it's likely that there's some kind of weirdness going on with the cross-process synchronization primitives for overlays, but that's just a theory since I can't see in to the internals of SteamVR much.

CChrisJAllan 2021-07-13 github

The async reprojection feature relies on having high-priority privileges, if SteamVR doesn't have them it turns the feature off, which ends up giving a better experience because the feature is broken.

Annoyingly, the reverse is also true: SteamVR will only use high-priority Vulkan queues if async reprojection is on and working. I guess vrcompositor using high-priority queues wouldn't make much difference normally.

Kkisak-valve maintainer 2021-07-13 github

Hello @mcoffin, the CAP also controls access to high priority compute queues in amdgpu and the new nVidia driver. Vrcompositor needs that so that it can get the GPU to prioritize the late submission to update the reprojected rendering before the next vblank over the game which is in the middle of its render workload. That is what we actually care about with CAP_SYS_NICE.

Ppodiki 2021-07-13 github

Thanks for the followups @mcoffin, @ChrisJAllan, and @kisak-valve. Seems like there is a timing/priority issue involved with async, as in #230 which is maybe what we are seeing here too?

CChrisJAllan 2021-07-13 github

Most likely. With async reprojection, SteamVR tries to guess when an frame is going to be late and tries to generate a new frame based off of the previous frame and the headset movement. A few possibilities on what's going wrong:

  1. SteamVR could be bad at guessing when a frame will be late and takes over too often
  2. Generating a new frame could take too long
  3. It could take too long for the GPU to switch back and forth between rendering the game and rendering guesses

The actual frame rate tanks, and you're left with a lot of garbage guessed frames.

ZZamundaaa 2021-07-13 github

@ChrisJAllan https://github.com/ValveSoftware/SteamVR-for-Linux/issues/269 is about the first point you mention, SteamVR fails with frame prediction - not only predicting a slow frame too often, but just randomly wrong in both directions.

If it would take too long to generate the frame then everything would lag behind the view a bit, which is not what's happening.
Same with the GPU taking to long to switch between rendering, the whole view would lag behind, not just part of it.

Most likely SteamVRs position prediction for overlays is different than for the rest and uses wrong timings, miscalculating the predicted movement, similar to https://github.com/ValveSoftware/SteamVR-for-Linux/issues/226 (where the driver reported garbage timings).

However that, too, just a guessing game and is not really useful. Someone from the SteamVR development team has too look into why their closed-source code doesn't work.

Mmcoffin 2021-07-14 github

[snip] That is what we actually care about with CAP_SYS_NICE.

Thanks for the clarification @kisak-valve! That actually makes sense as to why those two are related. I was beginning to think I was nuts. I didn't know that the high priority queues required elevated access. I learn something new every day. Accordingly. @tingvarsson, you should probably ignore pretty much all of what I wrote above, as I was flat wrong! The good news is, it now at least makes sense why that fix of yours is helping.

SSpookySkeletons 2021-08-04 github

Is clock_gettime proper for motion smoothing estimates?
Could blocking updates of the clock_gettime structures be causing the jitter frames?

SSpookySkeletons 2021-08-04 github

Uh, echoing current clocksource between tsc, acpi_pm, and hpet in sysfs creates VERY different performance scenarios. Is this normal behavior?

EDIT:
echo 'hpet' > /sys/devices/system/clocksource/clocksource0/current_clocksource
Massively degrades frametimes

EDIT2: Reducing your kernel's CONFIG_HZ to 100 significantly reduces the headstrain and probably how far the async frames get mis-sampled.

SSpookySkeletons 2021-08-10 github

Found the issue I think:

Steamvr relies on clock_gettime and clock_nanosleep to estimate frames.
clock_gettime is used modestly however clock_nanosleep fills a majority of the strace and both of them are using the clock_t CLOCK_MONOTONIC and CLOCK_REALTIME which rely more often on the hpet system clock to generate HIGH LATENCY HIGH PRECISION time requests that load the CPU unnecessarily and arrive far too late for motion estimation.

The correct clock_t enums to use are CLOCK_MONOTONIC_COARSE and CLOCK_REALTIME_COARSE as they do not have to query the hardware for the time every single time a thread wants to sleep or check the time.

Three ways of fixing:
User: Override the kernel's syscalls by replace incoming clock_t enums with the correct COARSE versions with a .so hack

User: Bin mod steamvr to send the correct syscalls for sleep and time checks.

Valve:
Use clock_nanosleep(CLOCK_REALTIME_COARSE/CLOCK_MONOTONIC_COARSE...
and
clock_gettime(CLOCK_REALTIME_COARSE/CLOCK_MONOTONIC_COARSE...
in steamvr instead.

SSpookySkeletons 2021-08-10 github
// redirect_gettime.c
#include <dlfcn.h>
#include <time.h>

static int(*real_clock_gettime)(clockid_t, struct timespec *) = NULL;

static int(*real_clock_nanosleep)(clockid_t, int, const struct timespec *, struct timespec *) = NULL;

int clock_gettime(clockid_t clk_id, struct timespec *tp) {
    if (real_clock_gettime == NULL) {
        real_clock_gettime = dlsym(RTLD_NEXT, "clock_gettime");
    }
    switch (clk_id) {
    case CLOCK_MONOTONIC:
        clk_id = CLOCK_MONOTONIC_COARSE;
        break;
    case CLOCK_REALTIME:
        clk_id = CLOCK_REALTIME_COARSE;
        break;
    }
    return real_clock_gettime(clk_id, tp);
}

int clock_nanosleep(clockid_t clk_id, int flags, const struct timespec *req, struct timespec *rem) {
    if (real_clock_nanosleep == NULL) {
        real_clock_nanosleep = dlsym(RTLD_NEXT, "clock_nanosleep");
    }
    switch (clk_id) {
    case CLOCK_MONOTONIC:
        clk_id = CLOCK_MONOTONIC_COARSE;
        break;
    case CLOCK_REALTIME:
        clk_id = CLOCK_REALTIME_COARSE;
        break;
    }
    return real_clock_nanosleep(clk_id, flags, req, rem);
}

//gcc -D_GNU_SOURCE -fPIC -shared -o redirect_gettime.so redirect_gettime.c -ldl
//LD_PRELOAD=/home/$USER/redirect_gettime.so
//Preload must be added somewhere where steamvr gets its environment variables.

This should do the trick. Not at my station yet so can't test but hopefully someone else can until I get home... This code might be wrong and is derived from https://github.com/pytorch/pytorch/issues/3390

WWebFreak001 2021-08-10 github

original issue (overlay stutters) not fixed with that code above, also doesn't feel like there are any improvements with the frame timing with async reprojection being off. (still slightly noticable delay)

Also there is a small issue in the function pointer type (should use int instead of int * for clock_nanosleep)

SSpookySkeletons 2021-08-10 github

@WebFreak001 I took a look, did you try inserting your LD_PRELOAD in /home/$USER/.local/share/Steam/steamapps/common/SteamVR/bin/vrenv.sh

Corrected the pointer, fixed the warning.

WWebFreak001 2021-08-10 github

I didn't try inserting it into vrenv before, but it seems it doesn't do any changes with async reprojection being either on or off either.

Is your fix for this supposed to fix the wobbly overlays as described in the issue with async reprojection being on or the slightly delayed frames (possibly giving headaches) when async reprojection is off?

Also couldn't reproduce noticably degraded frametime changing between hpet and tsc clock source as in your comment before. (using linux-zen on archlinux) Didn't try changing my kernel CONFIG_HZ (my value is 1000)

Headset: Valve Index

SSpookySkeletons 2021-08-10 github

You can fix the overlay latency here with this kernel patch https://github.com/ValveSoftware/SteamVR-for-Linux/issues/21#issuecomment-808969247

Trying to address the async jitter with this preload hack

SSpookySkeletons 2021-08-12 github

Bad news catching and rewriting the time syscalls with more performant coarse ones didn't seem to help any. So it's probably steam's motion algo doing it.

Best way I can describe the issue with async reprojection is a divergence where the location of true image location to both the sides of the HMD motion with timestep. With 90hz being 11.1ms refresh I only really see it begin to diverge above 16ms which means it performs almost normally up until that point.
The divergence appears equally simultaneously in either eye but alternates per frame rendered to the headset.
Spikes in frametime above that 16ms threshold for my headset make the divergence immediately apparent as sharp jitters in image location and produces the sickening wobble and thrashing motions as you turn your head.

The whole motion estimate could be incorrect to start with and legacy, which cuts out just half the frames, is removing half the divergence but still produces one half of the rotation slip and makes it still feel laggy or floaty without the jitter.

Yyshui 2021-11-18 github

Really wish this can be fixed. On the one hand async reprojection made some games much smoother for me. For example, without it VRChat is a jittery mess. But OTOH enabling it made the dashboard, chaperone and the "empty" world wobbly, along with the corrupted frames.

Is there anything we can do to help the Valve devs here?

TTheDrawingCoder-Gamer 2021-12-28 github

I also have this issue, and it only affects the overlay (which also includes the "loading" world):
Distro: EndeavourOS (rolling, 5.15.11-arch2-1)
SteamVR Version: 1.21.4 beta
Steam client version: Built Dec. 16 2021, at 22:39:26
Opted into beta: Yes
Graphics Driver Version: 470.90
Gist for System Info: https://gist.github.com/TheDrawingCoder-Gamer/0d045d31816a557031e615e08ff27ecf

BBitwolfies 2022-01-22 github

Still happens as of steamvr beta 1.21.6, Using EndeavourOS KDE, along with Nvidia-DKMS 495

Bbblacher 2022-02-22 github

Also having this issue, the "wobbly" overlay/chaperone drives me crazy. Really hope this will be fixed in a future update.

EEquinoxTheGryph 2022-04-12 github

I can confirm this issue as well here. Disabling async does resolve the menu issues, but in turn makes turning your head in VRChat (only game I tested so far, just recently got this headset) very stuttery. Downgrading SteamVR to 1.14 results in VRChat not being able to launch, so I have to chose if I want to disable async or not in the current version.

Info:
VR: HTC Vive (headset, controllers and base stations)
SteamVR: Version 1.21.12 (1647034158)
Operating System: Manjaro Linux
KDE Plasma Version: 5.24.3
KDE Frameworks Version: 5.91.0
Qt Version: 5.15.3
Kernel Version: 5.16.14-1-MANJARO (64-bit)
Graphics Platform: X11
Processors: 12 × AMD Ryzen 5 2600X Six-Core Processor
Memory: 15.6 GiB of RAM
Graphics Processor: AMD Radeon RX 580 Series

Kkisak-valve maintainer 2022-04-12 github

Mildly off-topic, but if you're testing Proton with the SteamVR linux_v1.14 legacy branch, then the game needs to be set to run with Proton 5.13 or older, and not need Steamworks or OpenVR interfaces that are newer than what was available in February 2021 when Proton 5.13-6 was released. It's generally expected that SteamVR always provides the same or newer OpenVR API version than what's used by VR games and this legacy branch has unfortunately stuck around long enough for that caveat to appear.

Specifically for VRChat, this situation was previously pondered on its compatibility report with comments starting around https://github.com/ValveSoftware/Proton/issues/1199#issuecomment-889498053.

MMeister1593 2022-04-20 github

Hello
i recently tested stable and beta versions of steamvr and had some sort of delayed/wobbly output, which makes playing some games (like beat saber) pretty much unplayable. This is all expected and i knew this happens.
On 1.14 beta branch however, it is much better. So, i have AMD_DEBUG=zerovram to remove issues with weird artifacts around my viewport and on that version (plus proton 5.13-6), beat saber works quite well and not wobbly at all. But after some time (or more like, actions), in my case - one song passed - image suddenly "jumps" (once) and it becomes wobbly. After this, there is 50/50 chance that it will snap back and stop wobble, but it is mostly random.
Overlay didn't work at all, had to launch game through desktop without hmd
I use htc vive, rx 6600 xt gpu, steam flatpak (steamvr couldn't apply something at startup and always asks about it), on fedora kinoite
Edit:
Forgot to mention that i use CoreCtrl on the background with high power profile always on because automatic profile doesn't seem to recognize gaming in general and clocks poorly

FFl0ux 2022-05-18 github

Since the latest Manjaro upgrade, I do not have this problem anymore !

Kernel 5.17.6-1-MANJARO
Mesa 22.0.3

MMeister1593 2022-07-10 github

Replying to https://github.com/ValveSoftware/SteamVR-for-Linux/issues/395#issuecomment-1104109946

I think something is happening there on steamvr side, some logic does this...
when i had nice frametime bellow 11.1 ms (90hz) in beat saber (proton 6.3-8) i had perfect display and pretty much nothing wobbled at all. Very pleasant picture. But when i opened steamvr dashboard couple of times, frametime jumped each time and at once i had wobble effect. To remove that effect i had to again open dashboard couple of times and it will snap back into perfect display. Not sure if it's relevant, at the time i had desktop capture working (it's working only like, 30% of times out of all launches)
This is on latest steamvr beta branch, gnome xorg, fedora silverblue 36 with xanmod-edge kernel , steam and steamvr inside arch distrobox container.

I did also test out Monado + libsurvive + OpenComposite, overall not suitable replacement (for vive, tracking is not as good on regular valve drivers) but a good playground for testing out these theories. And aside from shutter, image over-saturation, incomplete bindings and barely any vr game working, it did not have that crazy wobble effect.

MMeister1593 2022-07-12 github

Okay not sure if that will help or not people, but: this is very likely related to usb and linkbox kindof adds even more latency.
For me connecting usb without linkbox directly significantly reduced that latency, almost to the point where it is on-par with windows (almost! about 70-80% out of what on windows)
Edit:
This however did not fix wobble effect...
It's likely more related to #21

SSpookySkeletons 2022-09-10 github

@kisak-valve
Found the crux of this issue, will take a little explaining...
https://www.pcmag.com/news/steamvr-motion-smoothing-reduces-lag-on-lower-end-pcs
https://steamcommunity.com/games/250820/announcements/detail/1696061565016280495
Comes from an official explanation of one of the valve developer.

It turns out that steamvr saves the last two frames in a buffer or at least feeds them into the encode engine of the GPU and draws out a motion vector field of the framebuffer between the last two frames and then takes this image and smears the very last frame generated to produce a frame for the vsync.

The behavior in the case of the motion vector being zero is simply to 50:50 the images as overlayed directly atop one another. This works for UI elements held in hand in the direct foreground that match the rotation of your face even on windows, it does not feel acceptable for background elements and causes split vision. THIS IS A HUGE PROBLEM ON LINUX

VAAPI/ VDPAU video APIs have always been pure jank on linux and valve did not bother to implement a solution to properly talk to the GPU to determine this motion vector field so it just feeds zero into everything, the resulting image is full of zero field info and just defaults to a 50:50 fade between the last two frames of the image which is incredibly nauseating.

Three solutions:

Go into the binary and NOOP or hard disable the function attempting to insert these hallucinated frames. Noticed words such as "halucination" around these function names. This will create a raw ATW simple motion smoothing where the image can look choppy but the viewport is smooth, much preferable.
1.14 does this and is why it looks okay!

Expose the necessary video API through the task of wiring up (ffmpeg?) an entire video encode/ decode setup to draw out vector fields from these two last images and then feed the vector field back into the hallucination functions to draw the next frame properly.

Change the behavior of the 50:50 and CUT OUT A FRAME. At least using the very latest frame will have the effect as solution 1 instead of a transparent blend.

😮‍💨👃

PLEASE VALVE FIX, WE KNOW THE ISSUE HERE
Wrap ffmpeg??? or something in the current API if you have to or at least give us the source code so we can do the job ourselves, OpenVR is in a truly rotten state and even the windows version could benefit from some eyes and contributions!

I realize your super secret headset is in development, and will natively ship linux as standalone and will require you to fix this as time permits, but its simply been years and without the necessary transparency this has been a bleeding mess thus far.

It would be nice if OpenXR was the standard, Monado was free range, and we didn't have to deal, but OpenVR is here to stay as a matter of the sheer amount of applications run atop it already so we would like some source pls!

SSpookySkeletons 2022-09-10 github

As a demonstration:
Have your application framerate capped artificially less than not equal to half the HMD refresh, then force the motion vector field to all zeros and watch what happens.
This will work under both linux and windows if you need a demonstration of how bad this issue is and how to resolve it on your end.

This has been present since version <1.14 and is in dire need of a ifndef(__LINUX) statement to repair it.

SSpookySkeletons 2022-09-25 github

Doing more digging, it looks like steamvr may handle this kind of motion interp using a dx shader for windows and a SPIR-V shader for linux!
SteamVR/resources/shaders/vulkan/
frame_hallucination_ps.spv
frame_hallucination_vs.spv

Anybody want to try deleting some of these shaders & more to see if it helps with FPS under half?
https://github.com/ValveSoftware/SteamVR-for-Linux/issues/430

If all this is handled in shaders it could be most of the shaders are just written incorrectly same as the ones rendering the OBJ files have depth buffer issues that invert the faces.

@okawo80085
Are you still willing to take a peek at any of these shaders?

Ookawo80085 2022-09-26 github

@okawo80085
Are you still willing to take a peek at any of these shaders?

As soon as i get a free minute i'll check it out

Ssanteri3700 2022-09-26 github

@SpookySkeletons I tested running native SteamVR Beta 1.24.3 with asynchronous reprojection enabled and the mentioned files moved elsewhere (and some others on a second attempt). That didn't seem to affect anything. The overlay still works and is wobbly/jittery.

I'm willing to do more testing on AMD and Nvidia GPUs if needed (btw there is no depth buffer issues on Nvidia with proprietary drivers as far as my experience goes).

Deleted vulkan shaders:

  • frame_hallucination_ps.spv (only these were deleted on the first attempt)
  • frame_hallucination_vs.spv (only these were deleted on the first attempt)
  • motion_attenuation_ps.spv
  • motion_attenuation_vs.spv
  • motion_filter_attenuation_ps.spv
  • motion_filter_blur_ps.spv
  • motion_filter_cost_ps.spv
  • motion_filter_early_out_vs.spv
  • motion_filter_median_ps.spv
  • motion_filter_vs.spv
  • motion_smoothing_debug_ps.spv
  • motion_smoothing_debug_vs.spv
  • motionvector_cost_cs.spv

Here are some of my specs:

  • HTC Vive Pro 2
  • AMD Radeon RX 5700 XT (VR power profile activated)
  • Arch Linux with Xfce4 (X11)
  • Linux 5.19.9 (modified, 1000Hz clock freq.)
  • Mesa 22.1.7 with RADV
Mmittorn 2022-10-19 github

@lostgoat

There is a linux_1.14 branch available in SteamVR as a temporary solution to this problem.

Is there any way to make new proton versions work with 1.14-1.15 correctly?

SSpookySkeletons 2022-12-15 github

Found it.
https://github.com/fholger/openvr_fsr/commit/779dac6c83a67ccda4239bae061798718a7deff0

SteamVR submits the result from the previous eye if the game uses the same texture to render both eyes sequentially.
This leads to a divergent double vision as you rotate your head. Notably this takes effect on reprojection under linux.

This can be fixed in the steamVR implementation. Just need to flag down valve for a fix here or have individual games implement it when they detect proton as a quick and dirty fix.
Scratch that this is baked into the reprojection code so valve will need to address the skewed offsets.

ZZarathustraDK 2022-12-22 github

@SpookySkeletons Are you saying that SteamVR reprojects frames from the left eye to the right and vice versa under some circumstances? If so that's quite a wild goose chase the reprojection-thread has been on for 1+ years, thinking it had something to do with delayed/premature frames being displayed to the correct eye.

But yeah, the particular offset experienced makes sense in that case. Theoretically this would mean that far away objects are less displaced than objects closer to you, no? As in switching between looking with the left/right eye will "move" a finger you're holding up in front of you further than some object you're focusing on on the far wall. If the problem was delayed/premature frames to the correct eye the reprojection offset would be uniform regardless of distance to the object.

RRinLovesYou 2023-05-20 github

Loving the radio silence, guys. Thanks valve.

Yyaomtc 2023-09-08 github

The artifacts in SteamVR (not overlays specifically) have now been fixed, see #588

I don't have any overlays set up at the moment (not sure how to, haven't used them before), can someone test to see if artifacts are fixed for overlays too?

Aamalon 2023-09-08 github

I can confirm on AMD RX 6700 XT, HTC vive, SteamVR 1.27.5:

  • SteamVR system menu no longer wobbles
  • SteamVR background no longer has glitchy green small square artifacts when GPU not clocked up with CoreCtrl
  • OpenXR quad composition layer (which I blindly presume is roughly analogous to IVROverlay) no longer wobbles (tested with Flightgear splash screen)
Kkisak-valve maintainer 2023-10-21 github

Closing per the last couple comments.

SSpookySkeletons 2023-10-21 github

@kisak-valve Underlying issue of the overlay itself actually seems to be much improved, thankyou. However I would like to add: to address other such issues under heavy load, might I make a recommendation that you verify that linux's GPU preemption is functioning as expected?

Proton versions

Upstream links