This issue is consistent with a driver bug, possibly with glamor. If you have the time and are willing, please also follow http://mesa3d.org/bugs.html and file a bug report over there as well if it has not been already.
It looks like this bug was already reported here, https://github.com/ValveSoftware/steam-for-linux/issues/2732
Closing as duplicate
The only case I can replicate without issue is when trying to install a game not available for the platform. It's happened in a few other cases, but I can't recall how.
The only way to close the dialog is to hit escape.
This is using the open source radeon driver. Have not tried with the amd provided driver.