Currently, steam in-home streaming is unable to stream some fullscreen applications (like dolphin emulator in vulkan mode) from gnome shell. This is not exclusive to steam, I detected the same behavior with ffmpeg's x11grab.

The symptoms are: streaming screen freezing after very few seconds.

Gnome documentation recommends to disable mutter's fullscreen unredirect when recording the desktop, as it avoids screen compositor redirection for fullscreen applications making some applications to look frozen when capturing X11. And it seems steam does not care about this quirk.

Disabling mutter's unredirect can be done from any language with gnome bindings (please note that this API has changed screen to display on gnome-shell 3.30) https://developer.gnome.org/meta/stable/meta-MetaCompositor.html#meta-disable-unredirect-for-screen or via gdbus, as shown below.

This will work for gnome-shell < 3.30

gdbus call \
    --session \
    --dest org.gnome.Shell \
    --object-path /org/gnome/Shell \
    --method org.gnome.Shell.Eval \
        "Meta.disable_unredirect_for_screen(global.screen);"

And the same for gnome-shell >=3.30

gdbus call \
    --session \
    --dest org.gnome.Shell \
    --object-path /org/gnome/Shell \
    --method org.gnome.Shell.Eval \
        "Meta.disable_unredirect_for_display(global.display);"