Have you tried this on an actual Windows system? As I recall, the dosbox config id shipped with the Doom titles was rather iffy. I suspect they expected most people to just grab the WADs and throw them at a native port.
This is likely due to a bug in the Steam game overlay. Can you try disabling the Steam overlay (either in the Steam client settings, or just for this game in its own settings) and see if that helps?