Graphics corruption after using Steam Overlay

BlitzMax Forums/BlitzMax Programming/Graphics corruption after using Steam Overlay

Grey Alien(Posted 2015) [#1]
Hi all, I recently noticed that if I run my game in Steam and any kind of Steam overlay occurs that I get graphics corruption (e.g. the intial Steam popup, or an achievement popup, or if you use Shift+Tab to bring up then hide the Steam Overlay).

https://www.dropbox.com/s/shm83psz8n7yaca/RSCorruption.jpg?dl=0 (yes I know it's a jpg but the corruption is still very evident)

On the left is how it normally looks, and on the right is after a Steam overlay incident. Notice how the fonts look blocky and so do the button rims. It's as if filteredtexture is being turned off by the Steam overlay!

Also worth noting is that it ONLY happens if I use the DX9 driver, it doesn't happen with OpenGL. If I recreate the graphics window it goes away (like by going to windowed mode then full screen again), and also Alt+tabbing fixes it. I'm guessing this is because all the textures get refreshed in VRAM.

Now I know that this doesn't happen in (most) other games, so it must be Steam+BlitzMax combo. I've also googled this issue and heard about a few other games that suffer from it, and the solution is "turn off Steam overlay", which isn't a great solution really, especially as a player will get a bad first impression with that corruption and may not think to turn off the overlay.

Perhaps if it was possible to query the texture states and discover they were corrupt (or filtered image was turned off) I could refresh them somehow?

Any ideas for a fix? Thanks!


H&K(Posted 2015) [#2]
The only thing I can think to ask you, what mechanism exits for your program to KNOW that a steam overlay has occurred? (This might be what you are asking already sorry)

Obviously my solution is gonna be, let it happen them deal with it, where as I expect you want a - don't let it happen solution.


therevills(Posted 2015) [#3]
It looks like anti-alias has been turned off...

Heres a couple of threads I've quickly found:

http://steamcommunity.com/groups/airforsteam/discussions/1/617330406657138525/

http://forums.steampowered.com/forums/showthread.php?t=1970609

http://superuser.com/questions/753484/force-direct3d-anti-aliasing-in-a-direct3d-game


Grey Alien(Posted 2015) [#4]
@H&K There's no mechanism via Steam to let me know when that has occurred, but yeah I'm wondering if there's a way in BlitzMax to see if the textures have been corrupted or "dirty". I believe in Blitz3D there used to be something like that. I know that Mark wrote some code to reupload all the textures to the videocard after an alt+tab (DX surface lost or something) so I might be able to trigger that if I could detect they were corrupt.

@therevills I don't think so. I haven't turned on anti-aliasing on my video card, don't even know how to do that in code. Remember that older video cards didn't even have FSAA (if that's what you are talking about). The smoothness is simply the fact I'm loading in the textures with FILTEREDIMAGE in the LoadImage flags. It seems like Steam overlay is changing something in the way things are being rendered, and it looks like FILTEREDIMAGE stops working for some reason. In fact, here is someone talking about that: https://github.com/ValveSoftware/Source-1-Games/issues/1646

Possible workarounds are:
1) Once the game has loaded (and the steam popup has appeared and disappeared), recreate the graphics window before the title screen, or main game screen appears. This will at least clear the issue periodically. But it won't solve the problem if they used Shift+Tab to call up the overlay, or if they get an achievement popup, or I guess if they get a "your friend is playing X" popup (haven't tested that).

2) Start the game in OpenGL (there are no issues as far as I can tell). Also provide a game launcher to allow players to change driver and full screen/windowed before they launch the game (in case OpenGL fails as it can be less reliable). However, I should warn them to disable Steam overlay if they select the DX driver.

Has anyone made a nice game launcher like that (with MaxGUI) btw, and put it in the archives?


Grey Alien(Posted 2015) [#5]
Looks like there's a function that tells your code when it's running: SteamUtils()->IsOverlayEnabled()

So perhaps I could track that switching from True to False and then recreate the graphics window (or better still, just find a way to refresh all the textures in VRAM as that would take less time and wouldn't result in a momentary black screen)

But Skid doesn't have that in the steamstub.lib. Not sure how easy it would be to add, besides it would also mean recompiling the the .lib which I'm not set up to do.


Grey Alien(Posted 2015) [#6]
Also I spoke to someone who said if the filtering is being turned off, it should be safe to reenable it each frame with this:

 PDevice->SetSamplerState(0,D3DSAMP_MINFILTER,D3DTEXF_LINEAR);
 PDevice->SetSamplerState(0,D3DSAMP_MAGFILTER,D3DTEXF_LINEAR);

Though that would mean writing some C++ code or plugging that into the BlitzMax rendering code somewhere, but I've no idea where to start with that.


Grey Alien(Posted 2015) [#7]
Well, I believe I have fixed it with the help of Cliff Harris who knows his way around DX.

However, I'd love to hear if anyone thinks this fix is dodgy in any way:

Basically you need to add this to the bottom of d3d9max2d.bmx ResetDevice() just above _d3dDev.BeginScene

'JB
_d3dDev.SetSamplerState(0,D3DSAMP_MINFILTER,D3DTEXF_LINEAR)
_d3dDev.SetSamplerState(0,D3DSAMP_MAGFILTER,D3DTEXF_LINEAR)
_d3dDev.SetSamplerState(0,D3DSAMP_MIPFILTER,D3DTEXF_LINEAR)
'This is for the black line issue
_d3dDev.SetSamplerState(0,D3DSAMP_ADDRESSU,D3DTADDRESS_CLAMP)
_d3dDev.SetSamplerState(0,D3DSAMP_ADDRESSV,D3DTADDRESS_CLAMP)


What seems to be happening is the Steam overlay destroys the setting on each texture that specifies it it is a filtered image. It also appear to destroy the clamping.

However, the code above sets things up correctly via the SamplerState and seems to bypass the incorrect/destroyed setting on each individual texture.

I did try commenting out the if statement from the code below in TD3D9ImageFrame.Draw() so that the code in the middle gets called every draw, but for some reason that didn't work:

		If _texture<>_bound_texture
			_d3dDev.SetTexture 0,_texture
			_d3dDev.SetTextureStageState 0,D3DTSS_MAGFILTER,_magfilter
			_d3dDev.SetTextureStageState 0,D3DTSS_MINFILTER,_minfilter
			_d3dDev.SetTextureStageState 0,D3DTSS_MIPFILTER,_mipfilter
			_bound_texture=_texture
		EndIf


So possible side effects are: I guess if you really want to draw a texture without filtering, this code isn't for you, as now everything will be drawn with filtering.

Also that UV clamping code seem to work well but it could have weird side effects, I don't know.

WHY THE CLAMPING?

I forgot to say why I'm using the clamping code, well that's because after the overlay appear in full screen mode, I saw black lines on the edges of some textures (like the top of the options button texture, which has blank space around it) here:

https://www.dropbox.com/s/nlxvf2tphru7d9h/Blackline.jpg?dl=0

This is because in full screen mode, I'm no longer drawing textures at 1:1, I'm scaling everything up to the desktop resolution (via SetVirtualResolution), and normally this is fine. But for some reason the Steam overlay breaks it. Again it probably breaks the clamping on a per texture level, and the code fix enables clamping at the SamplerState level.

ALSO be aware that maybe Steam destroys OTHER texture settings that I am not repairing and simply have not discovered yet...

So this is my kludge. Does it seem sensible or a nightmare? Thx


Kryzon(Posted 2015) [#8]
SetTextureStageState does not have D3DTSS_MAGFILTER, D3DTSS_MINFILTER or D3DTSS_MIPFILTER types, these types don't exist:
https://msdn.microsoft.com/en-us/library/windows/desktop/bb172617(v=vs.85).aspx

Those new lines that you added to ResetDevice, add them to the ImageFrame Draw() code.
This will ensure those states are set when you draw an image, so a spontaneous Steam overlay won't change them.


Grey Alien(Posted 2015) [#9]
Weird, because those consts are in this list: https://msdn.microsoft.com/en-us/library/ms886612.aspx

They must work in normal BlitzMax operation because it's the only place that textures are set to be filtered or mipmapped, and I know that works (I used filtered textures for most of the graphics in my games)

It doesn't seem that I need to add those lines from ResetDevice() to draw because somehow setting them there fixes the issue and Steam Overlay doesn't break anything any more. I don't know why though.

However, maybe it would be better if I could add them to Draw() and ONLY set the ones relevant to the states of _magfilter, _minfilter and _mipfilter? Perhaps that's safer as then if something isn't supposed to be filtered it would not get drawn filtered.


Grey Alien(Posted 2015) [#10]
Also what is the difference between and SetTextureStageState and SetSamplerState? I don't even know.

I'm guessing SetTextureStageState is applied to the current texture (specified by _d3dDev.SetTexture 0,_texture) and that SetSamplerState just gets applied to all textures?


Kryzon(Posted 2015) [#11]
That documentation link you posted is for Direct3D 8.

When you press Shift + Tab to bring the Steam overlay while still inside the game, with that modified ResetDevice(), does it fix the issue?
If it does, you can probably leave it like that. No need to modify Draw().

In case you're interested about the Direct3D 9 API, the MSDN documentation is concise:

- https://msdn.microsoft.com/en-us/library/windows/desktop/bb174462(v=vs.85).aspx
- https://msdn.microsoft.com/en-us/library/windows/desktop/bb174456(v=vs.85).aspx


Grey Alien(Posted 2015) [#12]
Thanks @Kryzon: Yep, it does fix it just with the code in ResetDevice() so in theory I could leave it there, but what was bothering me is the original BlitzMax code uses SetTextureStageState within Draw() to set the filter/mipmap on a per texture basis. So in theory you could have a mixture of textures with different filtering required and using SetSamplerState just at the start of a frame will override any individual texture filter/mipmap right?

Also could calling _d3dDev.SetSamplerState(0,D3DSAMP_MIPFILTER,D3DTEXF_LINEAR) even though I haven't loaded in any textures with the mipmap flag be a problem? It does admittedly seem to work, so I guess it's safe.

Anyway, I'm wondering if a better/safer solution is to check _magfilter, _minfilter and _mipfilter, and then call the correct SetSamplerState every time I draw, or would that be too slow?

What do you think?

I'm also paranoid about what else Steam trashes when it shows the overlay. I know it messes up the uv clamping which I have a fix for above, but maybe it messes other stuff up too?

For example, ResetDevice() also sets these things:

		_d3dDev.SetRenderState D3DRS_ALPHAREF,$80
		_d3dDev.SetRenderState D3DRS_ALPHAFUNC,D3DCMP_GREATEREQUAL

		_d3dDev.SetRenderState D3DRS_ALPHATESTENABLE,False
		_d3dDev.SetRenderState D3DRS_ALPHABLENDENABLE,False
		_active_blend=SOLIDBLEND
		
		_d3dDev.SetRenderState D3DRS_LIGHTING,False
		_d3dDev.SetRenderState D3DRS_CULLMODE,D3DCULL_NONE	
		
		_d3dDev.SetTexture 0,Null
		_bound_texture=Null
		
		_d3dDev.SetFVF D3DFVF_XYZ|D3DFVF_DIFFUSE|D3DFVF_TEX1
		
		_d3dDev.SetTextureStageState 0,D3DTSS_COLORARG1,D3DTA_TEXTURE		
		_d3dDev.SetTextureStageState 0,D3DTSS_COLORARG2,D3DTA_DIFFUSE		
		_d3dDev.SetTextureStageState 0,D3DTSS_COLOROP,D3DTOP_SELECTARG2
		_d3dDev.SetTextureStageState 0,D3DTSS_ALPHAARG1,D3DTA_TEXTURE
		_d3dDev.SetTextureStageState 0,D3DTSS_ALPHAARG2,D3DTA_DIFFUSE
		_d3dDev.SetTextureStageState 0,D3DTSS_ALPHAOP,D3DTOP_SELECTARG2
		_texture_enabled=False


Maybe I should set those at the start of each FRAME instead?


Kryzon(Posted 2015) [#13]
It sets the filtering at each Draw because these are persistent states (set on texture unit zero, like a channel, used to draw all Max2D images) and one image may have a different filtering than the previously drawn image. The filtering mode is specified by the programmer at LoadImage.

If the ResetDevice modification fixes all possible cases of Steam overlay, it's better to leave things like that.
Modifying the module further as a preventive measure may introduce other problems.

Making changes to the texture unit state (like with SetSamplerState etc.) modifies only the texture unit (the "channel" used to draw the textures), it doesn't need to have a texture bound to that unit.


Grey Alien(Posted 2015) [#14]
Ah I see, OK thanks.

Well it may be that BRL want to consider this "fix" for Steam.


therevills(Posted 2015) [#15]
We are getting some weird graphical issues with Steam overlay.

Using GA's code on post #7 fixes the filtering issue, but in OpenGL we get this, when using Aspect Ratio code:



And with DirectX9 it doesnt clip correctly on the first display, if we change to windows and back to full screen it works:



Any ideas?


Grey Alien(Posted 2015) [#16]
.


therevills(Posted 2015) [#17]
I think this is happening due to my graphics card having triple buffering enabled...

I've done a work around by overriding the Cls command which will hurt performance a tiny bit:



And you can test with any BMax game, by adding a non-steam game to steam. Here the full test code using James' virtual aspect ratio code:
http://www.blitzmax.com/codearcs/codearcs.php?code=2879