ReadPixels

pantson

(Posted 2015) [#1]

Hi
I want to grab an area of the game screen every frame and display it altered on top of everything in the game.
I assumed I could use ReadPixels as this will copy an area of the render screen and then I can convert to image and the draw that image.
What I'm finding though is that ReadPixels is not only copying the pixels, but also flushing the render area, so the original render screen is no longer there. Thus making the actual game not display.

I have written the below example to show it in action. Hit the mouse to start grabbing the pixels and you will see the text disappear. I would expect the text to always display.

Strict 
Import mojo

Function Main:Int()
	New example()
	Return 0
End Function


Class example Extends App
	Field counter:Int
	Field grab:Int
	Field pixels:Int[10000]
	
	Method OnCreate:Int()
		counter=0
		grab = 0
		
		Return True
	End

	Method OnUpdate:Int()
		
		counter = counter + 1
		
		If MouseHit(MOUSE_LEFT)
			grab = 1 - grab
		End
		
		Return True
	End

	Method OnRender:Int()
		Cls 
		DrawText counter,20,20
		DrawText grab,20,40
		
		If grab=1
			' grab to tmp array
			ReadPixels(pixels,0,0,100,100)
		End
		Return True
	End
End

I've only had chance to try this on XNA, so it would be nice if anyone can see this happening in other targets

pantson

(Posted 2015) [#2]

for the time being, I've solved this by drawing everything twice.
once before the readpixels and once after the readpixels.

ImmutableOctet(SKNG)

(Posted 2015) [#3]

So, this seems to be a bug with the XNA target.

Here's a fun refactored version of your example:

Strict

Public

' Preprocessor related:
#DRAW_OLD_IMAGEDATA_ON_CAPTURE = True

#If TARGET = "xna"
	#READPIXELS_REDRAW_FIX = True
#End

' This is just for fun:
#EXAMPLE_CUSTOM_CLS_COLOR = False ' True

#EXAMPLE_COLORTYPE_RED = 0
#EXAMPLE_COLORTYPE_GREEN = 1
#EXAMPLE_COLORTYPE_BLUE = 2

#If Not EXAMPLE_CUSTOM_CLS_COLOR
	#EXAMPLE_GENERATED_COLORTYPE = EXAMPLE_COLORTYPE_BLUE ' EXAMPLE_COLORTYPE_RED
#End

#EXAMPLE_DISCONNECTED_UPTIME = True ' False

' Imports:
Import mojo

' Functions:
Function Main:Int()
	' Start the example.
	New Example()
	
	' Return the default response.
	Return 0
End

Class Example Extends App Final
	' Constant variable(s):
	
	' This will be the rate at which this program will run.
	Const FPS:Int = 60
	
	' Check if we're using explicit colors or not:
	#If EXAMPLE_CUSTOM_CLS_COLOR
		' The colors used to clear the screen:
		Const Cls_R:Float = 205.0
		Const Cls_G:Float = 205.0
		Const Cls_B:Float = 205.0
	#Else
		' Lower this value to "speed up" color generation.
		Const GeneratedColorDelay:Int = 10
	#End
	
	' The size of the image we'd like to grab.
	Const ImageData_Width:Int = 64 ' 100
	Const ImageData_Height:Int = 64 ' 100
	
	' Loading states:
	
	' If nothing is currently happening, this state will be used.
	Const IMAGEDATA_STATE_NORMAL:Int		= 0
	
	' This state signifies that the 'OnUpdate' routine is requesting
	' that a portion of the screen be loaded off of the GPU.
	Const IMAGEDATA_STATE_LOADING:Int		= 1
	
	' This is used to tell the 'OnUpdate' command that we've
	' loaded the image-data in the 'OnRender' routine.
	' Such actions have to be done due to potential target-specific problems.
	Const IMAGEDATA_STATE_LOADED:Int		= 2
	
	' Defaults:
	#If READPIXELS_REDRAW_FIX
		Const Default_RedrawScreen:Bool = True
	#End
	
	' Constructor(s):
	Method OnCreate:Int()
		' Set the update-rate of this application.
		SetUpdateRate(FPS)
		
		'Self.ImageDataState = IMAGEDATA_STATE_NORMAL
		
		' Generate a "dummy image" to hold the data we retrieve.
		Self.ImageData = CreateImage(ImageData_Width, ImageData_Height, 1, Image.MidHandle)
		
		' Generate a new "pixel buffer" for temporary raw storage.
		Self.Pixels = New Int[ImageData_Width*ImageData_Height] ' [ImageData.Width(), ImageData.Height()]
		
		#If READPIXELS_REDRAW_FIX
			Self.RedrawScreen = Default_RedrawScreen
		#End
		
		' This is done for caching purposes.
		CalculateHalfDeviceSize()
		
		#If EXAMPLE_DISCONNECTED_UPTIME
			Self.Uptime = Millisecs()
		#End
		
		' Return the default response.
		Return 0
	End
	
	' Methods:
	Method OnUpdate:Int()
		If (KeyHit(KEY_ESCAPE)) Then
			' Close the application.
			OnClose()
			
			Return 0
		Endif
		
		' Check if we have image-data to load onto the GPU:
		If (ImageDataState = IMAGEDATA_STATE_LOADED) Then
			' Write the pixels we retrieved to the image-data.
			ImageData.WritePixels(Pixels, 0, 0, ImageData_Width, ImageData_Height)
			
			' Set the loading-state back to normal.
			ImageDataState = IMAGEDATA_STATE_NORMAL
		Endif
		
		#If EXAMPLE_DISCONNECTED_UPTIME
			' Just felt like changing the value to something else:
			Uptime = Millisecs()
		#End
		
		' Check if the left mouse button has been pressed:
		If (MouseHit(MOUSE_LEFT)) Then ' And ImageDataState = IMAGEDATA_STATE_NORMAL
			ImageDataState = IMAGEDATA_STATE_LOADING
		End
		
		#If READPIXELS_REDRAW_FIX
			If (MouseHit(MOUSE_RIGHT)) Then
				RedrawScreen = Not RedrawScreen
			Endif
		#End
		
		' Return the default response.
		Return 0
	End

	Method OnRender:Int()
		' Local variable(s):
		#If Not EXAMPLE_DISCONNECTED_UPTIME
			Local Uptime:= Millisecs()
		#End
		
		' Clear the screen:
		#If EXAMPLE_CUSTOM_CLS_COLOR 
			Cls(Cls_R, Cls_G, Cls_B)
		#Else
			Local Color:Float = GenerateColor(Uptime)
			
			#If EXAMPLE_GENERATED_COLORTYPE = EXAMPLE_COLORTYPE_RED
				Cls(Color, 0.0, 0.0)
			#Elseif EXAMPLE_GENERATED_COLORTYPE = EXAMPLE_COLORTYPE_GREEN
				Cls(0.0, Color, 0.0)
			#Elseif EXAMPLE_GENERATED_COLORTYPE = EXAMPLE_COLORTYPE_BLUE
				Cls(0.0, 0.0, Color)
			#End
		#End
		
		' Draw the up-time of this application to the screen.
		DrawText(String(Uptime), 20, 20)
		
		' Draw the current image-data state.
		DrawText(String(Int(ImageDataState)), 20, 40)
		
		Select ImageDataState
			Case IMAGEDATA_STATE_LOADING
				' Load part of the screen into the 'Pixels' buffer.
				ReadPixels(Pixels, 0, 0, ImageData_Width, ImageData_Height)
				'ReadPixels(Pixels, 0, 0, ImageData.Width(), ImageData.Height())
				
				' Set the image-data detection flag back to 'False'.
				ImageDataState = IMAGEDATA_STATE_LOADED
				
				#If READPIXELS_REDRAW_FIX
					If (RedrawScreen) Then
						Return OnRender()
					Endif
				#End
			Case IMAGEDATA_STATE_NORMAL
				#If Not DRAW_OLD_IMAGEDATA_ON_CAPTURE
					' Draw the image-data we retrieved, to the center of the screen.
					DrawImageData()
				#End
		End Select
		
		#If DRAW_OLD_IMAGEDATA_ON_CAPTURE
			DrawImageData()
		#End
		
		' Return the default response.
		Return 0
	End
	
	Method GenerateColor:Float()
		#If EXAMPLE_DISCONNECTED_UPTIME
			Return GenerateColor(Self.Uptime)
		#Else
			Return GenerateColor(Millisecs())
		#End
	End
	
	Method GenerateColor:Float(Uptime:Int)
		Return (Sin(Uptime/GeneratedColorDelay)*255.0)
	End
	
	Method DrawImageData:Void()
		DrawImage(ImageData, HalfDeviceWidth, HalfDeviceHeight)
		
		Return
	End
	
	Method CalculateHalfDeviceSize:Void()
		HalfDeviceWidth = Float(DeviceWidth()) / 2.0
		HalfDeviceHeight = Float(DeviceHeight()) / 2.0
		
		Return
	End
	
	Method OnResize:Int()
		CalculateHalfDeviceSize()
		
		' Return the default response.
		Return 0
	End
	
	' Fields:
	#If EXAMPLE_DISCONNECTED_UPTIME
		Field Uptime:Int
	#End
	
	' The current state of the image-data.
	Field ImageDataState:Int
	
	#If READPIXELS_REDRAW_FIX
		Field RedrawScreen:Bool
	#End
	
	' An 'Image' object used to represent the current state of the texture on the GPU.
	Field ImageData:Image
	
	' The buffer we'll use to hold the raw image-data we load from the screen.
	Field Pixels:Int[]
	
	' Just here for optimization reasons.
	Field HalfDeviceWidth:Float, HalfDeviceHeight:Float
End

Press left-click to capture the screen. On XNA specifically, you can hit right-click to disable the re-draw fix.

You should probably report this to Mark in the "Bugs" section. It doesn't even redraw as black, either, it's the default XNA color from the look of it. Also, I think I found a different bug, the XNA target doesn't care if you use 'SetUpdateRate' before running, it will just run regardless.

P.S. Please use the 'codebox' tag for long pieces of code, that way I don't have to scroll all the way through it.

pantson

(Posted 2015) [#4]

thanks Anthony
Ive been on this forum for such a long time and never knew about codebox ;-) I'll be using that from now on.

I can see that you are redrawing (calling onRender again) to fix the issue after grabbing the pixel data, a bit like my current fix in my game.

I'll report it as a bug.

Many thanks for testing and confirming.