Sharing memory between BlitzMax processes?

BlitzMax Forums/BlitzMax Programming/Sharing memory between BlitzMax processes?

Vilu(Posted 2006) [#1]
So, what I'm trying to do is to do some load balancing for a CPU-intensive game.

Shared memory would be my method of choice to perform some tasks in parallel, as BlitzMax lacks the ability to do real multithreading.

Here's the deal:

I have a game project with potentially a lot of background calculation going on at all times, stuff that doesn't need to be calculated all the way through every single frame.

Currently I have implemented a crude way to spread the workload between frames: go through a small part of an array during every frame and let the game run while a large amount of data gets processed during a period of 100 or so frames.

I'd love to see a way to do it in a more elegant way, despite the fact I've never accomplished anything similar in any programming language.

I need some advice on how to spawn another process that takes its "work orders" from the main game process through memory buffers, chews on the the data on its own, and finally returns the results back to the main game through an output buffer. Or something like that.

Is there any way to create and access shared memory areas between two processes?


DStastny(Posted 2006) [#2]
Yes although it wouldnt be cross platform what platform are you interested in?

Doug Stastny


Vilu(Posted 2006) [#3]
Ah, it would be Windows XP.


Dreamora(Posted 2006) [#4]
Memory is shareable between threads but not between processes.

You will have to dublicate it or keep as much as possible dynamic and sent it back and forth through UDP.


Koriolis(Posted 2006) [#5]
Memory is shareable between threads but not between processes
Totally untrue. Memory is ALWAYS shared between threads, it is BY DEFAULT not shared between processes, but can be EXPLICITELY made shared betwwen processes.

For this you can use either file mappings, or shared segments. From BlitzMax your best bet is to use file mappings. A file mapping (Windows specific) lets you access to a shared area in the virtual memory space. Type "createfilemapping" in google and from there you should find the needed information.

As Budman said, that's totally platform-specific.


Panno(Posted 2006) [#6]
memory is shareable but i dont know the trick .
(winhex/rameditor)


DStastny(Posted 2006) [#7]
Ding Ding Ding!

Koriolis gets the prize :)

Doug Stastny


DStastny(Posted 2006) [#8]
Ill be nice here is some code

' Simple program to demonstrate sharing memory between process
' compile as console program and run one or more times the first run sets the 
' shared memory
' additonal runs read it

SuperStrict
Extern "Win32"
    Function CloseHandle:Int(Handle:Int)
	Function MapViewOfFile:Byte Ptr(hFileMappingObject: Int,dwDesiredAccess: Int, dwFileOffsetHigh:Int, dwFileOffsetLow:Int, dwNumberOfBytesToMap:Int)
	Function UnmapViewOfFile:Int(lpBaseAddress: Byte Ptr)
	Function CreateFileMappingA:Int(hFile: Int, lpFileMappingAttributes: Byte Ptr,flProtect:Int, dwMaximumSizeHigh:Int, dwMaximumSizeLow:Int, lpName$z)
	Function GetLastError:Int()
End Extern

Const PAGE_READWRITE : Int=4
Const FILE_MAP_WRITE : Int=2

Type TSharedMemory 
	Field _Handle : Int
	Field _Name   : String
	Field _Size   : Int
    Field _Owner  : Int
	Field _Data   : Byte Ptr

   
	Method Delete()
		Close
	End Method
	Method Open:Int(Name: String, Size: Int)
		Close()
   		_Name = Name
    	_Size = Size
       ' CreateFileMapping, when called with $FFFFFFFF For the hanlde value, creates a region of shared memory }
   		_Handle = CreateFileMappingA($FFFFFFFF, Null, PAGE_READWRITE, 0,_Size,_Name)
    	If _Handle = 0 Then Return False
        _Owner = GetLastError() = 0
       ' We still need To map a pointer To the handle of the shared memory region 
        _Data= MapViewOfFile(_Handle, FILE_MAP_WRITE, 0, 0, _Size)
    	If Not _Data
			Close
			Return False
		End If
		Return True
	End Method 
	Method Close()
  		If _Data
	    	UnmapViewOfFile(_Data);
			_Data=Null
		End If
		If _Handle 
			CloseHandle(_Handle)
			_Handle=0
		End If 		
	End Method
End Type

Print "Test Share Mem"
Local mem:TSharedMemory = New TSharedMemory 
mem.Open("TESTMEM",30)
If mem._Owner
	Print "Setting Title"
	MemClear mem._Data,30
	Local title:String="Budman Rocks"
	Local pTitle:Byte Ptr=title.toCString()
	MemCopy mem._Data,pTitle,title.length+1
	MemFree pTitle
Else
	Print "Getting Title"
	Local title : String = string.FromCString(mem._Data)		
	Print "The Title is "+title
End If
Input
mem.Close


Have fun
Doug Stastny


ImaginaryHuman(Posted 2006) [#9]
Aren't you basically asking to be able to provide a multitasking feature within your single process that uses as much `leftover time` as is available for background tasks and doesn't get in the way of the important stuff you need to accomplish? Why not just do the stuff you must get done at the start of the frame then with remaining time just do a loop where you do a bunch of background stuck and check the Millisecs() every now and then to see if the frame time has run out yet?


DStastny(Posted 2006) [#10]
There are somethings you cant with Timeslicing, due to the fact they block and you can not measure time slice. Which is why people want threading so bad.

Examples.
1. Networking, you really want to handle that as fast as possible not wait and poll.

2. Database work. Lets say you want to save player states for a MUD to a Database, you can wait for your interaction for the database to complete, as you might timeout network connections to the players.

3. You want to connect your game/app to a internet server but still be responding to user in realtime.

These are types of things that simple timeslicing cant resolve.

Now if your doing everything in your own code time-slicing can be done, but been there done that, its a pain in the rear, to code. Me I would prefer threads however Shared Process memory can help a feature limited BMAX.

Doug Stastny


ImaginaryHuman(Posted 2006) [#11]
Everyone says max is lacking features when their little obscure feature happens not to be supported ;-)

But anyway, yeah you are probably right. Threads would be nice especially to make use of the trend towards multiple core cpu's.


Vilu(Posted 2006) [#12]
Thank you all for your input, and thanks Doug for the snippet you provided (although I was prepared to do the research myself ;) )

The Windows API solution seems to work and is worth looking into.

=)


DStastny(Posted 2006) [#13]
No problem Vilu, I just translated some delphi classes, I had used is a project.

My only concern would be speed issues. Look forward to reading what you might do with this.

Doug Stastny


Alessandro(Posted 2006) [#14]
Hello,

another possbile solution could be using anonymous pipes (windows).

I used them (in Delphi) to send/receive info from different processes :-)

--Alessandro


Pantheon(Posted 2006) [#15]
@Budman
Hi,
I tried your source code a few days ago and converted it into my own class also included critical section support (Not threw API calls just a counter in the shared memory). It seemed to work realy well for a while but after like 40 runs of the program or more my computer started grinding to a hault and the application would randomly quit. It was made even worse when in debug mode.

Perhaps this method is realy unstable in BlitzMax? I didnt change much from your source only added to it. I would post the code but it dont have it with me sorry. Just thought you mite be interested.


DStastny(Posted 2006) [#16]
@Pantheon

Hello,

Nothing really special about BMax and this code should cause a problem. I suspect if you dont call the close method and ensure you Free the Shared memory you will run into problems.

You need to make sure that if the Application Terminates the Close Method has been called as that would leak API resources.

My demo was really simple example, but I would use a Global Var for the Shared Memory section and write an Onexit handler to ensure that the close method is invoked. If you just call "END" in BMAX you will have problems.

The one nuanices of these types of resources is they are avaialbe across processes so you need to be real sure your application cleans them up.


Now your code might be faulting due to both procesess accessing the Shareed memory at the same time. For that it really needs a real OS MUTEX/CRITICAL Section to ensure only one process is accessing the Shared memroy at that same time.

My example was trivally stupid and just to show it can be done. Its not really a full blown piece of code just the bare minimum.

If you want to share your code, Ill take a look at it and see if I can see what might cause your problem. I suspect it needs OS Mutex.

Doug Stastny


Pantheon(Posted 2006) [#17]
When I get back to my computer then I will post some code. You say that the two processes accessing shared memory at the same time can cause error? That would be the most likely case because I implemented two functions, Lock( ) and Unlock( ).

The shared memory size is increased by an integer and that becomes a variable to say how many times the memory has been locked. The lock call was blocking by using a while loop to check if memory was locked. I had a small delay in the loop and I guess I ran into trouble when I brought the delay time down.

If you can make sense of that then it all seems to pan out. I'll put the code up as soon as just for peoples intrest.

Thanks


bach(Posted 2016) [#18]
Hi good afternoon.
I once changed your program.

Write 1x filemapping
Read 1x file mapping

First, the listener then Start the evening program.

greeting



BlitzMax-Filemapping-Write-Integer:
SuperStrict

Graphics 300,200

Global timer:TTimer = CreateTimer(60)
global wert:int

Extern "Win32"
  Function CloseHandle:Int(Handle:Int)
	Function MapViewOfFile:Int Ptr(hFileMappingObject: Int,dwDesiredAccess: Int, dwFileOffsetHigh:Int, dwFileOffsetLow:Int, dwNumberOfBytesToMap:Int)
	Function UnmapViewOfFile:Int(lpBaseAddress: Byte Ptr)
	Function CreateFileMappingA:Int(hFile: Int, lpFileMappingAttributes: Byte Ptr,flProtect:Int, dwMaximumSizeHigh:Int, dwMaximumSizeLow:Int, lpName$z)
	Function GetLastError:Int()
End Extern

Const PAGE_READWRITE : Int=4
Const FILE_MAP_WRITE : Int=2

Type TSharedMemory 
	Field _Handle : Int
	Field _Name   : String
	Field _Size   : Int
  Field _Owner  : Int
	Field _Data   : Byte Ptr

	Method Delete()
		Close
	End Method

	Method Open:Int(Name: String, Size: Int)
		Close()
   		_Name = Name
    	_Size = Size
       ' CreateFileMapping, when called with $FFFFFFFF For the hanlde value, creates a region of shared memory }
   		_Handle = CreateFileMappingA($FFFFFFFF, Null, PAGE_READWRITE, 0,_Size,_Name)
    	If _Handle = 0 Then Return False
        _Owner = GetLastError() = 0
       ' We still need To map a pointer To the handle of the shared memory region 
        _Data= MapViewOfFile(_Handle, FILE_MAP_WRITE, 0, 0, _Size)
    	If Not _Data
			Close
			Return False
		End If
		Return True
	End Method 

	Method Close()
  		If _Data
	    	UnmapViewOfFile(_Data);
			_Data=Null
		End If
		If _Handle 
			CloseHandle(_Handle)
			_Handle=0
		End If 		
	End Method
End Type

Local mem:TSharedMemory = New TSharedMemory 
mem.Open("MyFileMapping",30)

Repeat 
  cls

  delay(300)
  wert=wert+100
  if wert > 20000 then wert=0
	drawtext "write integer : "+ wert,10,10
	MemClear mem._Data,30
	Local pTitle:Byte Ptr=varptr wert

	MemCopy mem._Data,pTitle,4
	MemFree pTitle

	Flip 
	WaitTimer(timer)
Until KeyHit (KEY_ESCAPE)

mem.Close
end


BlitzMax-Filemapping-Read-Integer:
SuperStrict

Graphics 300,200

Global timer:TTimer = CreateTimer(60)
global wert:int
global text:string

Extern "Win32"
  Function CloseHandle:Int(Handle:Int)
	Function MapViewOfFile:int Ptr(hFileMappingObject: Int,dwDesiredAccess: Int, dwFileOffsetHigh:Int, dwFileOffsetLow:Int, dwNumberOfBytesToMap:Int)
	Function UnmapViewOfFile:Int(lpBaseAddress: Byte Ptr)
	Function CreateFileMappingA:Int(hFile: Int, lpFileMappingAttributes: Byte Ptr,flProtect:Int, dwMaximumSizeHigh:Int, dwMaximumSizeLow:Int, lpName$z)
	Function GetLastError:Int()
End Extern

Const PAGE_READWRITE : Int=4
Const FILE_MAP_WRITE : Int=2

Type TSharedMemory 
	Field _Handle : Int
	Field _Name   : String
	Field _Size   : Int
  Field _Owner  : Int
	Field _Data   : int Ptr

	Method Delete()
		Close
	End Method

	Method Open:Int(Name: String, Size: Int)
		Close()
   		_Name = Name
    	_Size = Size
       ' CreateFileMapping, when called with $FFFFFFFF For the hanlde value, creates a region of shared memory }
   		_Handle = CreateFileMappingA($FFFFFFFF, Null, PAGE_READWRITE, 0,_Size,_Name)
    	If _Handle = 0 Then Return False
        _Owner = GetLastError() = 0
       ' We still need To map a pointer To the handle of the shared memory region 
        _Data= MapViewOfFile(_Handle, FILE_MAP_WRITE, 0, 0, _Size)
    	If Not _Data
			Close
			Return False
		End If
		Return True
	End Method 

	Method Close()
  		If _Data
	    	UnmapViewOfFile(_Data);
			_Data=Null
		End If
		If _Handle 
			CloseHandle(_Handle)
			_Handle=0
		End If 		
	End Method
End Type

Local mem:TSharedMemory = New TSharedMemory 
mem.Open("MyFileMapping",30)

Repeat 
  cls

	If mem._Owner
		wert = mem._Data[0]	
		drawtext "read integer : "+wert,10,10
	End If

	Flip 
	WaitTimer(timer)
Until KeyHit (KEY_ESCAPE)

mem.Close
end