Garbage Collector on stdcpp target

Monkey Forums/Monkey Programming/Garbage Collector on stdcpp target

ziggy

(Posted 2012) [#1]

I'm using the stdcpp target to code a sort-of script engine for Monkey and I was wondering how to deal with GC when using this target. The documentation says that any C++ based targets do perform collection when the control of the app is set back to the OS, but when I work on a standard console application using the stdcpp target, when does this happens? There's no way to "turn back the control to the OS" as it is not an event-based target. Any ideas? Am I lossing anything?

Samah

(Posted 2012) [#2]

I had a problem when porting Lua to Monkey that objects would be GC'd on the Monkey side and no longer be valid on the C side (dangling pointers). I created my own reference counter class for it. Retaining is easy, but releasing requires some smarts. Thankfully I could throw it into the __gc metatable function.

ziggy

(Posted 2012) [#3]

Keeping references is not an issue. I'm worried about them not being collected. So if they were in LUA I supose I don't have to worry.... ?

Samah

(Posted 2012) [#4]

Hmmm, I'm not sure what you can do there other than externing some code to force GC? Not sure if that can be done.
Also, Lua is a word, not an acronym.

ziggy

(Posted 2012) [#5]

But how did you get your objects being collected within the Stdcpp target? If this was a problem for you, maybe this means I do not have to worry abut it?

Samah

(Posted 2012) [#6]

Just make sure you always keep a reference somewhere. For any Monkey object passed to the externed code, I used a Monkey callback to add it to a reference counter. This prevents Monkey from GC'ing once the object goes out of scope (once it leaves the method). When a Lua table gets collected (done internally in Lua) it calls the __gc metamethod. I set the metamethod to call a Monkey callback to release the reference, letting Monkey GC it.

http://code.google.com/p/monkeylua/source/browse/src/monkeylua/monkeylua.monkey

Private
' A simple reference counting class to ensure that Monkey objects stored in the Lua state will not be GC'd by the Monkey runtime
Class ReferenceCounter
	Field referenceObjects:Object[] = New Object[100]
	Field referenceCounts:Int[] = New Int[100]
	Field references:Int = 0
	
	Method Retain:Void(obj:Object)
		For Local i:Int = 0 Until references
			If referenceObjects[i] = obj Then
				referenceCounts[i] += 1
				Return
			End
		Next
		If references >= referenceObjects.Length Then
			referenceObjects = referenceObjects.Resize(referenceObjects.Length*2)
			referenceCounts = referenceCounts.Resize(referenceCounts.Length*2)
		End
		referenceObjects[references] = obj
		referenceCounts[references] = 1
		references += 1
	End
	
	Method Release:Void(obj:Object)
		For Local i:Int = 0 Until references
			If referenceObjects[i] = obj Then
				referenceCounts[i] -= 1
				If referenceCounts[i] <= 0 Then
					references -= 1
					referenceObjects[i] = referenceObjects[references]
					referenceCounts[i] = referenceCounts[references]
					referenceObjects[references] = Null
					referenceCounts[references] = 0
				End
				Return
			End
		Next	
	End
	
	Method ReleaseAll:Void()
		For Local i:Int = 0 Until referenceObjects.Length
			referenceObjects[i] = Null
			referenceCounts[i] = 0
		Next
		references = 0
	End
End

http://code.google.com/p/monkeylua/source/browse/src/monkeylua/native/lua_post_extern.cpp

void ml_MonkeyLua::ml_Init(ml_lua_State *L)
{
        // create the metatable for objects and functions
        luaL_newmetatable(L->_data, MONKEYLUA_OBJECT_MT);
        // ...
        lua_pushstring(L->_data, "__gc");
        lua_pushcfunction(L->_data, ml_MonkeyLua::ml_Collect);
        lua_rawset(L->_data, -3);
        // pop the metatable
        lua_pop(L->_data, 1);
}

ziggy

(Posted 2012) [#7]

Ok, but my situation is the oposite to this. In the VM I'm designing the objects ARE handled by Monkey. I am worried about monkey not collecting them in the stdcpp target becouse of this text in the Monkey reference:

The current C++ garbage collector will only collect garbage when control is returned to the OS. In the case of C++ Mojo targets such as IOS and GLFW, this occurs after any of the 'On' methods such as OnCreate, OnUpdate etc return.

I was wondering what about the stdcpp target, as it is C++ based but does never gives the control back to the OS AFAIK as it is not a event-driver-based target. I honestly don't know what am I suposed to do here, or I just can assume garbage collection is properly handled as in any other managed target here.

ziggy

(Posted 2012) [#8]

Ok, this seems to show the problem:

Function Main()
	While True
		New list.List < String >
	Wend
End

This source code can eat your whole system memory if you let it run. It seems unreferenced instances of List are never collected. That's a problem, wonder if there's any way to force collection on the stdcpp parser, I don't care if it is a manual GCFlush or anything like this.

Does anybody know how to do this?

DruggedBunny

(Posted 2012) [#9]

You might be able to work from this quick hack which imports gc_collect and seems to do what it should...


' NB. Import of os/Sleep (Win32) just to avoid massive memory-hogging loop while testing! Can remove Delay line...

' Comment out Delay line to see it still works...

Import monkey.lang
Import os

Extern 
	Function GCCollect () = "gc_collect"
	Function Delay (ms:Int) = "Sleep"
Public

Function Main()
	While True
		New list.List < String >
		GCCollect
		Delay 1
	Wend
End

marksibly

(Posted 2012) [#10]

Hi,

GC isn't currently available for the stdcpp target. Well, it is, it's just that it only happens once Main() returns!

This means you can only use stdcpp for things like trans that just 'do something then exit'. But given the current limited state of the os module (ie: no sockets or any 'blocking' operations so you can't really write 'server-like' stuff), that's about all you can do at the moment anyway.

My initial plan was to use the hans boehm gc with the stdcpp target, but I decided to wait until the Monkey gc matured a bit and see if I could use that. Well, I probably could, but likely at a bit of a cost to glfw apps. Also, I don't consider the Monkey gc finished yet - eg: I want to have a crack at generational gc soon, hopefully make it realtime eventually...

So if you need gc for stdcpp, the best approach at this point IMO is probably for me to add the hans boehm gc. I don't think this'll be a major hassle as I've already had experience at integrating it into bmx, but it'll make stdcpp projects a little more complex that just a 'main.cpp'.

But like I say, with a lot of simple 'do something and exit' apps, gc isn't really even necessary. What are you writing exactly?

Note that James' hack wont work, the problem being that local variables wont be 'marked' - this is the fundamental hassle involved in adding gc to stdcpp. However, it's a hassle that will likely need to be overcome for ALL cpp targets as apps get more sophisticated, at which point one gc for all will probably happen.

DruggedBunny

(Posted 2012) [#11]

Hi Mark, just to clarify, does that mean the hack *is* freeing object memory in this specific demo (it appears to, as memory usage doesn't increase), but that it wouldn't free any memory allocated for 'non-object' variables in a real-world app?

Slightly OT but related to my hack, two small additions I've sometimes wished for in STDCPP/monkey.os are Delay and Input -- any chance of those making it in at some point?

Even just Input would be welcome, as I couldn't figure out how to return the input string when I tried to hack it in, but I think any console-type program ought to at least be able to take input.

ziggy

(Posted 2012) [#12]

Mark: I'm currently writing an interpreter (the compiler, and virtual machine) in Monkey. It is in very early status and not even considered a comercial project (it's open source), but more a personal learning project I'm using to stress test Jungle Ide and hopefully give back something interesting to this community. At this moment, if GC is not available on the stdcpp target by now, it's not a drama. My project is in a very early status, so not a big problem, I can live with it for some months. I can go on with the project but, if you ask me, I would love this to be as much compatible to c++ as possible in the future, so I supose at some point a proper GC for C++ would be more than welcome. Isn't a realtime generational garbage collector what Java and C# are currently using? Maybe this would make Monkey even more consistent between targets when compared to Java or .net targets?

ziggy

(Posted 2012) [#13]

@DruggedBunny:

I've sometimes wished for in STDCPP/monkey.os are Delay and Input

Here's the input:

#rem
	header: This module is a very simple I/O layer for the stdcpp target.
	It provides the basic functionality to send string data to the Standard Output Pipe, and to the Standard Error Pipe. It also provides an Input function to get data from the user, using the typical console prompt.
#end

#if TARGET="stdcpp" Or TARGET="glfw"
	'Private
	Extern private
	
	Class FILE
	End
	
	Global stdin:FILE
	Global stdout:FILE
	Global stderr:FILE
	
	Function fputc(c,file:FILE)
	Function fflush(file:FILE)
	Function fgetc:Int(file:FILE)
	'Const EOF
	Public
	
	'summary: Prompt for user input in the system console window and resturns a String
	Function Input:String(prompt:String=">")
		
		Local c:Int, result:String
		
		For Local i:Int = 0 until prompt.Length 
			fputc(prompt[i],stdout)
		end
		fflush(stdout)
		
		c=fgetc(stdin)
		
		While c<>10 And c<>13
			result += String.FromChar(c)
			c=fgetc(stdin);
		Wend
		
		fflush(stdin)
		
		Return result;
	End
	
	'summary: Sends text to the standard output pipe
	Function Output:void(value:String)
		For Local i:Int = 0 to value.Length-1
			fputc(value[i],stdout)
		Next
		fflush(stdout)
	End
	
	'summary: Sends text to the standard error pipe
	Function ErrOutput:void(value:String)
		For Local i:Int = 0 to value.Length-1
			fputc(value[i],stderr )
		Next
		fflush(stderr)
	End
#Else
	#Error "The Jungle stdio module is only supported in C++ based targets such as stdcpp or glfw."
#End 

#rem
	footer: Copyright (c) 2011-2012 Manel Ib��ez

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#end

If you're using Jungle Ide, you'll have to use it from the console, as Jungle Ide console does not support the input pipe of the running connection, so it is not possible to enter data on a input command (as it was not available on Monkey, I did not add this, but I'll do it soon, as I need it for my own projects).

marksibly

(Posted 2012) [#14]

Hi,

> Hi Mark, just to clarify, does that mean the hack *is* freeing object memory in this specific demo (it appears to, as memory usage doesn't increase), but that it wouldn't free any memory allocated for 'non-object' variables in a real-world app?

The problem is that the current GC doesn't 'mark' local object vars (non-object vars are a non-issue!). Marking is what keeps things from being collected, so this:

Local t:=New Thing
GCCollect

Could potentially result in 't' being prematurely freed if nothing else is keeping it alive.

This isn't an issue if GC only happens when control is returned to 'native' code as all local variables have gone out of scope.

> Mark: I'm currently writing an interpreter (the compiler, and virtual machine) in Monkey.

Could you perhaps write it in glfw for now, but keep the 'guts' of it separate so it can be reused with a potential future stdcpp target?

Or if you're just writing an 'interpret' command to be called from the command line, you can probably just get away with no GC, ala trans. This would in fact be effectively the same if you wrote it in glfw and interpret() was called inside OnUpdate.

> at some point a proper GC for C++ would be more than welcome.

For sure! The truth is that the stdcpp target (and os module) were really only written in order to get trans up and running (which was originally a bmx app) and I was undecided about whether or not stdcpp should even be released as an 'official' target in the first place. I'm glad I did though, as it's made life easier. The stdcpp target will definitely be improved over time, as will the os module (probably soonish!).

> Isn't a realtime generational garbage collector what Java and C# are currently using?

Neither are realtime, hence the dodgy 'collection pauses' people experience on both the android and XNA targets. Html5 suffers less from this but I suspect that's mainly because html5 apps are generally running on much gruntier hardware. I'm kind of proud of how Monkey's GC is now running on iOS though, esp. after the last incremental tweaks.

C#'s gc is generational, but I don't think Android's is - it uses some sort of 'region' collection algorithm instead which doesn't sound too useful for our purposes, ie: realtime games.

ziggy

(Posted 2012) [#15]

Ok then I'll continue working with the non collected stdcpp target, just to get my work "up and running" as soon as possible, and i'll be moving it to glfw or xna as my "next step" so it won't be suffering from this as much as it is now. As the interpreter won't be event-driven (at last initially) I see not having a regular GC as a problem in the long run. I mean a garbage collector that requires returning the control to the OS can be a bit dangerous for this kind of applications. But anyway, if there are plans to improve this at some point, I supose it's not an issue. first of all, I'll have to make tones of work before having anything usable.