Pointers

Community Forums/Monkey2 Talk/Pointers

abakobo(Posted 2015) [#1]
Hi,

Will we get pointers and function pointers?

Thanks for the great work and I hope we can quickly help the r&d with the crowdfunding!


ziggy(Posted 2015) [#2]
If we're getting delegates, I don't think we need function pointers. And, well not sure why would you need data pointers on a reference based language, but maybe I'm loosing something?


abakobo(Posted 2015) [#3]
Data pointers are dynamic and allow you to do whatever you want in your data structure... you can even use arithmetics to point!. Your imagination and memory is the limit... I know it's dangerous and source for headhakes, but i love them.


ziggy(Posted 2015) [#4]
The problem, as I see it, is that this kind of pointers can overcomplicate the GC to no end, and I'm not sure they introduce any improvement to the language. But Mark could know better than me if he's using a GC that is compatible with this, and also fast.


marksibly(Posted 2015) [#5]
> Will we get pointers and function pointers?

Yes, but as Ziggy points out, pointers are very unsafe as they don't play nicely with the GC.

Besides, in Monkey everything is already a 'pointer' when it comes to objects. Pointers to ints etc are a different story, but this can also be achieved safely via: Int[] or a wrapper object.

> If we're getting delegates

Define delegate!

I've tried, but I can't seem to find a definitive definition. I've seem them described as everything from a plain function pointer to a pointer to a member function to a C# style strongly typed function pointer.

In Monkey2, there is only really a function pointer, eg:

local somefunc:void(int) 'somefunc is a function that takes an int and returns void.

However, this can be used quite flexibily, eg:

somefunc=Blah 'assign global function (closure one day...)
somefunc=c.Etc 'assign instance method, ie: somefunc holds both an object and a method.

Function types can be alised to provide C#-ish 'delegates', eg:

alias SortFunc:int(int,int) 'similar to "delegate int SortFunc( int,int )" in C#

local t:SomeFunc=CompareInts
t=c.CompareInts

But these are not strongly typed as in c#. eg: in c#, delegate x:int() and delegate y:int() are completely different types (I think). You can't assign a var holding an x to a var holding a y. But in monkey you can, as they will just be aliases for the same function type. Which is both a good and a bad thing...but IMO it's more flexible.

So I'm not sure if monkey2 supports delegates or not!


Samah(Posted 2015) [#6]
@marksibly: Define delegate!

A combination function pointer with an object reference.
Local d:Delegate = New Delegate(object, method)
d(1, 2, 3) ' calls object.method(1, 2, 3)

Not sure how you'd define argument/return types, but that's the general idea.


GW_(Posted 2015) [#7]
neat.


JoshKlint(Posted 2015) [#8]
Any external library that uses callbacks needs function pointers.


ziggy(Posted 2015) [#9]
@Marksibly, yes, what Samah said. It's a pointer to a function that can also retain a specific instance of an object to access instance methods.

Idea:
Class MyClass
    Method CallWithADelegate
        Local myDelegate:= AddressOf Self.MyMethod
        myDelegate.Invoke(12,"hello world")
    End
    MyMethod(value:Int, data:String)
        Print value + " " + data
    End
End


This example is just silly, but I think you'll get the idea. They're very very useful to build event systems.

Example:
Class EventHandler<T>
    Field listeners:List<Delegate(sender:Object, arguments:T)>
    Method RaiseEvent(sender:Object, arguments:T)
        For each del:=listeners
            del.Invoke(sender,arguments)
        End
End Class

Class MyClassThatRaisesAnEvent
    Field ImDone:EventHandler<Bool>
    Method DoStuff()
        ...
        ...
        ImDone.RaiseEvent(Self, True)
    End
End

Class MyClassThatListensToAnEvent
    Field instance:=New MyClassThatRaisesAnEvent
    Method New
        instance.ImDone.listeners.Add(AddressOf Me.InstanceHasDoneStuff)
    End
    Method InstanceHasDoneStuff(sender:Object, success:Bool)
        ....
        Print "Listening to instance! It's done it's stuff!"
    End
End

This gives tones of flexibility compared to regular function pointers, as they play very nice with OO design.
I don't think it is important to have then "type safe" as you said. A delegate is just a sort of signature for a function or method with instance pointer. All delegates with identical signature can be the same object. If ou ask me, the big questions here would be how to treat references into delegates. They can be took into account in the GC or not, and if they're not, delegates need to be nullified when the reference is free by the GC. When a delegate points to a non instance method, it should act as a regular function pointer.


marksibly(Posted 2015) [#10]
> They're very very useful to build event systems.

I think that code is more or less covered - here's how it'd look in monkey2:

Class EventHandler<T>

	Field listeners:=New List<Void(Object,T)>
    
	Method RaiseEvent( sender:Object,arguments:T )
		For Local del:=Eachin listeners
    			del( sender,arguments )
    		Next
    	End
    	
End

Class MyClassThatRaisesAnEvent

    Field ImDone:=New EventHandler<Bool>
    
    Method DoStuff()
    		Print "thinking..."
		ImDone.RaiseEvent( Self,True )
    End

End

Class MyClassThatListensToAnEvent

	Field instance:=New MyClassThatRaisesAnEvent
	
	Method New()
		instance.ImDone.listeners.AddLast( InstanceHasDoneStuff )
	End

	Method InstanceHasDoneStuff:Void( sender:Object,success:Bool )
		Print "Listening to instance! It's done it's stuff!"
    End
End


Think I'm gonna try and get that building tomorrow!

You could also alias the function pointer type to make it a bit cleaner, eg:

Class EventHandler<T>

	Alias Listener:Void( Object,T )  'kind of like c# delegate? Listener is a 'type'.
	
	Field listeners:=New List<Listener>
...etc...


This also allows other code to get at the type of a listener, eg:

Local myListener:EventHandler<Bool>.Listener 'same as Local myListener:Void(Object,Bool)

Similar to the way the STL typedefs 'ValueType', 'IteratorType' etc inside containers.

> They can be took into account in the GC or not, and if they're not, delegates need to be nullified when the reference is free by the GC.

I don't think this will be too bad - I'm currently using 'copy on write' to handle 'function pointer values' which contain objects (or not) and I think it'll work OK. If not, I can fallback to make them real objects (with associated overhead).


ziggy(Posted 2015) [#11]
If this can be done, I'll be *finishing* JungleGui for Monkey2. It was sort of stopped because of the problems I was getting using reflection to "emulate" delegates at runtime. HAving all this handled at compile time would be a dream to me!


itto(Posted 2015) [#12]
Are you saying we will finally have real callback methods/functions instead of having to rely on interfaces? This would be amazing!


ziggy(Posted 2015) [#13]
Are you saying we will finally have real callback methods/functions instead of having to rely on interfaces?
Yes, that's what delegates are for! (and also can be used to do some malakary on multithreading)


marksibly(Posted 2015) [#14]
> problems I was getting using reflection to "emulate" delegates at runtime

Watching you struggle with that was what partly convinced me that good old function pointers could be very useful (that and learning Qts slots and signals stuff) - and that Monkey really needed a rewrite as it's one of several things that I would never have been able to add to monkey1 without going insane in the process.

There is ONE very curly issue with function types though, and that's parsing. In particular, when you actually declare functions, eg:

Function Update:Int()

The parser wants to parse the return type of Update as 'function returning an Int' because it sees 'Int()', but that's wrong. That would be:

Function Update:Int()() 'function returning a function.

As I did in BlitzMax, I am strongly tempted to just hack the parser to make it work, as I think it's worth it for the 'cleaner' usage, but then the idea of a nice clean 'grammar' goes out the window!

Any thoughts on this? Is hacking the parser like this ever justifiable? When?!?


ziggy(Posted 2015) [#15]
I would suggest a keyword-based identifier (I'm using Action as a keyword, but it could be Alias or whatever you like):
Action MyDelegate:Bool(param1:String, param2:String) '--> Defines a method signature here, using Action 

Function Update:MyDelegate(parameter1:Int) '--> Define a function with an integer parameter, and this function returns a function that returns an Bool, and has 2 string parameters, because it returns a MyDelegate type

Field myCallback:MyDelegate = me.StuffDone '--> Where StuffDone is a method that returns a Bool and has 2 string parameters
If myCallback("Hello ", "world")
    DoStuff...
End If


Another idea could be an operator intended to provide readabiltiy. Approach for the same example:
Function Update:&Bool(param1:String, param2:String)(parameter1:Int)


In my opinion, operator-based approach is a lot less readable:
Field myPointer:&Bool(param1:String, param2:String)
...
myPointer = me.DoSomething
If myPointer("hello ", "world") = True Then
    DoStuff
EndIf

It's obvious that the operator itself is not required for the compiler to solve the datatype, but it could help providing readability?

Those are the 2 ideas that come to my mind. Being honest, in my opinion, first approach it a lot better when it comes to understand and follow code. The second one can make code too cryptic for no good reason.

Now going back to your example:
Action MyIntegerFunction:Int() '-> That's the "delegate" signature definition.
...
Function Update:MyIntegerFunction() '->That's usage of that definition.

I think it's better than Update:Int()() approach


GW_(Posted 2015) [#16]
I'm finding the colon token ':' a bit strange here.
For Bmax and Monkey, it's always been <ident> : <primitive type | class type>
I thinking that maybe some other type of lexeme could be used to avoid confusion.
It seems kinda off-putting to see function prototypes after the colon.
Maybe i'm just confused.

btw: I second ziggy's opinion that a keyword would be preferable to '&' etc. and definitely no arrow '->' .
;-)


marksibly(Posted 2015) [#17]
> Action MyDelegate:Bool(param1:String, param2:String) '--> Defines a method signature here, using Action
> Function Update:MyDelegate(parameter1:Int) '--> Define a function with an integer parameter, and this function returns a function that return

So all function types would have to be 'named' before they could be used?

I think aliasing function types is definitely nice in a lot of situations, but I wouldn't like to make it compulsory, c# style. Still, it's an option.

> I'm finding the colon token ':' a bit strange here.

The general idea of ':' is it's always followed by a type, so in the case of function decls it's followed by the function return type.

Perhaps you could post an example of how you'd like function decls to look?

I think this more consistent than somehow specifying function return types *without* a preceding ':', as it breaks that rule.


Samah(Posted 2015) [#18]
@marksibly: Class EventHandler<T>...

That's not really an example of delegates though, that's just a listener pattern. For "Void(Object,T)" to be a delegate, it would need to have both a reference to an object, and a reference to one of that object's class's methods.


marksibly(Posted 2015) [#19]
It does! Checkout...

instance.ImDone.listeners.AddLast( InstanceHasDoneStuff ) 'a 'delegate' with (self,InstanceHasDoneStuff)

Ditto...

local c:=new C
SomeFunc( c.SomeMethod ) 'if SomeFunc takes a valid func ptr, a delegate with (c,SomeMethod) is created.

Effectively: {object}.{method} can be converted to a function ptr.

If there's no {object}, it's the same as self.{method}

Anyway, I got Ziggy's code translating (minus Eachin...), but not compiling yet due to forwards refs in c++...argh.


GW_(Posted 2015) [#20]
The general idea of ':' is it's always followed by a type, so in the case of function decls it's followed by the function return type.
Perhaps you could post an example of how you'd like function decls to look?

I was referring to this.
Action MyIntegerFunction:Int() '-> That's the "delegate" signature definition.
...
Function Update:MyIntegerFunction() '->That's usage of that definition.

Just seems strange to me as it's harder to reason about the return type. But I'm definitely reserving judgment until I see it in action, especially if I'm the only one who thinks it's not intuitive (most likely). Either way, whatever IDE Monkey2 uses will need to have some clever and robust code completion.


marksibly(Posted 2015) [#21]
> I was referring to this.

Ahh, OK.

It does actually make sense if you think of Action as a simple 'type alias' instead, eg:

alias RetType1:void(int,int)
alias RetType2:int
alias RetType3:List<Actor>
function func1:RetType1()   'returns a void(int,int) function
function func2:RetType2()   'returns an int
function func3:RetType3()   'returns a List<Actor>


You will definitely be able to do this in monkey2, and it can be very useful esp. when dealing with more complex types.

I'd just like to avoid forcing people to have to alias function types anywhere.


GW_(Posted 2015) [#22]
I get it. That example makes things more clear. Thanks


Samah(Posted 2015) [#23]
@marksibly: It does!

Based on the definition of delegate on Wikipedia, your example is "kinda" correct I guess. I'll resign myself to wishy-washy definitions.

It looks like what I'm talking about (and ziggy seems to agrees with) is a "method pointer" as opposed to a "function pointer". Since a method requires an instance, the instance and method pointer must be passed as a tuple.

This is an example of a "method pointer?", and an implicit generic operator like Java's <>
Class Foo
	Method Bar:Int(str:String)
		Print str
		Return 1
	End
End

' instance of class
Local inst:Foo = New Foo

' instantiate method pointer passing the object and a pointer to its method
Local mp:MethodPtr<Foo,Int(String)> = New MethodPtr<>(inst, Foo.Bar)

' execute the method pointer (prints "blah" and 1)
Print mp("blah")

' essentially the same as: Print inst.Bar("blah")



marksibly(Posted 2015) [#24]
What would you call this, if not a method pointer?

class C
   field t:=10
   method test:void()
      print t
   next
end

function main:void()
   local c:=New C
   local f:void()=c.test   'Note: "c.test" is the 'tuple' you're after...
   f()
end


Prints 10!

Sure, 'f' looks like a function pointer, but that's the point (ha!).


GW_(Posted 2015) [#25]
NM, Mark cleared it up. ;)


Samah(Posted 2015) [#26]
@marksibly: What would you call this, if not a method pointer?

That's different to your previous example though, because you're including both "c" and "test" in the same declaration. Previously you were keeping it as just a function pointer and enforcing the idea of passing an instance as the first argument.

local f:void()=c.test   'Note: "c.test" is the 'tuple' you're after...
f()

Your example here is much better, and something I'd love to see in the language.


Danilo(Posted 2015) [#27]
Somehow this can make large codes very unreadable:
class C
   field t:=10

   method test:void()
      print t
   end

   method s:int()
      return 2
   end
end

function func:void()
   print "func()"
end

function l:int()
   return 1
end


function main:void()
   local h:void()
   local i:int, j:int
   local k:int

   local c:=New C

   local f:void()=c.test   ' Note: "c.test" is the 'tuple' you're after...
   f()

   local g:void()=func     ' function pointers look exactly like method pointers?
   g()

   ' Without actually seeing the definition of h:void(),
   ' it's hard to guess what the following line does:

   h = func   ' this assigns a pointer-to-func to h,
              ' but in MonkeyX this could also be read
              ' as a function call (because parenthesis is not required),
              ' and it could as well be a variable assignment
   i = j
   k = l
end

It looks all the same, like "a = b".

No way to differentiate what is a function call, what is a function pointer assignment, and what is just
a variable assignment.
That's why I like it to have functions/methods use always parenthesis, and something like '@' or "adressOf" operator
to differentiate between function call and function pointer.
h = @func()            ' clearly a function pointer
h = addressOf func()   ' clearly a function pointer
i = j                  ' clearly 2 variables
i = c.t                ' clearly 2 variables
k = c.s()              ' clearly a method   call
k = l()                ' clearly a function call



ziggy(Posted 2015) [#28]
I agree having @ or AddressOf or any other identifier can help make the code easier to follow.


marksibly(Posted 2015) [#29]
> have functions/methods use always parenthesis

This is definitely happening! And with that in place...

> '@' or "adressOf" operator

...is redundant, and I don't think will make code any easier to follow once people get used to it. But we'll see how it goes in practice.

I also think the term 'function pointers' may be causing some confusion, and suggest we start using the term 'function variables' instead. These are variables that can hold global functions or delegate methods or closures or lambdas or whatever. They can be stored in lists, arrays etc just like another other variable.

See: std::function in c++!


Danilo(Posted 2015) [#30]
Thx Mark! So it actually looks like the following, if I understood correctly (from the many snippets):
class C
   method test:int()
      return 1
   end
end

function test:int()
   return 2
end

function main:void()
   local x:int()   ' function variable / method variable
   local y:int     ' int variable

   local c:=New C()

   x = c.test()  ' assign function variable / method variable
   y = c.test()  ' direct   method call
   y = x()       ' indirect method call

   x = test()    ' assign function variable
   y = test()    ' direct   function call
   y = x()       ' indirect function call
end

So it looks like Monkey2 (do you want to remove the 'X' from MonkeyX?) will differentiate itself
if you use a method or a function with a function variable?
Because internally in C++ it's a difference, the hidden 'this' pointer.


marksibly(Posted 2015) [#31]
Almost - the assignments to x wont work/compile because in both cases you're trying to assign an int (the value returned by the test functions, which is called since you added '()' ) to the function var x. You were probably after this:

x=test 'assign function to function var
x=c.test 'ditto (although it's really a method - but who cares?)

So you (almost) can't get it wrong anyway as the compiler will catch you, eg:

y=test 'wont work/compile - attempting to assign a function to an int var!

Basically, '()' means a function gets called, and everything else is a variable or value - if you consider declared methods and functions as values (which they are internally).

I dunno, it's pretty clear to me but I think it might be a little 'different'. Adding something like '@' is an option if people struggle with it, eg:

x=@...
x=@test

But IMO, that makes something that is actually pretty normal code (assigning a value to a variable) look more exotic than it really is.

> differentiate itself if you use a method or a function with a function variable?

This is all done behind the scenes when you assign an actual function or method to a function var. It automatically creates the 'right' kind of function var for you, and stores 'this' and any other hidden info in it if necessary.


Danilo(Posted 2015) [#32]
Thanks for your explanation, I think it's pretty clear and readable:
class C
   method test:int()
      return 1
   end
end

function test:int()
   return 2
end

function main:void()
   local x:int()   ' function variable / method variable
   local y:int     ' int variable

   local c:=New C()

   x = c.test    ' assign function variable / method variable
   y = c.test()  ' direct   method call
   y = x()       ' indirect method call

   x = test      ' assign function variable
   y = test()    ' direct   function call
   y = x()       ' indirect function call
end

It just means that variable names and function/method/property names can't be the same, otherwise
things like "x := test" and "x := c.test" would collide (variable/property or function/method delegate?).
But that shouldn't be a big problem generally. Such collisions could happen if we import a new module
and one of our existing variable names collides with a function name in the external module.


abakobo(Posted 2015) [#33]
I'm quite confused.

will the equivalent of all these ANSI C pointer commands will be available? Will we be able to extract the adress of a variable(functions included) and use arithmetics to point?

//This is ANSI C code reference! No monkey code here.

void * ->  Generic pointer type
NULL ->  Null pointer (GC panic in monkey?)
type *name ->  Declare pointer to type
type *f() ->  Declare function returning pointer to type
type (*pf)() ->  Declare pointer to function returning type
*pointer ->  Object pointed to by pointer
&name ->  Address of object name
pointer -> the pointer itself (is more or less an int)
pointer=5000 ->generate a warning:initialization makes pointer from integer without a cast



GW_(Posted 2015) [#34]
Pointers are in, along with reference arguments to functions.
Nearly all of you examples are possible with Blitzmax, and It seems like M2 will match and exceed Blitzmax's ability to use pointers.


ziggy(Posted 2015) [#35]
Mark please,
if you're finally going for the Update:Int()() syntax, please consider leaving this out for properties, so they can still look like fields.
This looks very weird:
myObject.X()+=1



ziggy(Posted 2015) [#36]
@abakobo: Monkey as a managed language is based on references. That is, when you need to use an object, you access it by using a reference. Each reference is indeed a pointer, with the given plus that object lifetime can be handled automatically by the language runtime. In this scenario, arithmetic over pointer "address" is not always possible and type safe. If you need to make weird things on memory blocks you can use a databuffer, use arrays, etc. This ensures type safeness and allows the GC to handle objects lifetime, being the only exception objects that wrap limited resources (as VRam, etc). those ones need to provide an explicit destructor (discard or free method) to free resources early (before the GC reclaims them, if it ever does!


marksibly(Posted 2015) [#37]
> It just means that variable names and function/method/property names can't be the same, otherwise
things like "x := test" and "x := c.test" would collide

You should get exactly the same 'collisions' as you do for calling functions, as it's pretty much the same logic internally. So basically, if you can call it, you can assign it to something.

> if you're finally going for the Update:Int()() syntax, please consider leaving this out for properties,

Not sure what you mean here - can you post a fuller example? I haven't given much thought to how properties work with this...


ziggy(Posted 2015) [#38]
Well, consider this a function:
Function Whatever:Int(param1:String)

When you reference its address, you do it be function name:
Field myFunctionPointer:=Whatever

Now, if you classify Properties to be methods, given this getter and setter:
Method Position:Int() Property
Method Position:Void(value:int) Property

Any getter usage would require to be written with a () appended, to differenciate it from a pointer to its associated method, and the same goes for the setter
myInstance.Position() = 45  'This looks very weird
Print "current position is " + myInstance.Position() 'That's weird too, makes it look like a method
meInstance.Position()+=5 'This looks syntactly very weird.

I would get rid of any pointer to a property, in order to maintain a cleaner syntax. Properties are meant to provide a clean syntax for getter/setters, making code easier to read, while keeping a nice level of encapsulation and control of internal class attributes changes. I think it would be very sensible to keep a syntax that expresses this too.

This is even more important if we finally get a better syntax to declare properties as you suggested somewhere else:
Class C
   Property Foo:Int
      Field foo:int   'can be accessed ONLY by members of this class via Foo.foo
      Getter
         Return foo
      Setter
         foo=value
   End
End 

Don't force people to use Foo as Foo() for both getter and setter, to differenciate it from a potential getter or setter method pointer declared as Foo without parenthesis.
EDIT2: By the way, I love the idea of having a private field inside property. It adds an even nicer level of encapsulation. I've always wondered why .Net does not allow it.
EDIT:
By the way, if we use function names to express a function pointer, how would overloading be solved?


Nobuyuki(Posted 2015) [#39]
he means that he wants the calling syntax for Properties to resemble this (old Monkey syntax here, ignore the definition code in MyClass):

Class MyClass
Private
  Local _a:Int
Public
  Method A:Int() Property
    Return _a
  End
  Method A:Void(value:Int) Property
    _a = value
  End 
End Class

Function Main:Int()
  Local Foo:=New MyClass()

  Foo.A = 42   'Notice no parenthesis to call member A
  Print (Foo.A)  
End 

Obviously this syntax would have to be amended to support parenthesis if you plan on incorporating multi-param properties (for tuples, swizzling, etc), but until then, this is the expected behavior for Properties for most people, and does pass trans in Strict mode (as it should!). The calling convention is identical to VB.net.

Edit: Whoops, beaten to the punch...


marksibly(Posted 2015) [#40]
> Any getter usage would require to be written with a () appended, to differenciate it from a pointer to its associated method, and the same goes for the setter

Ah, OK.

Monkey2 will be much stricter about properties - you wont be able to 'call' them like methods (unless they hold a function!) if you feel like it the way you can in monkey1. They will always have to be used like fields. I don't think we really have a choice here for the reasons you've pointed out. And I agree it's better that way.

> myInstance.Position() = 45

Whatever happens, that wont!

> By the way, I love the idea of having a private field inside property.

I've actually gone off this idea a bit after having looked through some of my 'real code', but more on that later.

> By the way, if we use function names to express a function pointer, how would overloading be solved?

Very trickily! I've got it mostly going, and it works internally by using a 'FuncListValue' in place of a FuncValue if a function is overloaded. Only when a conversion or invokation happens is the overload resolved into a unique FuncValue. Conversions happens when something is assigned or passed as a parameter etc. However, when you're coding, you can just think of a funclist as a single function.


Samah(Posted 2015) [#41]
@marksibly: Monkey2 will be much stricter about properties...

Strict is good. Needing to add the Strict keyword is evil. Please remove it. XD

Also, slightly off topic, can we please have static properties? It would make singletons look much neater rather than requiring parentheses on the getter function. Non-compiling example from Monkey X:
Class Foo Final
Private
	' singleton instance
	Global instance:Foo
	
	' private constructor to prevent external instantiation
	Method New(); End
	
Public
	' static property to auto-create the singleton
	Function Instance:Foo() Property
		If Not instance Then instance = New Foo
		Return instance
	End
	
	Method Hello:Int()
		Return 10
	End
	' dragons be here
End

Function Main()
	' as a property
	Foo.Instance.Hello()
	
	' not as a property
	Foo.Instance().Hello()
End

Subtle difference, but I think I prefer the property access.