Callback problem which is impossible to solve

BlitzMax Forums/BlitzMax Programming/Callback problem which is impossible to solve

LordChaos

(Posted 2009) [#1]

Hi,

I'm thinking about a problem for 2 days already and it looks like there is no solution to this problem. Not in BMax and not in C++ (maybe in C#, I don't know.)

I'm designing a script language and a virtual machine which should actually run bytecode. I also want to use the WinAPI with that language.

The problem I have:
Functions like CreateWindowEx require a callback address. Of course, I can't pass an address with my script function / vm bytecode to these functions. So I need to setup a wrapper function in my VM which I pass to the WinAPI functions and which run the actual VM code.

But there is one question: How do I keep track of the actual code I need to execute? These callbacks can appear, of course, totally random and I also want to have as much callacks as I need.

It would be easy with method pointers/(maybe delegates in C#)

To show some code:

(no actual code from the projekt)


Type tWrapper

 Field vm_func_address:Int

 Method Callback(arg1:Int, arg2:Int, ...)

   PushArgs(arg1, arg2, ...)
   RunVMFunction(vm_func_address)

 End Method

 Function AddCallback:Byte Ptr(vm_func_address)

   Local n:tWrapper = New tWrapper

   n.vm_func_address = vm_func_address

   Return Byte Ptr(n.Callback()) ' <- big problem, method pointers do not work for known reasons

 End Function

End Type

I appreciate any hints. I am desparing of it right now!

markcw

(Posted 2009) [#2]

~~What about Lua?~~

LordChaos

(Posted 2009) [#3]

I have a serious problem which I would also like to be taken seriously. I'm not designing my own scripting language because I am not aware of other ones.

grable

(Posted 2009) [#4]

Why dont you use the lparam (user data parameter) that most (if not all) win32 api callbacks have?
Passing in the address of you vm function, and using a generic stub for each api..

That is essentially what you posted, except it must be a function marked with "win32" instead of a method ;)

Of course your vm should be re-entrant or give callbacks their own contexts, as the api call will block while doing work and calling its callback.

Brucey

(Posted 2009) [#5]

Passing in the address of you vm function...

Or an object, from which you can call a method...

LordChaos

(Posted 2009) [#6]

The problem is more complicated. I'm designing a general purpose scripting language.

I need to support _any_ possible function callback which can be required by an external DLL function. In my language I can pass any kind of function around as a parameter.

The WndProc was just an example.

It could look like the WndProc

WndProc:int(hWnd:int, Msg:int, lParam:int, wParam:int)

(note that I am using ints all the time and do casts later).

Or like this:

AnyCallback:int(Random1:int, Random2:int)

Having a wrapper function, I am already limited by a ceratain number of parameters, but that is ok. But I can not do something like that:

WndProc:int(ref:Object, hWnd:int, Msg:int, lParam:int, wParam:int)

How should the WinAPI know? It would put the hWnd in the ref argument, the Msg argument in the hWnd argument etc.

It seems like there is no way to track from which external function the callback appeared. And without knowing from where it appeared, I can't execute the correct vm function.

grable

(Posted 2009) [#7]

You still need to wrap your state somehow.

Only way i can see is either using a global state (like MaxGUI does, with lists of all windows) or
you have to use each api's "user data" placeholder.

In the case of WndProcs, you can use GWL_USERDATA (SetWindowLong/GetWindowLong) to associate data with a window handle.
But its not perfect as components and super classes them selves can be using it already.

Next up is SetProp/GetProp which also associates data with a window handle.

It seems like there is no way to track from which external function the callback appeared. And without knowing from where it appeared, I can't execute the correct vm function.

Thats what the lparam of all callbacks is for...

Htbaa

(Posted 2009) [#8]

Haven't fully read the topic but I've been busy with callbacks today as well whilst wrapping up the server part of my XML-RPC module.

I found out that, and it's quite useful, that I can create a function pointer and pass it to a external C library (XMLRPC-EPI in this case). XMLRPC-EPI has a build-in callback type which is used to register functions to the XML-RPC server. By passing the function pointer to C I don't have to care about the amount of parameters my BlitzMax function requires.

So instead of that the callbacks are being called from within BlitzMax, they're called from within the C library.

Here's some code for passing the function pointer.

xmlrpc.h:

// The callback type
typedef XMLRPC_VALUE (*XMLRPC_Callback)(XMLRPC_SERVER server, XMLRPC_REQUEST input, void* userData);

int XMLRPC_ServerRegisterMethod(XMLRPC_SERVER server, const char *name, XMLRPC_Callback cb);

wrapper.bmx:

Extern "C"
Function XMLRPC_ServerRegisterMethod:Int(server:Byte Ptr, name:Byte Ptr, cb:Byte Ptr)
End Extern

myapplication.bmx:

Function MyTestFunction(a:Byte Ptr)
   DebugLog "test"
End Function

XMLRPC_ServerRegisterMethod(server, name, MyTestFunction)

When the XML-RPC Server calls the registered function it uses the MyTestFunction pointer. The callback definition in xmlrpc.h actually expects 3 parameters, but they can be omitted in the BlitzMax function.

Not sure if this is of any help to you. But thought I should share it.

About passing any amount of parameters of any type, how about the stack principle that's being used by Lua? Works great.

LordChaos

(Posted 2009) [#9]

Ok sorry guys, it looks like I am very bad at describing problems. Here's another try:

I have no problem calling any kind of DLLs using a self-written wrapper function. This works great!

The problem:

As my language allows passing functions as parameters, it also allows passing functions to external DLL functions. Passing functions is no problem when:

- I am passing a function of my script to another function of my script
- I am passing an external function to another external function
- I am passing an external function to a script function

It becomes a problem in this situation:

- I am passing a script function to an external function

As described in my previous posts, I can't simply pass a pointer to my vm bytecode to the DLL function. The DLL function requires the address of a true machine code function which then would execute the script function.

I hope that clears up the case a little bit! My problem is about a general implementation for callbacks. It has nothing to do at all with the WinAPI. This was just used as an example.

Htbaa: Thank you for sharing that information. Unfortunately it does not help me.

Dreamora

(Posted 2009) [#10]

You can not use BM functions as callbacks. if they are ever called from the outside, the GC will blow up as it does not tolerate any concurrent acces.

This might be possible to circumvent through the new multithreading app "thread safe" GC. You would need to test that.

But without this, there is definitely no way. BM function pointers within the regular non threaded build are for use within BM only.

Brucey

(Posted 2009) [#11]

If the application is running in a single thread, it isn't a problem - hence, no concurrent access.

Dreamora

(Posted 2009) [#12]

Yes if.
But if you hand it out to a dll that runs in its own thread, it will end as concurrent access and blow it up, especially on multi core systems.
But yes, potentially less likely a problem in this case.

As for a solution attempt: how about a wrapper function that the object / data to work on from the vm and then calls the corresponding script end functionality? Kind of VM DMA if you want to call it like that.

Generally thought I would say that your design is just flawed seriously.
If your system has any reason to expose OS -> scripting, there will be trouble ahead.
Scripting is meant to work within clean defined borders so OS -> sandbox / application -> scripting
in that case you also wouldn't have the problem you currently are facing, as the application would work as a management layer with all the data required in the callback present.

LordChaos

(Posted 2009) [#13]

Dreamora: I _never_ experienced _any_ problems with callbacks. And that is also not my problem. I would like to stay on topic.

Nate the Great

(Posted 2009) [#14]

He isnt talking about BMax functions being passed to external functions, he is talking about his functions in his script being passed to external functions. right?

I honestly cannot think of how to get around the problem you mentioned though. Sorry

LordChaos

(Posted 2009) [#15]

Yeah thats right! You got it, Nate the Great!

Sadly it really looks like an impossible task. And I can not imagine any workaround. :(

Dreamora

(Posted 2009) [#16]

VM stuff does not exist for outside exposure unless you have some very specifically written VM that basically compiles bytecode external access wrappers. You can only expose real bytecode, which therefor would be BM stuff if you work with BM (unless you pass C function pointers from extern naturally).

In either case, see above posting, edited it.

LordChaos

(Posted 2009) [#17]

Dreamora I wrote about the idea of a wrapper function in my very first post (even gave a code example).

Telling me to use a wrapper function know does not help me at all! The problem is that it is not possible to keep track of the actual vm function which the wrapper should execute.

And, excuse me, IMHO my design is NOT "flawed seriously".

TaskMaster

(Posted 2009) [#18]

Could you not remember what script function was to be the callback, then give a BMax function as callback? Then when the callback occurs, have the BMax function call the script function?

I am afraid I just don't completely understand your problem?!?!

LordChaos

(Posted 2009) [#19]

No I can't remember the script function, that's the whole problem.

The only solution I can imagine are method pointers. But they don't work in most languages.

marksibly

(Posted 2009) [#20]

Hi,

You will need to pass come user data around (which is what 'delegates' really do anyway).

Whether or not this is supported is up to the API whose callback you are setting.

Some SetCallBack style functions take both a function AND some user data. Your function is then later called with your user data.

In the case of a scripting language, the user data would be something like a string containing the script name, and the function would be a simple 'dispatch/stub' function that invokes the script.

In the case of callbacks that don't allow you to specify user data, you're out of luck - unless you can somehow 'key' the user data via one of the callback's parameters.

For example, the WndProc callback doesn't include any user data, but the HWND can sometimes be used (via a map, or perhaps GetWindowLong GWL_USERDATA) to associate user data with a window. Ditto, maybe instead you want to key off the 'message' param...

Perhaps you should explain in more detail the actual problem you're trying to solve?

LordChaos

(Posted 2009) [#21]

Hey Mark,

the problem is that I don't know how to pass the user data around.

As said, I want my scripting language to have support for any general kind of callbacks like in a true compiled language (not just support for the WinAPI).

Let's assume this script:

Extern "some.dll" SetCallback(void Callback(int arg1, int arg2))

void Func(int num1, int num2) {
 doSomeStuff(num1, num2)
}

SetCallback(Func)

In the VM I now need to pass a wrapper function to 'SetCallback' which then calls 'Func'.

The question is how my wrapper function knows which script function it should execute.
For the wrapper function the contents of it's arguments (in this case num1, num2 respectively arg1, arg2) are completly random. But it somehow needs to know that it should push the first two arguments it gets onto the stack and then execute 'Func'.

grable

(Posted 2009) [#22]

Since it has to be fully generic, i would generate the machine code in stead.
That way you can encode your vm state directly into the stub.

edit: I just had to try this, since i wrapped gnu lightning a while back hehe. Took me a few minutes and looking at the asm output, its so simple you could easily roll your own generator ;)

LordChaos

(Posted 2009) [#23]

That's a pretty interesting approach, grable!!!

I had the idea yesterday to eventually do the wrapper function in C++0x by using the new closure possibilites. Unfortunately there's no implementation in GCC yet.

I'm very hopeful regarding your solution. But to be honest I don't know x86 assembly at all. It would be very kind of you when you could post some source example. :)

grable

(Posted 2009) [#24]

It would be very kind of you when you could post some source example.

Sure, no problem :) first you will need grb.lightning.mod.x86.rar, its x86 only for the moment. (and im still on bmx 1.30, so you might have to recompile)

Here is the example, generating a stub for a very simple vm interface.
Note that i had to add a few extra functions to gnu lightning for stdcall compatibility.

Import GRB.lightning

Local stub:Byte Ptr = GenCallbackStub( $FF, 3)
Local func:Int( a:Int, b:Int, c:Int) "Win32" = stub

Print "result=" + func(1,2,3)


' vm interface
Function VM_Push( value:Int)
	Print "push " + Hex(value)
EndFunction

Function VM_Call( addr:Int)
	Print "call " + Hex(addr)
EndFunction

Function VM_Pop:Int()
	Print "pop"
	Return -1
EndFunction

Function GenCallbackStub:Byte Ptr( vmaddr:Int, numparams:Int, stdcall:Int = True)
	Local code:Byte Ptr = MemAlloc(256)
	Local param:Int[numparams]
	
	' setup prolog & parameters
	JIT_set_ip(code)
	JIT_prolog(numparams)
	For Local i:Int = 0 Until numparams
		param[i] = JIT_arg_i()
	Next
	
	' call VM_Push for each parameter, in reverse
	For Local i:Int = numparams-1 To 0 Step -1
		JIT_prepare(1)
		JIT_getarg_i(JIT_R0, param[i])
		JIT_pusharg_p(JIT_R0)
		JIT_finish(VM_Push)
	Next
	
	' call VM_Call
	JIT_prepare(1)
	JIT_movi_i(JIT_R0, vmaddr)
	JIT_pusharg_p(JIT_R0)
	JIT_finish(VM_Call)
	
	' call VM_Pop
	JIT_prepare(0)
	JIT_finish(VM_Pop)	
	
	If stdcall Then
		JIT_reti_bs( numparams * 4)
	Else
		JIT_ret_bs()
	EndIf
	
	JIT_flush_code( code, JIT_get_ip())
	Return code
EndFunction

The resulting code would look something like this in bmx:

Function Callback:Int( a:Int, b:Int, c:Int)
  VM_Push(c)
  VM_Push(b)
  VM_Push(a)
  VM_Call(vmfunctionaddress)
  Return VM_Pop()
EndFunction

As you can see, its not all that much going on.. What stands out is the VM_Call() with the embedded vm function address.

Mahan

(Posted 2009) [#25]

There has already been lots of good advice here in the thread, so I'll just add yet another place to look for code that does what you need:

http://sourceforge.net/projects/jnative

jNative is a Java library for calling DLL's and even generating the DLL-calls in runtime. jNative supports callbacks in the DLL's aswell.

The part that I think might be interesting for you is the native part (.DLL/.so) included in jNative that makes this possible, and its complete source is also available in the project.

Browse the C++ source online here:

http://jnative.cvs.sourceforge.net/viewvc/jnative/JNativeCpp/

LordChaos

(Posted 2009) [#26]

Thanks for the hint Mahan.

Luckily grable just showed the perfect solution to my problem! Thank you so much. :D

Still need to have to look at it in detail, but the sample you posted is exact what I am needing. AWESOME!