BlitzMax Long values, Lua, and C...

BlitzMax Forums/BlitzMax Programming/BlitzMax Long values, Lua, and C...

LT(Posted 2015) [#1]
I apologize if this has been covered somewhere, but I just can't find it! I've been using LuaJIT's FFI to wrap Blitz functions and have come across a problem with long values that I've been unable to solve thus far.

If I do this...

C code:
__declspec(dllexport) int64_t getalong() { return 20000000000L; }

Lua code:
ffi.cdef[[
int64_t getalong();
]]

local LV = ffi.C.getalong()
local result = LV + 1LL
print( "Type: " .. type( result ) )
print( "Result: " .. tostring( result ) )

It works - giving me these results...

Type: cdata
Result: 20000000001LL

But if I do this instead...

C code:
int64_t bb_getalong();
__declspec(dllexport) int64_t getalong() { return bb_getalong(); }

Where the BlitzMax extern looks like this...

Blitz code:
Extern "C"
  Function getalong:Long()
End Extern

Function getalong:Long()
  Return 20000000000:Long
End Function

Using the same Lua code...it crashes and burns!

Does anyone know what I'm doing wrong here?


Brucey(Posted 2015) [#2]
Shouldn't you declare bb_getalong() in your C code as "extern" ? I'm pretty sure you also don't need "__declspec(dllexport)" in your C...
And you don't need to declare the Extern part in Max as that is where your actual function is.

Anyway, there are issues returning Long in BlitzMax, as far as external code is concerned. I would try passing it back by reference instead..
void bb_getalong(int64_t *);



LT(Posted 2015) [#3]
Shouldn't you declare bb_getalong() in your C code as "extern" ?
I am doing that - just didn't type it out. Of course, it wouldn't have worked at all without that... :p
I'm pretty sure you also don't need "__declspec(dllexport)" in your C
The LuaJIT FFI doesn't work without it - don't know why.

Thanks for your suggestion, I will try that! :)


Yasha(Posted 2015) [#4]
This is something I have no experience with, but is that really the right syntax to export `getalong` from your Max file? It looks to me like that's importing a C definition and then immediately overriding it with a Max one. 'Cause I've seen code that looks like:
Function getalong:Long() "C"
    Return 200000000000:Long
End Function

...floating around these forums on DLL-related topics before.


LT(Posted 2015) [#5]
Well, I don't know of any other way to get the FFI to "find" it. The Blitz function definition doesn't work, by itself.

EDIT: To tell the truth, I've no idea what that "C" does.


Floyd(Posted 2015) [#6]
This looks like a working example of BlitzMax to C, with 32-bit integers.

http://www.blitzbasic.com/Community/post.php?topic=83635&post=943458


LT(Posted 2015) [#7]
Thanks, Floyd, but this is specifically a problem with longs(64-bit integers). BlitzMax's handling of them is different.

---

Brucey, your suggestion worked for the example I presented above, but unfortunately not for my actual use case. /sigh

What I'm really trying to do is create C functions that allow LuaJIT access to the various arrays (via FFI). I have all of the array types working, except Long[]. This is what it looks like, using your suggestion.

C code:
void bb_getindexlong( BBArray *array, int index, int64_t *val );
__declspec(dllexport) int64_t getindexlong( BBArray *array, int index ) {
  int64_t *v;
  bb_getindexlong( array, index, v );
  return *v;
}

Blitz code:
Function getindexlong( array:Long[], index:Int, val:Long Var )
  val = array[index]
End Function

I can only assume that accessing the array is the cause of my problem now. I'm not sure if this can be done without actually returning a de-referenced value.


Brucey(Posted 2015) [#8]
How about...
Function getindexlong( array:Long[], index:Int, val:Long Ptr )
  val[0] = array[index]
End Function



LT(Posted 2015) [#9]
How about...
Gives me an Exception Access Violation. :/


Brucey(Posted 2015) [#10]
The only thing that comes to mind is that "BBArray *array" is not good...

Where does your BBArray* come from?


Yasha(Posted 2015) [#11]
Your C code is wrong. Pointer `v` is uninitialized. Should be:

__declspec(dllexport) int64_t getindexlong( BBArray *array, int index ) {
  int64_t v;
  bb_getindexlong( array, index, &v );
  return v;
}

One other idea: if your code is BBArray-aware, you're already including blitz.h, so why not solve your specific problem by inlining the functionality you need and dropping the requirement for a Max function at all? i.e.:

__declspec(dllexport) int64_t getindexlong( BBArray *array, int index ) {
  return ((int64_t *)BBARRAYDATA(array, array->dims))[index];
}

(although I guess that rules out using bounds checks in debug mode)


LT(Posted 2015) [#12]
Where does your BBArray* come from?
It worked in the other cases and showed up looking correct in the debugger.

Pointer 'v' is uninitialized.
Right you are. Thanks, that fixed it! :)

Thanks for your responses. Looks like it's sorted out, now.

EDIT:
One other idea
And it's a good one, thanks! I've been trying to do as much as possible in Blitz, but certain functionality might be better off coming from C functions.


LT(Posted 2015) [#13]
although I guess that rules out using bounds checks in debug mode
Not sure that matters because the FFI function is wrapped in a Lua function and that checks bounds. :)


LT(Posted 2015) [#14]
Come to think of it, it would be nice to simply export Blitz functions without needing the C wrapper. Is that possible?

LuaJIT requires __declspec(dllexport) - this comes straight from Mike Pall - but shouldn't that also work for bb_getindexlong? Or am I just confused?


Yasha(Posted 2015) [#15]
It won't work for the declaration of a BlitzMax function because it won't be true. If the BlitzMax function was built without the annotation (which it was because it's a completely different language), then the effects of said annotation haven't been applied to it. Adding it to an external declaration after the fact is effectively lying to the compiler about what it should expect to find at that function's code site. It doesn't go back and retroactively change the code to fit the external declaration!


Brucey(Posted 2015) [#16]
So what does "__declspec(dllexport)" do to a C function on Windows, exactly? Is it an alignment thing or what?


Floyd(Posted 2015) [#17]
After some reading, it is apparently a Microsoft extension to C: https://msdn.microsoft.com/en-us/library/dabb5z75(VS.80).aspx

The dllexport means it is specifying how symbols are exported. Beyond that I don't know.

That reminds me, whatever happened with creating DLLs with BlitzMax? That was an experimental feature for a while.


Yasha(Posted 2015) [#18]
It adds more information to the generated binary so that the function can be looked up by name without needing to use a second mapping file of some kind; instead, the name table is right there in the library. Being a JIT, LuaJIT links to functions at runtime and thus needs a table mapping names (as seen in the .h, which it processes at runtime) to the binary code.

BlitzMax does build in name information, but I don't know if it's in any way related to the __declspec semi-standard; there's no indication given anywhere that it is.


grable(Posted 2015) [#19]
Its too bad blitzmax doesn export symbols in a standard way...

But you can build the export table yourself, include it in the execubable and then LuaJIT would find them. Or you could even LoadLibrary that same executable and use GetProcAddress.

I have a small tool i use for that exact purpose, it extracts marked functions from source code and builds either a .def file or an export table.
The export table is just some assembler that you can import in blitz.

Attached below if anyone needs it.

makedef.bmx

EDIT: updated on 10.10.15 to fix some parsing bugs relating with "type" and handling of sub functions.


LT(Posted 2015) [#20]
Hmm, looks interesting, but I was unable to get it working. Lua is still not finding the symbols.

EDIT: Nevermind, it works. Thanks! I was expecting to see a slight speed increase, however, and that didn't happen. Oh, well...


Brucey(Posted 2015) [#21]
What advantage are you expecting to see from integrating with FFI ?
I thought you couldn't mix FFI with Lua/C ? In which case how are you taking a BlitzMax-made Object and using it in FFI?

:o)


LT(Posted 2015) [#22]
FFI is much faster*. You can access C structs directly from Lua and the JIT inlines all of it. Until recently, I've been doing things with the C-API. I was aware of FFI, but not its full benefits until I did some digging on the LuaJIT mailing list, where I was promptly informed that I was doing it wrong. :/

You're right, you can't mix FFI with Lua/C and that's not what I'm doing. I have an export program that takes a bunch of Blitzmax function definitions and generates glue code - similar to Lugi, but I couldn't use it because I needed a lot more functionality. My glue code used to generate Function( luastate:Byte Ptr ) and now it generates C declarations, among other things.

A CData can have metatables attached to it, same as userdata. But unlike userdata, its metatable cannot be altered after it's attached - on account of the JIT compiler.

*Faster - UNLESS you're running on iOS, which has security precautions that prevent JIT from functioning. Also, both JIT and FFI are turned off on consoles, but that bridge is a long way off...


LT(Posted 2015) [#23]
To tell the truth, I really wish that Lua had some kind of static-typed companion language (other than C) that complimented its strengths and also allowed for cross-platform, native compilation to match LuaJIT. Lua's metatable stuff allows for easy creation of project-specific languages. You can hand off a custom language to a designer that knows what you mean when you type "local actor1 = RomanSoldier + sword" - how cool is that!?


Yasha(Posted 2015) [#24]
The problem with Lua is that you can pick two: fast, ahead-of-time, full table+metatable support.

LuaJIT gets a large part of its performance by making site-specific optimizations, so that table lookups become (after a few runs through a hot path) as fast as a C struct field (at least where they're used that way, anyway) and code handling numbers gets reduced down to native math opcodes. You can't do this ahead-of-time because the whole design of tables means you don't have enough information: offsets can (and do) change all the time, objects of different layouts can pass through the same code block, and objects can have different metatable features or change them or whatever. The same piece of code will change its meaning as the program runs.

You can compile the functions to C functions, the loops to C loops, the ifs to C ifs, etc., but that will only take you a small part of the way. With very few (trivial) exceptions, you won't get math or table accesses any faster than in the interpreter.


LT(Posted 2015) [#25]
offsets can (and do) change all the time
Yes, because it's dynamic.

fast, ahead-of-time, full table+metatable support
It occurs to me that some kind of metatable locking mechanism could be useful. So I'm willing to give up full metatable support for cases where I need to speed up the lookups, if that was an option.