Question about Import...

BlitzMax Forums/BlitzMax Programming/Question about Import...

LT(Posted 2015) [#1]
I've noticed that I can do this...

Import "one.cpp"

But not this...

Import "one.cpp"
Import "two.cpp"

Yet if...

#include "two.cpp"

...is inside of one.cpp instead, it works! The contents are externs and they each work individually. Is it correct for me to assume that it has something to do with conflicts in the symbols produced?

---

Also, is there any COMPLETE documentation on Extern available? Throughout the forum posts, I've seen people reference all kinds of things that aren't explained. Extern vs. Extern "C" vs Extern "Win32" are somewhat self-explanatory, but it's not 100% clear what they do - and maybe there are other forms I haven't seen...? Inside Extern, I see things like...

Function foo(p:Byte Ptr) = "_foo@4"

I understand that you can rename functions this way, even though it's not explained in the docs, but what the heck is that @4 part?

Just hoping there is some comprehensive explanation for this stuff somewhere that I haven't yet seen. :)


NOTE: The first case only seems to be a problem for me when I do it inside of a module. I'm aware that I could be misdiagnosing the problem, but since I don't fully understand how (or when) the Imports work... ;)


Brucey(Posted 2015) [#2]
This should work :
Import "one.cpp"
Import "two.cpp"

I'm not sure what you mean by "The contents are externs". From your filenames, the contents would be C++ source?


LT(Posted 2015) [#3]
Yeah, that's what I thought...no idea why it doesn't.

Each .cpp file has something like...

extern "C" {
void bb_func1();
__declspec(dllexport) void func1() { bb_func1(); }
}

...inside, so importing both should work as long as there are no name conflicts, right? For some reason, that doesn't work for me (crash!).

Like I said, if I #include one in the other, it works. However, this problem seems to exist across all modules. If I put one of the import calls in a separate module, it still fails even though the function is being filled in later.

Ultimately, putting them in separate modules is the goal.


grable(Posted 2015) [#4]
If it compiles fine it may be imported with wrong calling convention. It looks to be CDECL so should be using Extern "C".

And there is no need to use DLLEXPORT if your not planning to put the code in a DLL.


LT(Posted 2015) [#5]
I suppose you mean Extern "C" in Blitz? I've tried it both ways. The functions are for exposure to LuaJIT's FFI - so the DLLEXPORT is needed.


Yasha(Posted 2015) [#6]
The annotations for Extern do two things:

1) The "C", "Win32", etc. let the compiler know which calling convention to use for those functions in the generated assembly. "Win32" is _stdcall, "C" and "MacOS" are cdecl, and there's also "Blitz" and "OS" which I forget right now (I assume "OS" changes depending on whether you use Windows). I believe the default is "C".

2) The optional `= "name"` after function declarations tells the compiler what the assembly-name of that function will be. If it's left off it'll generate one based on standard conventions from the Blitz-declared name, but you can rename the function to anything you like for use in your code with this mechanism.

The "@4" is just part of the function's name as exported from the assembly. It does mean something, but only according to convention - the function could be named basically anything. Assembly names can contain many characters that aren't usually allowed in high-level languages like @ and $, so splitting the name up with these is a good way to attach extra semantic information that would otherwise have been lost. By convention, _stdcall functions tell you how many bytes of stack space their arguments will need with this number, but it's purely informational.

EDIT: try Extern "Win32". LuaJIT examines the assembly names at runtime to get argument information for the code gen.


LT(Posted 2015) [#7]
Thanks, Yasha. That is helpful. I'd never even heard of the "Blitz" and "OS" options. Would be nice to see this stuff added to the official docs!


Yasha(Posted 2015) [#8]
Also you don't need to write C++ wrappers for your Blitz functions if all you want to do is change the calling convention - you can annotate BlitzMax functions directly. Take a look at the generated assembly for:

Function a:Int(b:Int)
	Return b
End Function

Function c:Int(d:Int) "Win32"
	Return d
End Function

Print a(1)
Print c(2)



LT(Posted 2015) [#9]
Here's a bit more context. Trying to keep my explanations as short as possible... ;)

I'm trying to come up with a simple method for exposing public functions in a module to my own Lua module so that it can auto-generate glue code and make the functions callable from Lua. This is quite easy to do with the C API, but now I'm trying to do it with FFI (because it's faster).

Right now, the best I'm able to do is extern pure C functions, which always seems to work fine, OR extern Blitz function definitions which are later replaced. This part is working now. The problem that I've been having is that if the initial function definitions come from different modules, the program compiles okay, but the functions are not found! It works fine if I put them all in one cpp...but that's not really acceptable.

There's actually a Part B, since I'd like the functions to NOT be over-ridden (the externs are only meant for the FFI), but defined in the modules.


LT(Posted 2015) [#10]
The difference in the generated assembly doesn't mean much to me, I'm afraid. In any case, "Win32" is not working for me. :(

The C++ wrappers are there so that I can dllexport them for the FFI. Have not found another way that works.


grable(Posted 2015) [#11]
You might need to DLLIMPORT functions from other modules, though i suspect it wont do anything.

Maybe run something like http://www.nirsoft.net/utils/dll_export_viewer.html on the binary as well, to see which functions are actually exported.

And should probably use OllyDebug or similar to narrow down where it crashes too, figure out if it calls your function at all.


LT(Posted 2015) [#12]
I already know it doesn't call the function because LuaJIT doesn't find it - UNLESS I include one cpp in the other rather than importing them separately.

I want to emphasize, this ONLY happens in modules. In the main program, I can import multiple cpps without issue. Sort of takes away from the modularity of it all, though. :(


grable(Posted 2015) [#13]
Its just that i am unable to reproduce your problem, if i have understood it correctly that is ;)

Below is my sample project with 2 modules doing what you describe, one imports a single C file, the other imports two.
And the test imports both modules, calling the functions in different ways.

https://drive.google.com/file/d/0BzVLNZSckvfhc2FYWVZRdmgtSms/view?usp=sharing

Please look at it, and if my assumptions are incorrect fix it and reupload.


LT(Posted 2015) [#14]
First of all, thank you for putting in the effort. I have learned a thing or two...

I didn't know I should define the internal Blitz command starting with bb_ - thought that was something that the compiler did automatically, so that's useful! :)

It's a bit closer now, but I still can't get separate modules to play nicely together.

--

The example you provided gives me an EXCEPTION ILLEGAL INSTRUCTION on the t1() line (25). However, I'm not using LoadLibraryW in my code, so I'm not sure if that will affect my own progress.

Incidentally, the GetProcAddress, LoadLibraryA, and LoadLibraryW commands are not in my documentation at all! :(


Brucey(Posted 2015) [#15]
You can read all about GetProcAddress here : https://msdn.microsoft.com/en-us/library/windows/desktop/ms683212(v=vs.85).aspx

(it's friends are on the sidebar to the left)

:-)


LT(Posted 2015) [#16]
Thanks, Brucey. Still seems like those commands should be in the official docs somewhere since they are directly accessible from BMX.

For the moment, it doesn't seem to impact what I'm working on. I think I'd prefer to avoid Windows-specific commands. Are these commands also used in other OSes (for .SO files, perhaps)?


grable(Posted 2015) [#17]
I didn't know I should define the internal Blitz command starting with bb_ - thought that was something that the compiler did automatically

It does, but only for globally scoped functions defined in the main source. Functions defined in modules get the module path like so "MOD_SubMod_function".

Are these commands also used in other OSes (for .SO files, perhaps)?

Yes, the Linux ones are dlopen, dlsym and dlclose.
Other than some extra flags they are functionally equivalent.

If the test fails on t1() then GetProcAddress must have returned NULL, meaning it didnt find the exported symbol.
Even though it should, as in it worked on my machine (i know i know :p)

EDIT: Did you recompile the GRB.Bogus modules? It isnt stricly needed, but since im using gcc 5.1.0 it just might.


LT(Posted 2015) [#18]
Functions defined in modules get the module path like so

I had actually tried that before, but made the mistake of adding an extra underscore to the front because that's what I saw in the asm...

the Linux ones are dlopen, dlsym and dlclose

Thanks, and the MacOS ones? Also wonder is all this will work in NG and if there are equivalents in iOS and Android...

Did you recompile the GRB.Bogus modules?

Yes, same result.


grable(Posted 2015) [#19]
Im just guessing but since OsX is a fork of BSD it probably has dlopen and the rest.
iOS and Android i cannot say, though both use *nix kernels so dlopen should be there somewhere, whether you have access to it is another question.

I allso checked LuaJIT2 sources, and it doesnt do anything funky like walking the import table by hand, it uses dlopen and LoadLibrary too.

EDIT: That the test fails is puzzling to me really, it seems that your setup has some trouble with the export table, maybe its overwriting itself or something...
Could you pack the whole project compiled with your setup (including test.exe) so that i might see if there are any differences?

EDIT2: And what versions are you working with? BlitzMax, GCC, FASM, Windows.
Mine are:
BMX: 1.50
GCC: 5.1.0
FASM: 1.71.39
WINDOWS: Windows 7 32-bit


LT(Posted 2015) [#20]
At this point, my program is doing everything your example does. I can call the functions from both modules. But the ffi.CDEF doesn't find the functions unless I put all of the C source into one file. Splitting between modules is just not working.

EDIT: The exception on t1() only happens in debug mode.


LT(Posted 2015) [#21]
Ah, there are some differences...

BMX: 1.48
GCC: 5.1.0
FASM: 1.69.14
WINDOWS: Windows 7 32-bit

EDIT: I'm hesitant to replace something like FASM because I don't know if upgrading will cause something else with Blitz to break. I prefer to use the official released version. However, I did a temporary upgrade and got the same result (still using BMX 1.48, though).


grable(Posted 2015) [#22]
Hmm.. Doesnt seem to be that much of a difference, there isnt any major changes that i know of for BMX or FASM that would account for something like this.

Btw i just tacked this to the end of my test.bmx (using Zeke.LuaJIT2 module).
And it worked as expected!
WriteStdout "~ncalling LuaJIT.FFI~n"
Local l:Byte Ptr = luaL_newstate()
luaL_openlibs(l)
luaL_dostring( l, ..
	"local ffi = require('ffi')~n" + ..
	"ffi.cdef[[ void test1(); void test2(); void test3(); ]]~n" + ..
	"ffi.C.test1();~n" + ..
	"ffi.C.test2();~n" + ..
	"ffi.C.test3();~n" )

lua_close(l)


Why it fails for you though is a mystery :( I would ask you to create a minimal example that fails, but since my test already fails there is no need.

EDIT: The only thing left i can think of is the use of C++ and extern "C", even though it should behave like C it still might do something unwanted with the exported symbols.
But then my test should have worked, so ... yeah :/

If you could upload my test (modules and all) compiled on your end, it might help me narrow it down.


LT(Posted 2015) [#23]
Well, the extern "C" works fine as long as it's all in one file. Same goes for plain C.

I will try to post a zip. Never done it outside of email before. :P

EDIT: Here it is...
https://drive.google.com/file/d/0B2KB6oc77s12d19ySjRwbkJBWkk/view?usp=sharing


LT(Posted 2015) [#24]
Just so you know, the FFI stuff you tacked on does work for me. It's just the t1() call that's not working (and only in debug mode).

Why my own FFI calls are not working (for only one module) is still a mystery. :/


grable(Posted 2015) [#25]
Ive compared the files now, and the library files (*.a) only have a changed timestamp so are identical.

Your executable though is 38k smaller than mine, which is a bit odd to say the least. Do you by any chance strip symbols or sections in bmk?

It was allso compiled in GUI mode, which is why no output was printed, though it does run successfully.

All the symbols are exported normally, and after tracing through it in OllyDebug i saw they all were found by GetProcAddress and called fine.

So there should be no noticeable differences between our systems.

EDIT: The reason it fails in debug mode is because it loads "test.exe" not "test.debug.exe" ;) fix that and it runs.
Or alternativly use GetModuleHandle(Null) instead of LoadLibrary/FreeLibrary.


grable(Posted 2015) [#26]
I cant say more without knowing what you do in that specific module im afraid.

Could you by any chance post your project? Or at least enough for it to compile and run with the offending module.
If its sensitive i understand, though i promise not to abuse it and to delete it once im done.


LT(Posted 2015) [#27]
Do you by any chance strip symbols or sections in bmk?
Not doing anything special. It's just standard Blitz. But maybe there's a difference between 1.48 and 1.50?

it fails in debug mode is because it loads "test.exe" not "test.debug.exe"
Ha ha, duh! ;)

Thanks for looking into it. It's very much appreciated. I'll keep trying and post here if/when I learn anything.


grable(Posted 2015) [#28]
Happy to help, i hope you figure it out :)


LT(Posted 2015) [#29]
Learned something interesting.

Take that example you sent me and comment out everything between Import and the FFI part. I think you'll find that the FFI functions won't execute. It seems that the C functions have to get called first... Hmmm.


LT(Posted 2015) [#30]
The symbols do not appear to be exported if the module functions are not directly called. Seems this only affects FFI calls.

If I wrap the exported module functions in other functions, the symbols are available to the FFI. It works, but seems kludgy to me... :/


grable(Posted 2015) [#31]
Well that was certainly unexpected!

If i was to guess, it might be that the exported function gets compiled as a runtime trampoline.
But that still doesnt account for it not finding the exported symbol, which should be static either way.

Il have to look into that, i dont like this kind of behavior :(

EDIT: Ah, of course. It gets optmized away. Should have thought of that, just didnt think blitz was THAT efficient ;)


LT(Posted 2015) [#32]
Incidentally, I only had to wrap one function from one module and the rest suddenly worked. How odd!

EDIT: Realizing now that it's a matter of granularity. I think when it sees nothing is getting used, it just drops the whole thing. :)


grable(Posted 2015) [#33]
At least you dont have to call it, just reference one of them in a private function somewhere :)

I tried getting it to stick around by using __attribute__((used)) but its BCC that removes it.
I even tried using an assembly file that references it and even stored its pointer as data. Sadly nothing worked.

Im thinking BCC is a bit too agressive here. But then it does know 100% the function isnt called so i guess it does what it knows best.
Im betting though that this behavior is lessened when compiling a dll.

There should probably be some kind of flag for this, like NODEBUG. We could call it DLLEXPORT and get a double wammy ;)
Now maybe i have something to do with the opensource compiler hehe.


LT(Posted 2015) [#34]
Yeah, some kind of flag might be nice. Perhaps Brucey is reading this?

Putting it in a private function works, at least. :)