SWIG backend for BlitzMax (autogen wrappers!)

Community Forums/Showcase/SWIG backend for BlitzMax (autogen wrappers!)

Yasha(Posted 2014) [#1]
EDIT: now with prebuilt binaries!

Proudly presenting swig2bmx, an automatic wrapper generator for BlitzMax, built on top of the famous SWIG!

https://github.com/Leushenko/swig2bmx

You take a C++ library, you make a very small number of customizations to the headers, declare any special data types and their handlers (such as automatic conversion between Max and C++ strings), and it spits out thousands of lines and many megabytes of wrapper code like magic!

The test case (and justifying example) is Irrlicht, which required a whopping 12Kb of hand-written code to wrap (that's like three screenfuls, tops) against 50Mb of library. That's a great improvement on the manual-code-ratio of traditionally written wrappers!

You can download the resulting example Irrlicht wrappers and give them a play (for Mac and Windows) from here: https://github.com/Leushenko/swig2bmx/releases

swig2bmx uses SWIG to generate a C-style "flat" procedural interface to a C++ library, then reconstructs the OOP layer on the BlitzMax side from CFFI declarations. The end result looks pretty similar to code you'd write for a hand-crafted wrapper:



(you can use the procedural layer too if you're a masochist, but be warned that it's untyped - Byte Ptrs only)


At the moment the tool is seriously rough around the edges: the "script", for instance, is just a list of instructions for a human operator, not a proper automated shell script. I'll update it to a proper automatic script in due course and polish the edges a bit. Several of the files also require hand-correction (all included example files have already been corrected), e.g. you need to manually comment out three lines from irrlicht-base.bmx before it will compile.

Note that in the same way BMX files aren't intended for end users to see, this tool isn't for your average Maxer to run: it's for wrapper authors to use in getting libs out more quickly. It's quite possible nobody other than me and maybe Brucey will ever run this directly, and that's OK - the tool exists to make it more reasonable to ask another Maxer to wrap a lib for you! (TBH I'll be surprised if there's even a second person with the intersection of BlitzMax/Lisp/C++ needed to make this work...)

If you want to run it yourself you will need a *nix shell environment, a GCC-compatible compiler, SWIG 3, Gambit Scheme, and BlitzMax. Patience would be good, too (n.b. SWIG and the Gambit file are both very slow - may take a couple of minutes to finish compiling. I don't see this as a major problem). Do not under any circumstances attempt to open the generated .bmx files in MaxIDE unless you really like watching your system hang.


The .o and .a files provided are compiled for Mac OS X, if anyone wants to just jump in and test the Irrlicht example. Now with prebuilt downloads for Windows, Mac OS X, and Linux.


There are some limitations on the kind of C++ this is capable of wrapping, outlined in the readme (on GitHub), but "simple" or "clean" C++ - not too many templates, no silly games with operators, simple memory management, class-based structure - should be OK. I know for a fact that Leadwerks 3.1 is SWIG-compatible, and would be an excellent second practical example if/when it becomes available in 32-bit form (has it already? I stopped paying attention).


Brucey(Posted 2014) [#2]
Ooh :o)


Derron(Posted 2014) [#3]
Nice work (albeit not of use for me as I would prefer relying on Brucey doing that stuff for us :p). Keep improving it so you wrapper-guys can benefit from it.


quote from your github readme:

Overloading: Since BlitzMax does not support overloading, overloaded methods are
renamed. The wrapper makes two attempts to rename methods, by adding parameter
names, and then if that fails by adding their types instead; if both result in
a method that would clash with an existing export, the method is omitted.

In practice over 99% of methods can be exported in this way, although some of
their names may be a bit unpredictable.



Couldn't you avoid that clashes by adding an "_#" (# = number of same base names + 1) at the end? Of course this makes the names a bit hmm "odd looking", but this is how Mark "mungs" the names of variables and types/functions/methods ...


Maybe some kind of "predefined wrappernames" would be useable - a list describing how to call a specific function in the library:

DrawArea(pos,rect)##DrawImage(pos,rect)
Draw(pos)##DrawImage(pos)

So this would allow to keep compatibilities intact (people would not need to change codes that much).



bye
Ron


Matty(Posted 2014) [#4]
Some of you guys are so smart...


KronosUK(Posted 2014) [#5]
What Matty said..great work.

Should the irrlicht example work on windows or is it only OSX

I get this when I try and run irrtest-basic


Now find the most kick ass 3d engine(free) out there and wrap it to stop all those whiners moaning about the lack of a decent 3d engine for Blitzmax ;)


Yasha(Posted 2014) [#6]
The Irrlicht example is OSX-only at the moment. I don't have a Windows dev environment set up right now (I'll try to get on that soon).

I didn't name them very well... "-basic" is the one you probably don't want to use since it requires you to count _OVERLOAD_ numbers and use Byte Ptrs for everything. Look at "-obj" for the same example but in a native Max style.


Couldn't you avoid that clashes by adding an "_#" (# = number of same base names + 1) at the end?


That's basically what SWIG already does, and the style of name provided by the procedural layer. In practice I found it very difficult to work with that, and also note that the number doesn't provide any information about what the function actually does, so you have to be constantly checking the function list to confirm you got the right one.

What the object layer does is convert that like this - say you have these three C++ methods:

void SetPosition(Vector3 pos) {...
void SetPosition(float x, float y, float z) {...
void SetPosition(int x, int y, int z) {...


...it will generate these three Max methods:

Method SetPosition(pos:Vector3) ...
Method SetPositionWithXYZ(x:FLoat, y:Float, z:Float) ...
Method SetPositionWithIntIntInt(x:int, y:Int, z:Int)...


The first one gets dibs on the original name, the second one gets a name expanded with what parameters it takes, and the third one unfortunately used the same parameter names, so it falls back to specifying the types instead. (You have to hope the library authors declared them in a good order... if any of the overloads is nullary though it always gets to use the real name.)

In practice Irrlicht with its many thousands of functions only had about five functions that couldn't be exported this way and were omitted from the OOP interface, and they were all pretty obscure (I think only a dozen or so needed to use the fallback, too).

Hopefully a good IDE can help with any difficulties renaming might cause. (You can also rename methods to whatever you want by hand after the interface is generated.)


Derron(Posted 2014) [#7]
Ok, think I just misinterpreted that "omit" in

if both result in
a method that would clash with an existing export, the method is omitted.


in a way that a non-resolveable nameclash results in the method not being translated/wrapped.


@increasing numbers "_#"
I know that this then adds confusion what method to use, and so on ... it was just to make it "wrap"-able.

@renaming templates
Like said, I just thought about this to make "api breaking changes" transparent to the enduser. - Of course you might have to add some other wrapper functions in all cases, so this "renaming template" is more or less a tedious homework for the wrapper which results in no savings.



@3D-engine
Another thing would be a good replacement for the audio backend (streaming ogg playback seems to need some kind of "pointer" for a data-buffer-fill-function which isnt provided by freeaudio).


@Yasha
You have to extend your signature :p.


bye
Ron


Brucey(Posted 2014) [#8]
streaming ogg playback seems to need some kind of "pointer" for a data-buffer-fill-function which isnt provided by freeaudio

Maybe someone can ask skid nicely? :-)


Derron(Posted 2014) [#9]
Hmm considering the action in this forums it is like trying to find a sober guest in a shabby bar at 5 am :D (just check the other "unanswered" topics).

Maybe we can get his attention again when some blitz-evolution happened...


bye
Ron


Yasha(Posted 2014) [#10]
OK, I have totally reorganised the project (so it's less of a complete mess now). Also made a couple of minor improvements.

It is now usable; the script can now be run to automatically generate an Irrlicht wrapper by itself. Instructions are included (although all you really need to do is download/unzip/build).


For people who aren't interested in that, there are also now separate downloads of the prebuilt Irrlicht example wrapper for Windows, Linux, and OSX, found here: https://github.com/Leushenko/swig2bmx/releases . Just download and play.

So, what'll be our next trick? Leadwerks?


Derron(Posted 2014) [#11]
That "proxy object"-thingie might be of interest for Brucey (concerning memory blocks registered by LUA or other "external code" not maintained by the BlitzMax GC).

@releases
Do you happen to know how to "misuse" the tagging system to generate some kind of "download links" (source + _one_ of the samples?) without tainting the master with the samples/binaries ? Would be a nice way to include "prebuild" things in the github repo ... maybe one could create another repo (a "submodule", as submodules are not automatically fetched when fetching from a git repo) containing that binaries ... and tagging is then the version number of a binary ... hmm.


@Leadwerks:
I think it is upon YOU what to wrap next - each project will "generalize" your scripts more to handle more and more specialities of c-code :D


bye
Ron


KronosUK(Posted 2014) [#12]
I'll certainly be looking at the irrlicht wrapper for windows. I would love to see a full wrapper of Frank Dodds Irrlicht Wrapper to go with this. It simplifies Irrlicht a bit and has alot of useful functions in it. Aqualungs Blitzplus wrap was ok but incomplete.

What about a wrapper for a physics library like Bullet.

Seems to me Josh should commission you to provide a Leadwerks wrapper as it will be of direct benefit to his business. I should imagine it would need regular updates also.

Is there anyway your process could output some kind of help file listing all the wrapped functions and formats?

For instance in the Irrlicht example I am seeing things like:-

Dimension2du.MakeWithWidthHeight(Varptr(x), Varptr(y)
node.SetMaterialTexture 0, driver.getTexture(PathFromString("media/sydney.bmp")
cam.SetPosition Vector3df.MakeWithNxNyNz(0, 30, -40)

Its going to be difficult to figure out what form to take with expressions without some kind of a guide. Perhaps split the xxx_interface.bmx file into separate files for each type . Currently its so huge it takes time to search for things.


Yasha(Posted 2014) [#13]
@Derron that's pretty much what the "releases" tab does, isn't it? The binaries aren't in the source tree any more.


@KronosUK
Its going to be difficult to figure out what form to take with expressions without some kind of a guide.


I think the best option here is to use an IDE with autocomplete. BLIde and indevIDE both offer autocomplete; that way you can start typing in the function name as it would appear in C++, and see what the IDE offers you to match the arguments you want to put in.

In my experience so far it's only MaxIDE that chugs on the large interface files. Other editors (I've been working with jedit) are perfectly responsive, and in fact can handle files ten times that size without slowing down at all. So if you do need to search the interface manually, just make sure to do it in a different text editor.


A simplified and modified wrapper like Frank Dodd's is a nice idea, but would prevent using the original Irrlicht documentation. At the moment, once you get your head around the way methods are renamed (new -> Make, overloads -> *With, operators -> Add/Mul/etc), everything matches up 1:1 with the C++ documentation (and non-overloads are actually identical). I think that's the best goal to pursue, otherwise we're getting out of the territory of 1:1 automated matchups.

I've also gone back and fixed the lack of support for const& arguments/returns, so you don't need that embarrassing Varptr any more.


Derron(Posted 2014) [#14]
I thought that releases-one is mixing into the git-history so people "cloning" a repo will fetch that bunch of binaries too .. think I was wrong with my assumption.

I have read now, you can even request the download count using the github api ... hmm I might replace my games download area that way ... thanks for that hint :D


@maxide
Had that blocked MaxIDE also on debug builds crashing ... if it has to load big files it stops responding sometimes (have to "kill maxIDE" then + "kill bmk").


@varptr
The only one enjoying it like a raving lunatic is Brucey :p.


bye
Ron


Brucey(Posted 2014) [#15]
@embarrassing Varptr
Very :-p
Always nicer when you can hide away implementation things like that.


KronosUK(Posted 2014) [#16]
I'm having some problems with this simple code



I'm getting

"Compile Error: Identifier 'SetRotationDegrees' not found"

on the line indicated above.

Edit1:

I think I get whats happening. Whats the correct way to create an empty Matrix4?

Edit2:

Cracked it. Doing it like this seems to work




Who was John Galt?(Posted 2014) [#17]
Really nice work here Yasha, and very useful.

Cheers


KronosUK(Posted 2014) [#18]
Another question.

I note some of the functions for retrieving strings actually return a byte ptr. How do I get the string value?

eg

Node:iscenenode

Node.getname()


Henri(Posted 2014) [#19]
Something like:
Local s:String = String.FromCString(b:Byte Ptr)


-Henri


Yasha(Posted 2014) [#20]
Yes, Henri is correct.

The wrapper generator inserts some automatic conversions between Max strings and Irrlicht strings, but in this case the original function returns a C-style char*, not an Irrlicht string, so the wrapper can't return a true string either.

My reasoning is that an automatic conversion between Max strings and char* is a very bad idea, for multiple reasons (memory management, the fact that char* is a very generic type, obscuring the intent of the library authors, etc.). For instance, you'd need to read the Irrlicht documentation to know whether it's your responsibility to free that memory when you're done, or if it's still owned by an object, or...

So the methods only accept/return Max strings when the originals accepted/returned a higher-level string type with more well-defined semantics, like irr::core::string.


(Adding a conversion rule for char* would be perfectly possible, but you should only actually do it for a very tightly-organised library with extremely rigid rules, or it'll probably cause all sorts of problems. Irrlicht is too big and complex for such a rule, I think)


On post #16... that a method is being generated with the name "MakeWith" seems like a bug; I shall try to fix that in a sec (it should surely either be "with something", or not! I guess it's because the original parameter is optional). Fixed, all downloads have been updated. I'd failed to consider that an overloaded procedure might have all-optional arguments, so it was appending "With" to the name before checking for any mandatory arguments to follow it.

Thank you KronosUK for testing this and finding an error!


KronosUK(Posted 2014) [#21]
I'm a bit stuck with the EventReceiver.

From the irrlicht tutorials I believe you need to extend this class and override the onEvent method to do what you want.

OnEvent receives an sEvent type but I can't see equivalent in the wrapper.


Yasha(Posted 2014) [#22]
Ah, looks like we have an artifact of multiple inheritance. Both ICameraSceneNode and ISceneNodeAnimator implement this class, but both have it as their secondary type.

I don't know how to translate that to Max. Max doesn't support multiple inheritance or Interfaces, so it's somewhat stuck here.


If you have an instance of either of those classes though (ICameraSceneNode and ISceneNodeAnimator both apparently implement it), you can upcast it to IEventReceiver using _Cast:

Local er:IEventReceiver = IEventReceiver._Cast(o)


The C++ object is given a new Max object that does what the new interface specifies. Remember that this subverts the Max type system and will work for all objects without objection, so be sure what you're doing.


Edit 2016: I have changed this so that multiple-inheriting types have an As_Base method for all types after the first one which can be used as a cast. Note that this creates new proxy objects that don't have ownership over the original data.


As for actually extending a C++ type... I think you need to be willing to write at least a few lines of C++ for that. I don't think the Max FFI will allow it.


KronosUK(Posted 2014) [#23]
Thanks for your answer. Unfortunately this seems to be as far as I can go with this for now. Still, I learnt a couple of things along the way;)