Presenting Protocol Buffers

BlitzMax Forums/Brucey's Modules/Presenting Protocol Buffers

Brucey(Posted 2008) [#1]
Protocol Buffers is a newly open-sourced Google data serialization API.

It is fast at both serializing and parsing, and creates data streams much more compact than the equivalent XML.

I'm currently working on a BlitzMax "compiler" and integration of the protobuf library into a module. The compiler generates BlitzMax Types and C++ glue code which integrates directly with the core generated C++.
This makes for the best of both worlds - you get the pure speed of the native C++ parsing/serializer with a proper BlitzMax interface to the data :-)

Once you've run the "compiler" on the .proto file, all you need to do is Import the generated .bmx file, and away you go, creating and using the new Types.

It really is as easy as :

... part of addressbook.proto
message AddressBook {
  repeated Person person = 1;
}


.. using the generated code
Import "addressbook.pb.bmx"

Local address_book:AddressBook = New AddressBook.Create()


The module provides support for TStream in serializing and parsing, which opens up the doors to all kinds of protocols ;-)
It also supports String and raw Byte arrays, which hopefully covers all the methods that you might want to throw your serialized data around.


There are some issues with BMK (and BCC) currently (heh, aren't there always when I'm hacking?), but we are working to get these resolved in the not too distant future.

Anyhoo... just thought some of you might be interested. I should hopefully have something available to play with soon.

Comments and stuff welcome as always :-)

<EDIT> - I should probably add, it is all actually working as of now. Tested on Win32 and OS X. ;-)


plash(Posted 2008) [#2]
I've been watching your worklog - very exciting!

we are working to get these resolved in the not too distant future.
WE? ;)

Anyhoo... just thought some of you might be interested. I should hopefully have something available to play with soon.
Can't wait.


plash(Posted 2010) [#3]
Bump =)
I noticed the protobuf wiki page was updated recently (Feb 04), does that mean we should be seeing it anytime soon (I can't find it in the SVN trunk.. maybe I'm just blind)?


Brucey(Posted 2010) [#4]
Finally committed it :-p

Very rough around the edges, but the example works.

To test :

1) Run the generate.bmx first. It will "compile" to proto and generate a bunch of files.

2) Try building main.bmx. It's a very basic example which will attempt to create an object, serialize it and then read it back.

The other two examples, write_message and read_message are a bit more interactive.


Since I enoy this whole code-generation thing, it's fun to play with, but I'm not sure where it will be directly useful - Perhaps as a(nother) object persistance thing for BlitzMax? Who knows ;-)


slenkar(Posted 2010) [#5]
it could be faster than bah.persistence?


plash(Posted 2010) [#6]
Thanks!

Since I enoy this whole code-generation thing, it's fun to play with, but I'm not sure where it will be directly useful - Perhaps as a(nother) object persistance thing for BlitzMax? Who knows ;-)
I was thinking of using it for message serialization over the network. We'll have to see how that goes :p

EDIT: You forgot to do something for Linux?



Brucey(Posted 2010) [#7]
I haven't reached Linux yet, unfortunately. (I commit to repository on Mac, update on Win32, check and fix, commit, then the same process on Linux - well, that's the usual order - sometimes it takes a day or so to get to Linux)
I will do in a few hours though.
Just missing some Linux-specific settings, I imagine.


Tommo(Posted 2010) [#8]
I'm using lua for serialization. With precompilation, it can run quite fast.
Protobuf should be quite handy for data-heavy jobs.
:)


Brucey(Posted 2010) [#9]
Choice is good :-)


plash(Posted 2010) [#10]
Just missing some Linux-specific settings, I imagine.
That's what I figured, but I have no idea what's supposed to be in there :P


slenkar(Posted 2010) [#11]
I got the same errors on windows
e.g.:
C:/BlitzMax/mod/bah.mod/protobuf.mod/protobuf.debug.win32.x86.a(glue.cpp.debug.win32.x86.o):glue.cpp:(.text+0x1b): undefined reference to `google::protobuf::compiler::CommandLineInterface::CommandLineInterface()'



Brucey(Posted 2010) [#12]
Interesting... Are you guys using the latest BlitzMax?
That one should have support for compiling .cc files.


Brucey(Posted 2010) [#13]
Linux is now working - as per the other two.

I'm using 1.38 and my bmk on all platforms, but the official bmk should work too (since it now correctly supports .cc files (or it should by now!)).


plash(Posted 2010) [#14]
I'm using 1.38.
It's still showing the same errors for me.


Brucey(Posted 2010) [#15]
Ah. It seems that bmk isn't in fact fixed yet... I guess y'all will need to wait until the next release of BlitzMax for that to be resolved.

At least bcc doesn't barf over .cc files these days.


plash(Posted 2010) [#16]
$ bmk makemods bah.protobuf
ERROR
[string "function bmk_default_cc_opts(...)..."]:17: attempt to call field 'AddOption' (a nil value)Unrecognized Import file type: src/google/protobuf/stubs/common.cc

Any chance you can update the linux packages on your mingw page?


Brucey(Posted 2010) [#17]
Any chance you can update the files on your mingw page?

Oops. Yeah.


plash(Posted 2010) [#18]
Hum, still not working; now it's only showing the import error (the AddOption error was fixed):
Unrecognized Import file type: src/google/protobuf/stubs/common.cc


P.S. Sorry for not responding sooner, I got distracted :P


slenkar(Posted 2010) [#19]
i also used 1.38


Brucey(Posted 2010) [#20]
I had (mistakenly) thought that BMK (for 1.38) had finally been fixed to support .cc files.

I suppose, at least the modules compile without error, which will keep some of the wolves from my door for a while :-p


plash(Posted 2010) [#21]
I suppose, at least the modules compile without error, which will keep some of the wolves from my door for a while
Clarification: The error I'm having with bmk_ng is when compiling the module, not the generator code.


Brucey(Posted 2010) [#22]
What version are you using?

./bmk -v


eg.
$ ./bmk -v
bmk 2.07 mt-macos-x86 / gcc 40201



plash(Posted 2010) [#23]
$ bmk -v
Command line error : Unknown command '-v'


Strange. I know it's one of yours though, not only because I just double-checked, but because the normal bmk doesn't give help info when you give it no arguments.

EDIT: Also, if you're referring to the official bmk, I get this:
~/blitzmax/bin/138$ ./bmk -v
Command line error



Brucey(Posted 2010) [#24]
You could download the source and build it yourself... that way you'd know what version you had.

Of course, there's the "wait until 1.39 comes out" choice too :-)


plash(Posted 2010) [#25]
You could download the source and build it yourself... that way you'd know what version you had.
Ah, I was under the impression you updated the Linux package. I had never tried to build it from the source, will do..


Brucey(Posted 2010) [#26]
Ah, I was under the impression you updated the Linux package.

I have also. But sometimes it's just better to D.I.Y :-)


plash(Posted 2010) [#27]
Righty then, bmk -v reports 2.07 as intended, module compiles, generator compiles and runs.
However, trying to compile main.bmx, write_message.bmx or read_message.bmx gives a strange set of errors (which claim system headers are b0nkers):