NewsForumsCodeLogsGallerySpecsSearch

Two WIP modules...

BlitzMax Forums/Brucey's Modules/Two WIP modules...

Brucey(Posted 2008) [#1]
Hallo again !

Since it's the weekend again, I've been playing around with a couple of new modules, neither finished as yet, but perhaps of interest anyway :-)

The first, BaH.Persistence, is an Object serializer, which can handle most things you can throw at it. The only things it doesn't like very much are multi-dimension arrays and Byte Ptr data. The arrays issue I think is due to the Reflection module.
It uses the Libxml module to build and load the persistence data.

Perhaps the most interesting bit about the module is that it was designed from the beginning to properly handle circular references. This means that you can serialize a TList and be sure that when you de-serialize it, you will end up with a TList in exactly the same state as it was in at the time.

...

The second, BaH.XLWriter, allows you to create .xlsx files using a very simple interface. xlsx is the file format for Excel 2007, and is also readable by apps such as Numbers (Apple) and Open Office. It is actually a special zip file containing lots of different types of xml files.

It's currently in the very early stages of development, but it is already able to create basic spreadsheets of text, numbers and formulas. At the moment, it is using BaH.Libxml, BaH.RegEx and wxMax. wxMax is required for access to the Zip output stream, which means it can create all the files for the zip as it creates the zip itself - ie. no external files created.

...

The Libxml that both modules require is the one currently in SVN at maxmods. I should hopefully be releasing the update fairly soon.

You can find the mods at the same location.


Docs are a bit on the sparse side for both at the moment, but there is a small example for each, and suggestions/comments are very welcome.

Anyway, thought you might be interested to see what's on-going :-)


SebHoll(Posted 2008) [#2]
The first, BaH.Persistence, is an Object serializer, which can handle most things you can throw at it.

Probably a stupid question, but what exactly is an object serializer (what does it do)?

The second, BaH.XLWriter, allows you to create .xlsx files using a very simple interface. xlsx is the file format for Excel 2007, and is also readable by apps such as Numbers (Apple) and Open Office. It is actually a special zip file containing lots of different types of xml files.

This seems interesting... :-)


Brucey(Posted 2008) [#3]
what exactly is an object serializer

Imagine you have an Object instance of a Type, say :
Type TRect
	Field x:Int
	Field y:Int
	Field w:Int
	Field h:Int
End Type

...and you want to save it somewhere, but you don't want to have to write some save code for all your types...
Print TPersist.Serialize(New TRect)

...

<?xml version="1.0"?>
<bmo>
  <TRect ref="01VKY0">
    <field name="x" type="int">0</field>
    <field name="y" type="int">0</field>
    <field name="w" type="int">0</field>
    <field name="h" type="int">0</field>
  </TRect>
</bmo>


Yes, you could store that away in 16 bytes using a custom format, but then you'd have to write the custom format, and test it....


SebHoll(Posted 2008) [#4]
That may be extremely useful... :-)


DavidDC(Posted 2008) [#5]
Two excellent additions. Well done Brucey!


DavidDC(Posted 2008) [#6]
For those still not catching on to how useful TPersist is, this real world example might help. I just converted this xml loading code from:


to
Return TSettings(TPersist.Deserialize(LoadString( "Settings/" +_s_file)))


And of course, the save function is now a one-liner also.

This:


Becomes 
New TPersist.SerializeToFile(Self, "Settings/" +_s_file)

You've got to love that! :-)


plash(Posted 2008) [#7]
Didn't reflection already have serialization?


Brucey(Posted 2008) [#8]
It uses reflection to provide the serialization.


plash(Posted 2008) [#9]
Just looked at the docs, I thought there was a standard serializer already, my bad.


Brucey(Posted 2008) [#10]
XLWriter is now able to add font-formatting to cells, which is kinda cool - that's things like different fonts, sizes, bold, italic etc.
...which adds to text, numbers and formula in cells.

Now that the core for styles is implemented it should be fairly trivial to get borders, fills, and colours on board too. And then cell formatting (like text-align, or numbers with 2 decimal places, etc)

:o)


nadia(Posted 2008) [#11]
This sounds all very exciting :-)


Brucey(Posted 2008) [#12]
Borders and colours working now. (See provided examples for usage).

Fill and formatting styles to come.


Glenn Dodd(Posted 2008) [#13]
excuse my ignorance but what url am i needing for the persistence and xlwriter modules for svn?

will the same address give me all the other bah.* modules?

cheers
glenn


Glenn Dodd(Posted 2008) [#14]
will bah.xlwriter also let me READ xls files?


DavidDC(Posted 2008) [#15]
You should find them at

http://maxmods.googlecode.com/svn/trunk

will bah.xlwriter also let me READ xls files?


No. The idea is to use Excel/Numbers for that.


Glenn Dodd(Posted 2008) [#16]
cheers


Brucey(Posted 2008) [#17]
Yep, just for creating xlsx files.

Unless you were wanting to create your own spreadsheet app, there wouldn't be too much need for xls(x) reading.

The XLWriter module makes it easier to create formatted spreadsheet-compatible documents.
Formats like CSV don't give you formatting - which is something you might wish to provide for your end-users.

Basic formatting support has now been added to the latest version (see example_01). Also improved the documentation.


Glenn Dodd(Posted 2008) [#18]
I often need to take spreadsheet data and save as csv or txt so i can read the data into a program, manipulate it, then write something back out as csv or txt, and finally resave in xls format with formatting. I was just hoping to save a step of converting from xls to csv/txt and back again.


Difference(Posted 2008) [#19]
BaH.Persistence sounds great.

I'd like to be able to combine it with xsl transforms, so that I can make a "write" xml stylesheet, that will format the xml in a certain way and also a "read" stylesheet that will transform my custom xml into the format that BaH.Persistence expects.

I realize I can do this simply by reading and transforming the resulting xml files, but I'd like to see it integrated into BaH.Persistence, so that I could pass the loading and saving stylesheets as a parameter (or something like that) to (de)serialize.

Could this be made possible?


Brucey(Posted 2008) [#20]
@Glenn
Bit of a big job, and not something I'd really fancy attempting - building a reader for some proprietary binary file format, which would have limited mileage anyway.

@Peter
It certainly sounds possible, and would be quite useful.
Not sure if it wants direct integration into that module, as you then add libxslt requirements onto apps that might not necessarily use that functionality.


Glenn Dodd(Posted 2008) [#21]
that would be why i never attempted it too. Not to mention the fact that it would be too hard for me :)


Difference(Posted 2008) [#22]
@Brucey

Ok sure, but if there could be an alternative to SerializeToFile() and SerializeFromFile()?

Something like:
SerializeFrom(doc:TxmlDoc) and Serialize:TxmlDoc(obj:object)

then the post and pre transforms could easily be done with an TxsltStylesheet.applyStylesheet(doc) command.


Brucey(Posted 2008) [#23]
Okay, how's about :

Method SerializeToDoc:TxmlDoc(obj:Object)

...for serializing.

You'll have to free the returned object yourself.

For DeSerializing, I've changed the DeSerialize() Function to take an Object instead of a String parameter. If you pass in a TxmlDoc (rather than a String), it will use that doc as the tree to process. Internally, it uses :

Method DeSerializeFromDoc:Object(xmlDoc:TxmlDoc)

But you don't need to call that specifically if you don't want to.
Again, the code won't free up the TxmlDoc you pass in, so you can do what you like with it otherwise, before you Free() it.


Difference(Posted 2008) [#24]
Absolutely great!

[EDIT] If you want I can make an example that you can include, when it's ready.


Volker(Posted 2008) [#25]
@Brucey: Feature request

It would be nice to have a metadata switch to
include only the fields with {serialize}.
How about that?


Brucey(Posted 2008) [#26]
The problem I have with an inclusion switch is for cases such as :

You have a Type, which includes a TList that you wish to also serialize. You tag your TList field with the metadata, which enables serialization for that field. However, since the TList has no metadata for its own fields, none of the TList content is serialized.

I much prefer catch-all.

What about a tag for *not* serializing?
Unless you have a good reason for otherwise? :-)


Volker(Posted 2008) [#27]
> What about a tag for *not* serializing?

Yes, this will solve my problem!
Saving Meshes with Persistence.mod seems to be no
good idea. My TList file is bigger then 60 Mb with just one object.
(no bitmaps, a simple cube)

> Unless you have a good reason for otherwise? :-)

Not yet :-)


Difference(Posted 2008) [#28]
My TList file is bigger then 60 Mb with just one object.

Compressing in memory before you save will reduce that size drasticly.
You can get an idea of how much by zipping the xml.

@Brucey: Could som space be saved by using a different format for arrays?

Taking a hint from collada and svg:

<field name="numbers" type="array:Int">
<val>1</val>
<val>2</val>
<val>3</val>
<val>4</val>
<val>5</val>
<val>6</val>
</field>

could be 

<field name="numbers" type="array:Int">1 2 3 4 5 6</field>



Brucey(Posted 2008) [#29]
Fine for numbers, but not so great for strings - unless you encode them.

I don't see a problem doing it for numbers though.


Brucey(Posted 2008) [#30]
Hmm.. looks like I've found a reflection bug with arrays and Long/Doubles :
        <field name="numbersl" type="array:Long">
          <val>1</val>
          <val>8589934592</val>
          <val>2</val>
          <val>12884901888</val>
          <val>3</val>
          <val>17179869184</val>
        </field>

...
	Field numbersl:Long[]
	this.numbersl = [ 1:Long, 2:Long, 3:Long, 4:Long, 5:Long, 6:Long ]


Ho hum...


Brucey(Posted 2008) [#31]
Bug posted HERE with fix ;-)


Brucey(Posted 2008) [#32]
Hokay... latest version changes for BaH.Persistence :

* Added support for {nopersist} metadata tag for fields.

Example :
Type TRect
	Field x:Int
	Field y:Int
	Field w:Int
	Field h:Int
	Field ignoreMe:String = "Hello" {nopersist}
End Type

...

        <TRect ref="0A01TK">
          <field name="x" type="int">0</field>
          <field name="y" type="int">0</field>
          <field name="w" type="int">0</field>
          <field name="h" type="int">0</field>
        </TRect>


* Added Peter's "compression" technique for arrays of numeric values. String and Object arrays remain the same.
Note : Also introduced a TPersist.BMO_VERSION constant which enables backwards read compatibility with previous serialization formats. If you are building your own XML, you should probably use the constant when setting the "ver" attribute of the root "bmo" node.
eg.
<bmo ver="1">


* Enabled global access to the "format" field, which now defaults to False. Setting this to True will enable "pretty" XML formatting of the serialized data. Run the example to see both types.

Any other ideas or queries?... :-)


Difference(Posted 2008) [#33]
Absolutely excellent. Works like a charm.


Volker(Posted 2008) [#34]
One more question:
Why is there no method "deserializefromfile"?
"deserializefromdoc" needs a TXMLdoc??

At the moment I helped myself this way:
Function SaveObject(obj:Object, file:String) 
	' Test for existing file TODO
	CreateFile (file) 
	Local f:TStream = OpenFile (file) 
	Local tp:TPersist = New TPersist
	tp.SerializeToString(obj) 
	WriteString (f, tp.SerializeToString(obj)) 
	CloseFile (f) 
End Function

Function LoadObject:Object (file:String) 
	Local f:TStream = OpenFile (file) 
	Local st:String = ReadString (f, StreamSize(f)) 
	Local obj:Object = TPersist.DeSerialize(st) 
	CloseFile f
	Return obj
End Function


But I think this is not the way it' s meant to be used.


Brucey(Posted 2008) [#35]
Why is there no method "deserializefromfile"?

I suppose it never occurred to me :-)

So... I've added... DeSerializeFromFile:Object(filename:String) method.

While I was tinkering, I've also added support for Streams.

There's now a method SerializeToStream(obj:Object, stream:TStream) which will write the serialized object to the passed in Stream.
It is up to the user to Close() that stream when they are finished with it.

The DeSerialize() function now also accepts a TStream object, which it will attempt to read serialized data from.
I've also added a DeSerializeFromStream:Object(stream:TStream) method, if you want it.

The example now demonstrates using the stream too.
It now effectively shows the following :
Object -> String -> Object -> File (via Stream) -> Object (via Stream) -> String (formatted)

:o)


slenkar(Posted 2008) [#36]
does the serialization module use XML to save the objects?


plash(Posted 2008) [#37]
Yes.