optimize persistence?
BlitzMax Forums/Brucey's Modules/optimize persistence?
| ||
| I gave the new persistence module a good test and it works great, Just a little cheeky request... Im saving some arrays [80,80] which are mostly empty, the save file amounts to 9MB of mostly [val][/val] Is there a way to optimize it so that emptiness is not recorded? thanks |
| ||
| Zip it? The mod will even read zipped xml directly I think?. |
| ||
| libxml supports compression, so I suppose I could add another "flag" to the module to enable compression of your serialized data? Are you using the non-prettified xml output? (ie. that without indentation etc) |
| ||
| oh yeah sorry, I didnt mean to say that, I meant to say that it takes a long time to load and save the game, which makes debugging and programming slow I thought maybe you could say 'skip 46' which would fill the next 46 slots in the array with zero or null values |
| ||
| @Brucey : An optional compression flag sounds absolutely great! It might even speed up loading and saving times if disk acccess is the bottleneck. |
| ||
| disc access could be the bottleneck, because i put some debuglogs into the serialization source and it sent the info to the stream pretty quickly, then there was a long pause |
| ||
| Seems it might be reflection that slows things up a bit when you have masses of data. I've added a new "example" which highlights the general speed when it has a *lot* of object data to serialize. It also demonstrates a couple of new features : * compression : Added a new global "compressed", which you can set to True to enable compression. At the moment it is only supported via File serializing (rather than with streams or Strings). That's down to libxml itself though. I don't think it changes anything time-wise, but it certainly shrinks the output! * node parsing depth : Added a new global "maxDepth", which you can tweak if you have issues when deserializing your data. Internally, it sets the libxml xmlParserMaxDepth variable. I've also tuned the output a little so that empty strings result in empty nodes, rather than a node with nothing in it. (ie. <val/> instead of <val></val>). And I fixed a bug related to inlined Strings... |
| ||
| the save file amounts to 9MB That's a lot of object! Is that in non-formatted terms too? |
| ||
| I put all the game information into one object it has about 5 arrays of [80,80] and thats just for the game map the </val> thing should half the size of the file Im glad you put the xmlparser depth flag in there because I had to edit an obscure value in parser.c before |
| ||
| I just tested maps and I found they work just as fast as arrays and probably save a lot of filesize too. |
| ||
| Depends what information you are storing, I suppose :-) Maps aren't so good with "int" keys, as you have to convert them to something else (like a String, or a custom int-type), but they work very well otherwise, especially if you need to access the data randomly (ie. not sequentially). Direct array access is of course faster than via a Map, but with these things there always a trade off between perhaps a faster algorithm with a slower, easier to maintain algorithm... |
| ||
| Fixed a bug with serializing TMap - some nodes refer to a private "nil" global node, which wasn't de-serializing to point to the current nil node object. This could cause issues when using the TMap after loading. Which is a bit of a bugger really. Don't really want to add type-specific handling to this, but in some cases there's not much I can do about it. I suppose a preferred option would be to have some kind of pluggable interface where Joe Coder would be able to add his own type-handling for special cases... which all seems a wee bit over-the-top for a simple object persistence module. argh... |
| ||
| hmm what about having the tmap as a field of a type? would that make any difference? |
| ||
| You can have a TMap anywhere you like. The latest SVN revision fixes the bug. |
| ||
| thanks for the bug fix, whats the issue with having to add 'type handling' i dont understand it from your first description |
| ||
| Just a possible issue with references to "global" objects, once which a module or app create, but which the serialized data refers to. Makes it difficult to de-serialize the data without building in some kind of custom-handler to deal with special cases. Nothing to worry about, probably :-) |
   |