Economic Saving of an Array or bank

BlitzMax Forums/BlitzMax Beginners Area/Economic Saving of an Array or bank

CandlestoneDave(Posted 2012) [#1]
Hi All

I am trying to save a tremendous amount of data to a disk. It is essentially a 'save game' file, but it needs to record the entire state of the game world.

The game world is 1000 x 1000 x 200 blocks.

The method I am using currently is as follows:

Local linestring:String
Local out:TStream
linestring = "savegame/" + "save1" + ".bin"
out=WriteStream("savegames/save1.Bin")
i = 0
For z = 1 To 200
For y = 1 To 1000
For x = 1 To 1000
i = i + 1
PokeByte (Myworld:TBank,i,world[x,y,z])
WriteBank(MyWorld:TBank, out:TStream,i,1)
Next
Next
Next

Where the data I am saving is held during the game as an array, world [x,y,z]. The above converts it to a bank, then saves it to a stream.

It works, but the problem is that it takes a very long time to save and then load again. It also creates a save file of 200MB, which seems ridiculously huge.

Does anyone have any ideas of a more time/space economical way of saving the data?

Thanks

Dave


Jesse(Posted 2012) [#2]
writing one byte of data at a time is the slowest. you can write the whole file to the bank stream then write the whole bank at once with FlushStream. Also, you might want to compress the file before writing to a file. you can use the zlib included with Bmax or us a third party one like gman's zip engine:
http://www.blitzmax.com/Community/posts.php?topic=97329#1132795


matibee(Posted 2012) [#3]
Do your game 'blocks' begin with some default value; ie from a map, or even empty? Then you could just write the blocks that are different from the default.

A simple method would be to write [x,y,z,block value] for each changed block, a better way would be to find a chain of changed blocks and write:

x,y,z (start position of chain)
n (length of chain)
n[] values

Compression will also help as Jesse said. Further hints might depend on the world your working with. Perhaps it's a minecraft-a-like? How many types of block are there? If there's 16 or less, two blocks could be written as one byte. etc.


col(Posted 2012) [#4]
Run length encoding would probably be better if youre saving a lot repetitive data, which is generally the case with arrayed game worlds. when combined at the bit level you could potentiallly squeeze your 200mb down to about 2mb upwards.

If you tell us a bit more about the kind of data in each array slot then we can give you a more detailed suggestion.


CandlestoneDave(Posted 2012) [#5]
Thanks a lot for the comments. I'll go through each one in detail after work today. Col, the data in each slot is simply a number between 1 and 256. Thanks.