Bank size greater than capacity!

BlitzMax Forums/BlitzMax Programming/Bank size greater than capacity!

Taron(Posted 2014) [#1]
Oops, should've gone into this forum here, sorry...

Hi all,

I'm having trouble with banks right now. I seem to be running into the size limit (32bit, I guess) of banks for compression?

Does anyone of you know how I could get around this?
Is there a way to append two compressed banks into one file, maybe? Or even an easier solution to this?

It's quite scary, because I really don't know exactly how to go about it.

Thanks in advance!


Derron(Posted 2014) [#2]
if you reach the 32bit limit: would another "bank" fit? (a > x => a+b > x too {a,b,c > 0})

Did you think about "filestreams", like a streamable "cache" for your binary data. A never ending "flow" of data should only be limited by the partition system (fat32, ntfs, reiserFS,...).

so instead of having the "big chunk" in memory you only have "2MB" alive each time. Compare it to streaming media - just reversed.


bye
Ron


Taron(Posted 2014) [#3]
I need to compress the bank, you know. I don't know how I could do that with such a solution?
I was wondering, if it was possible to append banks. It's the first time I really work with those. I guess, if I could somehow splice them, knowing when I run into a limit, so to say, then compress one bank, start a new one, compress that and then save them both into one file?
Upon loading I would then have to know where one ends and the next begins and somehow unpack them into the proper whole again. (OMG)

This is all rather scary to me, but I gotta somehow figure it out!

Thanks already for the suggestion, though! I really can use all the help I can get! 8|

Oh, and this is really just about loading and saving, of course.


Derron(Posted 2014) [#4]
zipstream?

think koriolis made a module in the past for this.


bye
Ron


Derron(Posted 2014) [#5]
Forget my last post... zipstream is for "reading" only.

Like suggested: You have a Filestream fs.

When writing a bank to the filestream:
prepend a "splitter"-signature (eg a string "<SEGMENT>").
After writing the binary data, append another signature "</SEGMENT>"

Now, when reading back the big bunch of data: you first "seek" the position within the stream (seek until "<SEGMENT>" is found the n-th time. Now calculate the position of the first data after the ">" sign. This is the "start". The End mustn't be searched: just read in until you reach "</SEGMENT>".

Of course you could first search for </SEGMENT> to enable "block reading". Another option is: have "block headers:"

<SEGMENT>
<HEADER>
<SIZE:XXXX>
</HEADER>
RAWDATA
</SEGMENT>

using the above "structure" you could read the amount of bytes to read in a "block" without much hassle. Multiple segments could be stored in one file without problems.


For compression: This "rawdata" could be "compressed" or not ... you have to decide. You could also do something in the likes of: exporting each chunk as "chunk00x.tmp", compress it - and append this (similar to the method above) to a file. Removing the tmp-file afterwards is recommended :D

bye
Ron


Taron(Posted 2014) [#6]
That does sound very promising, thanks! I'm not sure how to do it, yet, but I'll dig through as much as I can find, as usual.

My brain is so fried already, it's embarrassing... 8P

I get the concept, though, it's just about practical issues, but you sent me well on my way, I believe. It'll be interesting. <shiver>

Here's from the top of my head an idea, but I'm uncertain about the Offset and Count arguments of the writeBank() function. I'm not sure why it wouldn't automatically know at least one of those, similar to ordinary writeFloat or writeByte and so on...as they know a float has 4 bytes and all that.

I really hope you don't mind I'm running all this by you. I'm a little bit desperate, as you probably guessed already.

CreateFile(filename)
			
Local datastream:TStream = OpenStream(filename, True, True)
FlushStream(datastream)


Local Bank:TBank = CreateBank() 
Local BStream:TStream = CreateBankStream(Bank)

local segment:String = "<seg>"
BStream.WriteString(segment)

local damnnearinfinitiy:int = thatmuch

local seg_size:string = "<SIZE:"+Bank.Size()+">"

if(Bank.Size()+thatmuch > $7fffffff-seg_size.Length)
  BStream.WriteString(seg_size)
  '...now what? 
  Bank = compressBank(Bank)
  '...and then? Oh dear...somehow uhm...writebank probably.
  WriteBank( Bank ,datastream,offset,count )
  'offset and count,hmm. I'm a bit confused about them. 
  Bank.delete()
  BStream = CreateBankStream(Bank)
else
  For local omg:int to damnnearinfinity
    BStream.WriteFloat(stuff)
  next
endif 
CloseStream(datastream)



No matter what happens, thank you so very much already! I'm happy you've already responded so supportive!


Derron(Posted 2014) [#7]
I'm happy you've already responded so supportive!


Ach, kein Problem!
In return I enjoy your artworks (look at mine - http://www.tvgigant.de and you see how far I am away from your skill).

EDIT: finished the sample




Took a bit longer than wanted but "bank.Read()" made some trouble as soon as size was > 230...

Of course you could write that "bankSize" just within the first 4 integers after the "<SEGMENT>" part - and then read it - but somehow I wanted to include a "reading" function so you could add even "strings" (eg. copyright notices)

Hope that helps somehow.

EDIT2: using framework now (only used modules) - and replaced that "bank:TBank var"-referencing params. Feel free to reiintroduce it to reduce memory usage.

bye
Ron


Taron(Posted 2014) [#8]
Waaah, PIXEL KUNST, Hahaha, da steh' ich doch drauf! ;o)
Thanks so much, Ron, that's awesome! I'll try it out right away!
I just realized once I got the bank pieces under control, I could literally go ahead and have each layer in its own bank to make sure it won't ever happen again that a capacity is reached. One of these days I may get to support big resolutions and then I'd be spared a scare! :}

Uh, I'm excited, I'll go through it now!

You oh so totally rock! 8D


Taron(Posted 2014) [#9]
I GOT IT! I HAVE GOT IT!
OH my freaking lordy, I'm so happy!

DANKE DIR, RON!

Wow, that was actually amazingly easy after all. The tricky part was, that I had to make sure old files could still be recognized and loaded properly, which are quite different, of course.

Ok, You, Sir, will most definitely be in the credits! 8)))

Thanks a heck of a lot! I think I would've gone nuts by myself!


Taron(Posted 2014) [#10]
Oh, and tell me how I would refer to you, or if you want me to post a link as a thank you!


Derron(Posted 2014) [#11]
First of all I have to admit one thing:

The "format" how banks are stored with my code is not "professional". A better version should work how it worked years ago:

You define a file format including headers, block amount, ...

Type TSaveFileV1
field header:TSaveFileHeader
field dataBlocks:int
was used
...
End Type

Type TSaveFileHeader
Field safeFileVersion:string = "" 'set this within TSaveFileV1
Field author:string = ""
Field otherMetadata...
Field evenMoreMetadata...
End Type

This SaveFile is specific for your fileformat 1.0 (so you can have multiple to "allow old ones".
When saving - you save the "header" which you write into your savefile in a way you can read it back without problems:
a) create a bank and only use "writeline", write the "bank".size() as the first integer of your savefile, then the bank
b) when reading back: first int of the savefile has size of a "headerBank", read 1 int -> bankSize. Now read the headerBank in.

The headerBank can be then used in conjunction with "readLine" to read your header in a "readable format". you now got back all the needed information: which file version is it (this could even be done with an "2nd int" in the file... but we talk about ~1kb of data, so that isn't so "slowing down") and most important: how many blocks you have stored.

Each "stored block" is basically this: an int describing the size of the "upcoming" bank, and the bank data.
So right after the headerBank you would store an int with the "dataBank".Size(). Next is the "dataBank".

a) as you know how many banks are coming you can count if the file is complete
b) just ignore the amount of banks and read until you are not able to find "size-value"+"size*bytes" for reading a bank.


Mine has the "advantage" of readable "parts" in the code (like within an xml) but the "strict binary one" will surely be more "professional" and "faster" .



@Credits:
I think that isn't needed, really, just thank for the help of the community and we are done :D.
But maybe you get some spare time somewhen ... and want to paint a small - freely (open source -> so "cc sa" or so) asset for my game. Needing especially "generic movie genre pictures" (see bottom left of the screenshots). If not - no problem (I am even scared you provide a "horror movie"-asset with a monster not "nice" enough for halfhearted people :D).
Edit: no, think that is too much work to do: I think in "return" you could download the game, give it a short play and give back some critics so we/I know what to improve (at the moment only a bunch of people is giving feedback 400dl/version and 4 people returning feedback in the forum). If you do not like economic simulation games at all: no problem, stay with the "community thanks".



bye
Ron


Taron(Posted 2014) [#12]
Ah, cool, yeah, I've actually already done that, but I'll make that official thanks to the community more prominent!

Funny enough, because I had to merge my old file format, it actually has a lot of that, really. I could put the whole "author" thing in as well, sounds nice, if anyone bothers writing his name, haha. Not very common with images.

Speaking of, well, I'll have a look and see what I can do for you! Sounds fun! 8D

I'm just so happy, though! You really gave a big boost of joy there!
When I look at Verve now, it becomes increasingly more bewildering, that I write this whole thing in Blitzmax, haha, not even using MaxGUI, hehe...it's really such a fun language and I feel like I only know less than half of it.

Really exciting it gets, once I find some way to try compiling for the mac. Need a mac OS first...OSX, I guess. Anyway...


Derron(Posted 2014) [#13]
Brucey has a mac - and some talent finding/removing bugs ... if he is not doing some other magic.

@author
instead of "writing it everytime" - I would suggest to add the option to setup "defaultAuthor" and "defaultCopyright" etc.

So every image done with your "customized" Verve will have the information included. If you use a "FileSaveAsDialogue" you could check if you can add a "[x]include copyright information" entry.

As exported file formats like "png" could include this information too - this isn't as useless as you might think.


Another thingie for this is: "Export with Watermark" - which just prints the author/copyright string into the footer area of the image (or where you want to display it).

As your app is win32 only I wasn't enjoying it here on my linux machine - my virtualbox-windows isn't speedy enough I think (also the only drawing-pen I have is an aaaaged A5-pen-tablet with a stupid plastics-pen ... never got really used to it).


bye
Ron