ZipEngine 2.0 for BlitzMax

BlitzMax Forums/BlitzMax Module Tweaks/ZipEngine 2.0 for BlitzMax

gman(Posted 2005) [#1]
just want to thank TommyBear again for starting this module up.

all furture info and downloads regarding this module have been moved to the gprogs.com forum.


Grisu(Posted 2005) [#2]
Just woke up and saw ir... :)

Thanks a lot.... testing in progress!

edit:
Ok, all seems running well. Especially like the debug info u added... :)

Some small stuff in case u touch this again:
+ give the zip a versionno. "20", so we know what we download
+ add a small readme: general info + commandset + changelog


Red(Posted 2005) [#3]
So cool.
Bmax mod would be boring wihtout mod coding :)


boomboommax(Posted 2005) [#4]
can you add the ability to extract a zip to a folder without knowing the files inside?


Tibit(Posted 2005) [#5]
Is it possible to only zip a string? How big does a file have to be for zip to be useful?


Grisu(Posted 2005) [#6]
@Wave:

1.
Not sure what exactly you want to do?! But sure, you can simply write a textfile with Strings and afterwards compress it.

2.
It depends on your program structure. In general I would advise not to zip hundreds of small files or single strings in seperate files, as the access time to get a file and decompress it can get low with tons of files and you won't gain a benefit from that.

So what I mostly do is to subdivide my media into main parts. E.g. STATIC DATA, IMAGES, SOUNDS, SAVEGAMES.


Tibit(Posted 2005) [#7]
I meant, Can I zip a string in memory and then write it to a file or something else like a network package.

The smaller the String the faster is shoud take to zip/unzip it so this might be a good idea for file transfers. It would also mean free encryption.


Grisu(Posted 2005) [#8]
It is currently possible to extract a hard disc file to system ram. And to load media from there.

fileBuffer = zrObject.ExtractFile(f$)

But I doubt there are commands in for compressing a ram buffer yet.

Perhaps GMAN could help? :)
*takes cover behind a sofa*


gman(Posted 2005) [#9]
@{cYanide} - although i didnt wrap it up, i did add the capability to loop through all the files in the zip. you should be able to leverage this to do an ExtractAll() type method. see the unzip example for the looping. whenever i get to an update of this i will add and extract all.

@Wave - i will see if i can take a look into this. it will be a day or two.


gman(Posted 2005) [#10]
@Wave - turns out that ZLib is exactly what you are looking for. compressing and decompressing memory. i whipped up an example for you:
' memcomptest.bmx

Strict
Framework BRL.Basic
Import Pub.ZLib

' compression levels
Const Z_NO_COMPRESSION:Int=         0
Const Z_BEST_SPEED:Int=             1
Const Z_BEST_COMPRESSION:Int=       9
Const Z_DEFAULT_COMPRESSION:Int=   -1

' im not sure how you would guess the size for the buffer.
' i guess you just make it big and hope for the best.
Const BUFFER_LEN:Int=1024

' create a string to compress
Local test:String="this is my string.  it would be really really groovy if i could compress"+Chr(10)+ ..
	" this String And Then decompress it without loss."

' compression test

' allocate a buffer
Local buffer:Byte Ptr=MemAlloc(BUFFER_LEN)
Local bufferlen:Int=BUFFER_LEN

' compress the string
compress2(buffer,bufferlen,test.ToCString(),test.length,Z_BEST_COMPRESSION)
Local compressed:String=String.FromBytes(buffer,bufferlen)

' show the resulting reduction %
Local pct:Int=100-Int((Float(compressed.length)/Float(test.length))*100)

DebugLog("")
DebugLog("---")
DebugLog("original string len: "+test.length)
DebugLog("compressed string len: "+compressed.length)
DebugLog("string was reduced by: "+pct+"%")

' decompression test

' reset the buffer length to maximum
bufferlen=BUFFER_LEN

' uncompress the string
uncompress(buffer,bufferlen,compressed.ToCString(),compressed.length)

Local uncompressed:String=String.FromBytes(buffer,bufferlen)

DebugLog("---")
DebugLog("")
DebugLog("original: "+test)
DebugLog("---")
DebugLog("compressed: "+compressed)
DebugLog("---")
DebugLog("uncomp: "+uncompressed)

' free up the buffer
MemFree(buffer,BUFFER_LEN)



Tibit(Posted 2005) [#11]
Cool! I'll take a closer look at this tonight!


gman(Posted 2005) [#12]
heres a neat thought. TZLibStream... based on TStream but compresses and decompresses on the fly as you read and write. you could then use it wherever streams are used. not sure about implementation but i think its feasable. i might take a look into this.


Grisu(Posted 2005) [#13]
Sounds all very complicated. I don't care as long as it works... :D


Tibit(Posted 2005) [#14]
Streams are not that complicated really, you can extend the stream class pretty easy I think. Do you have any special use for zip-streams in mind?

Don't have time to look this through today, tomorrow..


gman(Posted 2005) [#15]

Streams are not that complicated really, you can extend the stream class pretty easy I think. Do you have any special use for zip-streams in mind?


not for me really, but when you mentioned sending ZLib compressed strings around in a networking package i thought about streams since they are usually the communication path. having a compression stream would allow transparent comp/decomp of send and receive data.


Hotcakes(Posted 2005) [#16]
That would rock gman.


Grisu(Posted 2005) [#17]
Would this also be possible:
You have some imagehandles in memory and you want to zip them.
So instead of writing them onto disc and compress them afterwards. You compress them in memory and only write the zip?


gman(Posted 2005) [#18]
@Grisu - if you could get the images into a TRamStream or somehow get at the bytes in ram then yes, you could zip something in ram directly to the zip file without having it actually be a physical file. if its in a TRamStream a simple change to the zipengine would be all thats needed. it would probably be a good thing to have anyway so i will put that in next time i update.


Grisu(Posted 2005) [#19]
:D

Call it Zip 3.0 :)))))))


TommyBear(Posted 2005) [#20]
Awesome work gman! Nice to see the community alive and well. Sorry I haven't been able to help out lately. I'm very busy on Tomb Raider 7 at the moment ;)


Grisu(Posted 2005) [#21]
Can someone please make this to work with BMX 1.12????


gman(Posted 2005) [#22]
updated :)


Grisu(Posted 2005) [#23]
Thanks Gman! I own you one more virtual pot of coffee... :)


Lomat(Posted 2005) [#24]
gman, any chance you could add this to the BlitzMax Wiki?


Hummelpups(Posted 2005) [#25]
Hehe, cool mod

IMurDOOM


Grisu(Posted 2005) [#26]
Gman, with bmx1.14 I'm getting new compiling errors... :(

"Compile Error: Expecting string literal but encountered malformed string literal"

=> zipengine.bmx - moduleinfo line 5

:/

Also when you use DEBUGMODE, and open many zips after each other. The program crashes without any error message.
Else all seems fine?


Rimmsy(Posted 2005) [#27]
"Compile Error: Expecting string literal but encountered malformed string literal"
This usually means there was a ~ in there somewhere which you'll have to replace with ~~ because of a change in the way escape characters work. I think. Not sure about the second error.


Grisu(Posted 2005) [#28]
There is no ~ in the hole source code as far as I can see.

And the other bug is caused by the new garbage collector, I would bet my green socks on this!


gman(Posted 2005) [#29]
ill try to take a look at this somtime today or tomorrow. busy on updating the Irrlicht mods ATM.


Grisu(Posted 2005) [#30]
Great! Thank you!


gman(Posted 2005) [#31]
k. the first issue was quick to resolve.

@Grisu - can you provide a quick sample to reproduce the crash?

thx.


Grisu(Posted 2005) [#32]
Is it already updated? (2.02)?!? I would test it again then!

Example is difficult, but I can e-mail you my project.
It isn't that big! (easy source). But data is (280 MB)... :(
And it is crashing cause of zip handling! :(

P.S.:
Do you use ICQ? Can I email you around 12 MB?
Need to get this fixed before xmas... :)


gman(Posted 2005) [#33]
the current build is available above. really didnt do anything but fix the first issue and rebuild though... but, given the philosophy of "fix what you know first", lets see what happens.


Grisu(Posted 2005) [#34]
It rebuilds without an error now!

Though it still crashes for me. :/


Grisu(Posted 2005) [#35]
You've got mail... :)


gman(Posted 2005) [#36]
@Grisu - there is a new version for download. i think i have the issue resolved finally.


Grisu(Posted 2005) [#37]
Issue resolved! :D


TommyBear(Posted 2005) [#38]
Found a bug, the readFileList() function is being called when using the ZipWriter class to create a new zip. To reproduce this, call OpenZip on a zipWriter instance with a zip that does not exist and it will generate a runtime error with"TZipReader.scanLocalHeader(): unable to read from file"

A way around this is to comment out readFileList() from OpenZip in zipWriter and AddFile too.


gman(Posted 2005) [#39]
thx for the update TB. fixed and new version available for download (at the top).


Grisu(Posted 2005) [#40]
Thanks!

btw Gman:
Is there a way to speed up file seeking in zips?

As I already know the filenames inside each zip,
is there a way to "pre-cache" this info when the app starts up first time?


gman(Posted 2005) [#41]

Is there a way to speed up file seeking in zips?


not that i know of. all the seeking and reading is done through the ZLib functions. essentially you call a function to position the internal pointer to the file you want to extract, and then you call the extract function. one thing you could do is track the currently open month or year (however your ZIPs are organzied) and just keep the current one open until the next one is needed.

something i could do in ZipEngine is add a way to work with the ZIP files without loading the header info. the bigger performance would probably be gained though by keeping the current zip open until the next is needed.


Grisu(Posted 2005) [#42]
Yepp, my guess the constant open / closing files is eating up the performance.

Would be great if you could add commands that would make it possible to access an "open" zip. So I only open / close the files when the year selector changes.

You see a lot of slowdown if the app runs from cd-rom.
So every tweak would be nice. :)


gman(Posted 2005) [#43]
since your zip object is already global, i think its possible now with a little reworking of your function that calls the open/close zip. see your email :)


Chris C(Posted 2005) [#44]
is it possible to directly load an image/texture/sound from a zip/ram stream ?


Grisu(Posted 2005) [#45]
yepp.

The way it works:
1. You extract data from a zip to a stream
2. Loadimage etc this stream to be used with bmx

If I have understood you correctly...


gman(Posted 2005) [#46]
Grisu is correct. use the ExtractFile() method to get an instance of a TRamStream. pass this into the LoadImage() function.


TommyBear(Posted 2005) [#47]
Thanks gman! :)


gman(Posted 2005) [#48]
glad i could help :) thx for the initial effort TB!


Chris C(Posted 2005) [#49]
use the ExtractFile() method to get an instance of a TRamStream. pass this into the LoadImage() function.

thats real powerful...

Things like this real show the potential of max!


Grisu(Posted 2005) [#50]
Btw, is there a way to count how many files are in a certain zip file?


gman(Posted 2005) [#51]
@Grisu - sure is. getFileCount() method. should work on instances of both reader and writer.


Russell(Posted 2005) [#52]
Nice!

Good work, fellas! I hope Mark can incorporate this into the next update, as it seems to be fairly solid (and quite useful).

Russell

p.s. Unrelated, but the newest version of the free Blender 3D model/animate/render program just came out, and it has a very impressive list of features. Check it out! www.blender.org


Grisu(Posted 2005) [#53]
GetFileCount() worked. :)

But I think I have found a small issue with that:
The method counts the folders inside a zip as files as well!


Gavin Beard(Posted 2006) [#54]
Dont suppose this module does/will work under linux at all? i get an error saying bmax cant find interface pub.zipengine

EDIT* after successful recompile of mods it works ok, thx


Grisu(Posted 2006) [#55]
Note: Tested with bmx 1.18 and working...


Regular K(Posted 2006) [#56]
Sadly, it isn't working for me :( Nothing has been working for me in 1.18

I get this error "Can't find interface for module 'pub.zipengine'".

And if I try to build the module I get "Module does not match commandline module"

Sounds like I placed the module incorrectly, but its under pub.mod, it worked fine in 1.14.


Dreamora(Posted 2006) [#57]
The error "modules does not ..." means that you try to build and run the module like a regular code file which does not work.

to build it you must use the "Build modules" options.


Regular K(Posted 2006) [#58]
I got it to work after some tinkering (Build Modules would fail while compiling win32listbox so I just removed the MaxGUI modules and then built)


gman(Posted 2006) [#59]
FYI, i put a 1.18 built version out in my forums:

http://www.gprogs.com/forum/viewtopic.php?pid=72


Lomat(Posted 2006) [#60]
I'm having a small issue with the ZipEngine module (blitzmax 1.18).

When adding files to a zip archive i sometimes get a...

"TZipReader.scanLocalHeader(): unable to read from file"

runtime error. I have looked around and it does not relate to any specific file being added to the zip archive. It seems to be almost at random as sometimes the files get added without issue.

I created a small test script to try to narrow down the issue and i still get the same problem, sometimes i get the error othertimes it completes without issue. However in my test script if i put a delay() into the script then the error is 100% reproduceable.

Also the getFileCount() method always seems to return 0 when creating a new zip archive.

With the above issues it seems that the root of the problem is probably with the call to readFileList() in the ZipWriter.AddFile() method, however I dont have the ming stuff installed on my current machine so am unable to modify and rebuild the module myself to investigate further at the moment.

The test script is as follows...

SuperStrict
'
' You need an icons directory with files in.
'
Import PUB.ZipEngine

Local zw:ZipWriter = New ZipWriter
zw.OpenZip(AppDir$ + "/test.zip", False)

Local dh:Int = ReadDir("icons")
Repeat
	Local file:String = NextFile(dh)
	If file = "" Then
		Exit
	End If
	If file[..1] <> "." Then
		Local fileName:String = "icons/" + file
		If FileType(fileName) = 1 Then 
			Print("Adding " + fileName)
			zw.AddFile(fileName)
			Print("zip file count " + zw.getFileCount())
		End If
	End If
	Delay(300)
Forever

zw.CloseZip()
zw = Null



Grisu(Posted 2006) [#61]
@GMan:
I noticed another small glitch in the module.

When I create a new zip and display the contents of the zip file via winrar or other professional programs, the time and date of the files is totally messed up. 00.00.1980,00:00 or a random date of 1958! - When extracting the files to the hdd, the time and date stamp is back ok though?

I can live with that, but if you know how to fix that issue, that would be awesome too. :)


gman(Posted 2006) [#62]
thx for the bug reports. v2.05 is ready for download.

http://www.gprogs.com/forum/viewtopic.php?pid=72

i fixed both the added files count problem and the no datetime for added files issues.

enjoy :)


Grisu(Posted 2006) [#63]
Thanks!


JoshK(Posted 2006) [#64]
Thanks for putting this together.


gman(Posted 2006) [#65]
v2.06 is ready for download.

www.gprogs.com/forum/viewtopic.php?pid=72

i added an AddStream() method to the zipwriter to allow writing a stream to the zip.

also, please note there is now an RSS feed available in the ZipEngine forum description.


North(Posted 2006) [#66]
Thanks a bunch!


Space_guy(Posted 2006) [#67]
Sounds very usefull :) ill be using this functionality sometime in the future im sure.


Gabriel(Posted 2006) [#68]
That's great, thanks so much for adding this!


Grisu(Posted 2006) [#69]
Thanks.

Could you explain what the advantage of AddStream() is? Example?


JoshK(Posted 2007) [#70]
Can you please change the mod folder to gman.mod? You should not use pub.mod, because the contents will be deleted every time you sync mods.


Grisu(Posted 2007) [#71]
I sync mods every day and still the mod isn't deleted?

If you mean bmx updates. No need for that as bmx is seldom updated. The bmx 1.24 update is EIGHT MONTHS old.


plash(Posted 2008) [#72]
If a zip is opened with the append parameter does it overwrite existing files?

This wont work:
If zwriter.OpenZip("assets.zip", True) = True
	'Write stuff to stream (not shown)
	memstream.seek(0) 'Seek back
	zwriter.AddStream(memstream, "config")
End If


Archive shows up as corrupt, and has *two* 'config' files in it.

EDIT: memstream is a TBankStream.
EDIT2: Closing the zip removed the corrupt error but there are still two files in the zip with the exact same name, is there a way to remove a file from a zip? (I don't see and functions/methods to do so in the module)


plash(Posted 2008) [#73]
Using APPEND_STATUS_ADDINZIP doesn't work either.