NewsForumsCodeLogsGallerySpecsSearch

ZipStream module

BlitzMax Forums/BlitzMax Programming/ZipStream module

Koriolis(Posted 2007) [#1]
This module adds two new stream protocols for zip support in streams : "zip" and "zip?"
"zip" is for reading files that are in the specified zip.
"zip?" is for reading files that are in the specified zip OR (if not found in the zip) on the hard drive (or any other location supported by streams).

The zip stream works over another stream. This means that you can not only access zip files on the hard drive, but also zp files that are incbined, or even zip files within other zip files, and so on.

Example of valid urls: "zip::myGame/myZip.zip//someFile.txt", or even "zip::incbin::myGame/myZip.zip//someFile.txt" if the zip file is incbined.

So you could do by example: LoadText("zip::myGame/myZip.zip//someFile.txt")

Use as you please.

Grab it here (you need to be logged in the forum to see the download link):

http://www.koriolis-fx.com/forum/index.php?topic=15.0


popcade(Posted 2007) [#2]
Seems it load directly into mem and won't generate temp file, nice.

Thanks for the handy module


Tachyon(Posted 2007) [#3]
MacOS and Linux compatible?


Koriolis(Posted 2007) [#4]
Seems it load directly into mem and won't generate temp file, nice.
Yes. That's the whole point, the zip stream works over another stream, making it very versatile, and letting you access a zipped file wherever it's stored.

MacOS and Linux compatible?
It's based solely on ZLib so it should definitely work on Mac and Linux. I just didn't bother to compile it on Mac or Linux yet. If anyone wants to compile it and check that it works OK for these platforms, I will happily include the resulting binaries.


degac(Posted 2007) [#5]
Thanks for the mod but I have a problem, maybe you can explain me where is the error

Local te:tstream = OpenStream("zip::test_bmax.zip//test_zip.txt")

If te
While Not Eof(te)
	Print ReadLine(te)
Wend
CloseStream(te)
Else
	Print "no stream found"
End If

Local testo:String=LoadText("zip::test_bmax.zip//test_zip.txt")

Print "Text: "+testo


I have created a .zip file (test_bmax.zip) that contains the file test_zip.txt.
I have the error 'No stream found' (te=NULL) but LoadText() works perfectly. What I miss?


Koriolis(Posted 2007) [#6]
Silly question, but did you actually import koriolis.zipstream?


xlsior(Posted 2007) [#7]
checking this one out now, sounds like it could definitely come in handy...


popcade(Posted 2007) [#8]
If it support password it'll become killer.

Can it add something like $zippass to specify password?


Filax(Posted 2007) [#9]
Great module Koriolis !! Many thanks :)

Good job :)


degac(Posted 2007) [#10]

Silly question, but did you actually import koriolis.zipstream?


Of course, the second part of the program 
Local testo:String=LoadText("zip::test_bmax.zip//test_zip.txt")
Print "Text: "+testo

works perfectly: the 'zip::' protocol is recognized. but only for the LoadText command.


Filax(Posted 2007) [#11]
"but only for the LoadText command" ???

I can't load an image from this zip ??


degac(Posted 2007) [#12]
If FileType("zipped_image.zip")
	Print "File exists"
End If
Local zimage:timage = LoadImage("zip::zipped_image.zip/back_muro_2.png")
If zimage
	Print "Image loaded"
	Else
	Print "Image NOT loaded"
End If

This is a test for loading an image from a file 'zipped_image.zip'. But - for me - it doesn't work. Can anyone else do some tests please? So only to understand if there is a problem with my configuration.


Koriolis(Posted 2007) [#13]
If it support password it'll become killer.
Yes it does. You can register a zip password globally, using SetZipStreamPasssword. Once done you can access the passworded file just like a unpassworded file.

@degac: you need a double slash to separate the zip path from the file inside the zip.
With Local zimage:timage = LoadImage("zip::zipped_image.zip/back_muro_2.png") youare trying to load the zip file "zipped_image.zip/back_muro_2.png" but you want to access the zip file "zipped_image.zip" and get the file "back_muro_2.png" inside of it. You need to do:
Local zimage:timage = LoadImage("zip::zipped_image.zip//back_muro_2.png")

But as said below there is another problem then.

I can't load an image from this zip ??
I just tried with LoadPixmapPNG and it works.
I then tried with LoadImage and apparently it doesn't work because the ZipStream is not seekable. Is that the error you get ?
The problem is I don't know yet how to make the zip stream seekable, as I can't find any function in zlib that lets me do that.


assari(Posted 2007) [#14]
Perhaps you can try the loadimage(loadbank()) trick as when trying to loadimage from a http:: stream


Koriolis(Posted 2007) [#15]
Well, actually I just made a simple fix: For the special case when Seek(0) or Seek( current pos ) is called, the seek now works. This is limited, but is enough to make LoadImage() work with the zip stream.

Please redownload the module and let me know if you still encounter problems.


Grisu(Posted 2007) [#16]
What's the difference to GMan's Zip module?


Koriolis(Posted 2007) [#17]
GMan's zip module provides everything you need to mainpulate zip files in pretty much every way you wish, BUT using a special dedicated API.
On the contrary, the zipstream module let's you read zipped files using the standard TStream interface. This means that any piece of code that works on a TStream can work on zipped files.
If you (or anyone) have written code that works on TStream, you don't have to write another version to handle zipped files: just use the "zip::" protocol and that's it.
By example you can already use the standard functions LoadText and LoadImage with zipped files.


degac(Posted 2007) [#18]
Ok. New version works perfectly with LoadImage(zip::).
Thank you very much!


Grisu(Posted 2007) [#19]
Thanks for the explanation.

Btw: Gman's version can also extract data directly into the ram and read it from threre.


Koriolis(Posted 2007) [#20]
Gman's version can also extract data directly into the ram and read it from threre.

Yes, but it's still not a stream protocol, and perhaps more important it requires to extract the whole file into memory. On the other hand, my zip stream extracts data on the fly.
I think both modules are very much complementary.
By example I personnaly use GMan's zipengine to create zip files, and my zip stream to read it back (from another program).


Grisu(Posted 2007) [#21]
So your solution is faster in terms of extracting files?


Koriolis(Posted 2007) [#22]
It should overall be a bit faster, yes. To be clear, the raw extraction is probably slower, but there is no need for a second pass (reading from memory). More important, there is no need for a block of memory as big as the file itself.


Gavin Beard(Posted 2007) [#23]
thx, this works great, is very fast too, have used it today just to test storing images and loading, very good.

only issue i have is with passwords:


Graphics 640,480;
SetZipStreamPasssword("test.zip","gavsta")
Global testimg:timage =LoadImage("zip::test.zip//untitled-1.jpg");

While Not KeyHit(KEY_ESCAPE)
Cls
DrawImage(testimg,0,0)
Flip
Wend

this always comes back with "Attempt to Access field or method of Null Object". if i turn off the password side it works fine?

**EDIT**
just ignore me, i'm having one of those days, would help if i had called the file test.zip and not test.zips


xlsior(Posted 2007) [#24]
Question: is it possible to *combine* two methods like incbin and zipstream, to add another layer of protection?

(e.g. resources inside of password-protected incbinned ZIP)


Filax(Posted 2007) [#25]
Here is an example with incbinned zip :)
Nice module ! really usefull !




Koriolis(Posted 2007) [#26]
Glad you like it :)


JoshK(Posted 2007) [#27]
As a temporary fix could you use replace the seek method with one that goes to the beginning and reads the offset number of bytes?


Koriolis(Posted 2007) [#28]
I'll check that, *maybe* this week end. If I forget, remind me.
I had already thought of doing it that way, I just didn't bother. Howerver, to avoid being totally inefficient I'd need to make the stream buffered (or else, if it happens that you do something like seeking to N, N+1, N+2 etc, then it will totally ineffieicnt as for every single byte it will restart from the beginning).


Boulderdash(Posted 2007) [#29]
Wicked, This is just what I need, If this can decompress Incbin'd Zips straight to RAM (Not Temp HDD file)

I dont have time to try this out right now but im definitely interested in using this in my software

Long Live BlitzMax and its Modules


bruZard(Posted 2007) [#30]
it does'nt work: i get "Stream is not seekable"


bruZard(Posted 2007) [#31]
i've found that the zipstream does'nt work with pantsons MPEG module:

Local movie:Tmpeg = OpenMPEG("zip::intro.zip//intro.ogg")


Koriolis(Posted 2007) [#32]
Yes, as stated earlier the stream is not seekable (yet) except for 2 special cases. I will do my best to fix that when I have time.


xlsior(Posted 2007) [#33]
i've found that the zipstream does'nt work with pantsons MPEG module:


Is there even any benefit doing so?

zip compression is negligible on an MPEG file, and I'd think that the added overhead of unzipping it would outweigh the minor size savings...


Grisu(Posted 2007) [#34]
Is there a command for counting the total files inside a zip file?


Dabz(Posted 2007) [#35]
Wow, really handy! :)

Thanks for sharing such a great mod free! ;)

Dabz


seriouslee(Posted 2007) [#36]
Very handy tool! Really like the way it integrates into the commands.

degac:

I have created a .zip file (test_bmax.zip) that contains the file test_zip.txt.
I have the error 'No stream found' (te=NULL) but LoadText() works perfectly. What I miss?

I played with this because I need it too. I found I can open the stream with:

Local te:tstream = OpenStream("zip::test_bmax.zip//test_zip.txt", true, false) where false tells it the file is NOT writable.

Hope that helps.

Silly question: (I really new to posting) How do you format text in the reply window and add things like the scrolling text box?


Koriolis(Posted 2007) [#37]
This is perfectly normal, the stream is read only. The intent is really to use zip files as in Quake by example : as big "pak" data files. For *writing* into the stream, either simply create the zip files using Winzip or whatever, or use GMan's ZipEngine module.
Thus you should not try to open the stream in write mode.
ReadStream(...) will work, as well as
OpenStream(..., ..., false).
But WriteStream(...) or OpenStream(..., ..., true) won't.


seriouslee(Posted 2007) [#38]
I've run into a problem. I can open text files, image files, and font files in the zip with no problem, but when sound is introduced into the game (not in the zip, just using standard files) I get "Attempt to Access field or method of Null Object" from weird places in my code and the standard editor bombs if I try to access the debug info. If I take out the zip:: references the code runs clean. Any ideas?


Koriolis(Posted 2007) [#39]
Most probably, you tried to load the sound file using the ZipStream, the loading failed, you didn't check that the load was successfull or not and tried to play that Null audio sample.

As mentioned earlier, there is still an issue : the stream is not seekable. I will look at the the issue when I have more time.
So if you load a sound file that is loaded using TStream.Seek, then it won't work. I tried with Wav files and it works, but it all depends on the file format you're loading, and the way the loading code is written.


xlsior(Posted 2007) [#40]
Most probably, you tried to load the sound file using the ZipStream, the loading failed, you didn't check that the load was successfull or not and tried to play that Null audio sample.


workaround?

loadsound(loadbank("zip::mysound.zip//sound.ogg"))


Koriolis(Posted 2007) [#41]
yes, this is the logical workaround, and will always work.
However I can't pretend it wouldn't be better to have it work without this workaround.
I *will* try to fix this, I just don't know when.


Koriolis(Posted 2007) [#42]
I have finally fixed the limitation concerning TZipStream : it is now entirely seekable. By example I can now load zipped ogg files (which require the stream to be seekable), it works perfectly (don't tell me zipping ogg files is stupid, it's just a test, and actually CAN make sense).
You can download the new version at the same place.


degac(Posted 2007) [#43]
Hi

I'm testing your last zipstream module, but I have 'Stream is not seekable' when I try to load a .ogg file.
I also have used the 'buf::' protocol, without success. Only with CreateBufferedStream() function it works.
(I'm using Import koriolis.bufferedstream)

Can you post a working example?

Thanks again


Koriolis(Posted 2007) [#44]
Working example:
Import koriolis.zipstream
Local sound:TSound = LoadSound("zip::D:\BlitzMax\samples\digesteroids\sounds\menu.zip//menu.ogg")
PlaySound(sound)
While Not KeyHit(1)
Wend

For your problem, it's likely that you're still using the old version. One reason might be that the zip files containing the modules will be extracted to zipstream.mod/zipstreamkoriolis.mod and bufferedstream.mod/bufferedstream.mod. But in the BlitzMax "mod" folder, you must have mod/koriolis.mod/zipstream.mod and mod/koriolis.mod/bufferedstream.mod. So be carefull to copy the zip content at the right place.


Chroma(Posted 2008) [#45]
Hi. This module is great but I'm not getting the password feature. Is it only meant for incbinned zip folders? Is it supposed to actually set a password for a zip folder? I can set a password then exit the program and open the zip folder normally using no password.

EDIT: Nm I got it. Create your zip folder. Click file->add password. Awesome!


Filax(Posted 2008) [#46]
Just few words to say that i'm using your module all the time! :)

Thanks Koriolis for this great piece of code :)


Koriolis(Posted 2008) [#47]
Thans for the thanks :)


Tachyon(Posted 2008) [#48]
Koriolis: this is an awesome module! Thank you so much for sharing it.

Now I must ask: any possibility of getting zip writing added? This would make for a great save game function when there are many files generated: throw them all into a zip file to keep them organized, and then password lock it to prevent hacking!

Either way, thanks again for sharing!


Koriolis(Posted 2008) [#49]
It is certainly doable, however I do no have much time for this. In addition I believe write support is much less important here, and if you really need it I highly recommend using the ZipEngine module from GMan.

If anyone feels like adding write support, I will happily test that and incorporate it in the module.


Stu_ovine(Posted 2008) [#50]
An excellent addition, thanks for sharing the code.


For those wanting to use a subfolder witin the ZIP file replace your "\" with "/" before you open the file.


filename = filename.Replace("\", "/")



Filax(Posted 2008) [#51]
Hi Koriolis : Any chances to get it work with blitz3D Sdk ?


Koriolis(Posted 2008) [#52]
Not much.
The module is totally dependant on BlitzMax streams.
blitz3D Sdk is just an SDK, I don't believe they have included any kind of general stream system in it.
Now let's imagine you add a general stream framework on top of blitz3D Sdk, or use an existing one (say C++ stl streams if you're using C++) it will be mostly worthless as the SDK itself won't be using it and thus you have zero chance of having the sdk load a model from yor zip.
Plus I don't own blitz3D Sdk and have no interest in it.

The only possibility would be if you're using BlitzMax AND the BlitzMax version of blitz3D Sdk is based on BlitzMax streams (which basically requires that the sdk is written in BlitzMax). In this case, the zipstream module should work out of the box with the sdk.


Filax(Posted 2008) [#53]
Ok thanks :)


Grisu(Posted 2009) [#54]
Is there a way to detect if a file stored inside a zip is present at all?


Brucey(Posted 2009) [#55]
What happens when you try to read a file from the zip that does not exist?


plash(Posted 2009) [#56]
What happens when you try to read a file from the zip that does not exist?
I'm itching to say.. :P

Grisu: If you try to open a stream to a zip path (or any path, for that matter), and it does not exist, your stream should be Null.


Grisu(Posted 2009) [#57]
Well, the app crashes completely which is not good.

I did check for the null stream but it doesn't help here. ;)

@Brucey:
See (TuneIn_Station)line:
Local zstation:String = LoadText("zip::"+fullzipdir)

Probably it's the loadtext function that has no fail check?! :D


Brucey(Posted 2009) [#58]
Probably it's the loadtext function that has no fail check

From memory... LoadText used to crash if given a Null filename - don't know if it was ever... fixed.


Grisu(Posted 2009) [#59]
It wasn't... :D

So what to do? Blame the coders... ;)


Brucey(Posted 2009) [#60]
You can always wrap the call in a Try Catch block... :-)


SLotman(Posted 2011) [#61]
Nevermind... it was a filename capitalization error. thanks!

Last edited 2011


Takis76(Posted 2012) [#62]
This is very interesting.
Where do I download?


matibee(Posted 2012) [#63]
The download instructions are here: http://www.blitzbasic.com/Community/posts.php?topic=71734#801600

Or are you having trouble?


Takis76(Posted 2012) [#64]
I wasn't able to register in the forums so I wasn't able to download.


Takis76(Posted 2012) [#65]
Ok now I downloaded the zipsteeam library.

I created one zip file with name "test.zip"
and I put one image .bmp with name "test_image.bmp"


[code]
global test_image:TImage
test_image = LoadImage("zip::test.zip//test_image.bmp", MASKEDIMAGE)
DrawImage(test_image, 0, 0, 0)
[/cod]

I have this error:

Unhandled Exception:Attempt to access field or method of Null object

What does this means?


Midimaster(Posted 2012) [#66]
did you add this line to your code?

SuperStrict   
Import koriolis.zipstream
global test_image:TImage
test_image = LoadImage("zip::test.zip//test_image.bmp", MASKEDIMAGE)
DrawImage test_image, 0, 0, 0



Takis76(Posted 2012) [#67]
Wow , perfect the image loaded. :D

I wasn't loaded the library

If I my zip have password how to load the image with the zip password?

I found it

I put this code before load everything from zip

SetZipStreamPassword("test.zip", "12345")


I think , this solve all my pack file problems.
Thank you very very much AGAIN :D

Last edited 2012


Pineapple(Posted 2012) [#68]
Is it possible to get this to work together with LoadDir?


Kippykip(Posted 2013) [#69]
Where can I get this now? Link is dead with that irritating robots.txt

EDIT:
Never mind found it
http://retroremakes-framework.googlecode.com/svn/trunk/modules/koriolis.mod/
EDIT 2: ZipStream appears not to work with minib3d when using loadMesh etc


Midimaster(Posted 2013) [#70]
MiniB3D with Koriolis on PC and Mac?

I often did combinations of Koriolis Zipstream and MiniB3D and it works on PC. On the MAC I needed to copy the files to a temp dir first.

DatenPfad$="ZIP::Incbin::tt1.zip//"

Global ImagePfad$=DatenPfad
?MacOS
	Print "mac os "
	ImagePfad=AppleResourceDir()
	Local CopyListe:TStream, CopyDatei$
	CopyListe=OpenFile(DatenPfad + "MacCopy.lst")
		While Eof(CopyListe)=0
			CopyDatei=ReadLine(CopyListe)
			Print "copy now " + copydatei
			If FileType(ImagePfad+CopyDatei)=0
				Print "yes"
				CopyFile DatenPfad +CopyDatei, ImagePfad + CopyDatei
			EndIf
		Wend
	CloseFile CopyListe
?



Cruis.In(Posted 2013) [#71]
this comes in a module download with timeline as well. Plus I think the area which hosts all blitz modules has it. Do a search in google I don't have a bookmark.