ZipEngine 1.0 for BlitzMax
BlitzMax Forums/BlitzMax Module Tweaks/ZipEngine 1.0 for BlitzMax
| ||
Hi everyone, As promised here is the ZipEngine module that allows BlitzMax to use the zlib compression library to create zip files, add files to them and then extract the data out again, either to a file or a TRamStream. File are: www.tomkatgames.com/publicfiles/zipmodules.zip www.tomkatgames.com/publicfiles/zipsample.zip Extract the modules into your pub.mod folder NOTE: You'll be replacing the current zlib module with an updated one. Extract the sample into your samples folder ZipEngine for BMX is for Win32, MacOS and Linux. You must do a build of your modules before you try to compile the sample. Have fun! Tommy. Example Code: Rem ZipDemo Written by Thomas Mayer Notes: Demonstration of the zipping files up into an archive and then getting them back using zipEngine End Rem Import pub.zipengine Main() 'Call our entry point Function Main() ' Create our zipwriter object (creates zips) Local zwObject:ZipWriter = New ZipWriter ' Create our zipreader object (extracts from zips) Local zrObject:ZipReader = New ZipReader ' Example 1 ' Open the zip up for creation. We pass ' false in to tell the ZipWriter that we do ' not wish to append If ( zwObject.OpenZip("data.zip", False) ) Then zwObject.AddFile("testdata.txt") zwObject.AddFile("zipdemo.bmx") zwObject.CloseZip() End If ' Example 2 ' Open the zip file we just created and extract ' a file If ( zrObject.OpenZip("data.zip") ) Then zrObject.ExtractFileToDisk("testdata.txt", "extracted_testdata.txt") zrObject.ExtractFileToDisk("zipdemo.bmx", "extracted_zipdemo.bmx") zrObject.CloseZip() End If ' Example 3 Local fileBuffer:TRamStream = Null ' Open the zip file we just created and extract ' a file to ram If ( zrObject.OpenZip("data.zip") ) Then fileBuffer = zrObject.ExtractFile("testdata.txt") zrObject.CloseZip() End If End Function |
| ||
Thanks very much Tommy, it works wonders :) The ability to read data to a RamStream is very handy. Can you ask Mark if your changes to zlib can be official (making the ZipEngine itself an official Pub module would be very useful) |
| ||
Thanks Robert! I've sent it to Mark already, it's up to him now... but at least it's here for people to use now... Cya! Tommy. |
| ||
Can you ask Mark if your changes to zlib can be official (making the ZipEngine itself an official Pub module would be very useful) My thoughts exactly. Cool stuff! |
| ||
Thanks TommyBear! You have been very productive in a very short amount of time. |
| ||
Can we zip\unzip Incbin'd data fairly easily? This could be useful as well. Russell |
| ||
If I extract zipped data to RAM, how can I load it with the Blitz commandset after that? I would need to load LOTS of zipped images into system ram and directly load them into Blitz from there. - Possible? Also, is it possible to password protect a zip? |
| ||
Refering to my post above I managed to acess the file by directly loading up the filebuffer: image=LoadImage(fileBuffer, MaskedImage) The reason why it did not work is strange: When enableing debug mode compilation fails each time, but when i disable debug mode all runs fine??? Same goes for your example. In debug mode it fails. Any ideas? |
| ||
did you forget to build module in debug mode? ( build module only builds the modules for the mode you had active but you need them build in both modes ) Great work tommy, worked here without probs :) |
| ||
Thanks Marc, that did the job. Though my ZIP password protection question is still open... ;) |
| ||
Speaking of the ZLib library in general: I noticed that there is a 'license' for the zlib/png module in the doc/licenses folder, that appears to be missing some information: It's attributed to: Copyright (c) {year} {copyright holders} |
| ||
would need to load LOTS of zipped images into system ram and directly load them into Blitz from there. - Possible? Zipping images would just add extra overhead in most cases, since jpg/gif/png are pretty much uncompressable, it would just add time to decompress the images without saving any space... Unless you are looking to simply protect the media by using password-protected zip's, of course. |
| ||
Its pain to install 10.000 single files. So grouping them is a nice idea. This also saves some HDD space. So making zip "modules" is nice. How do I open a password protected zip with these commands? |
| ||
How do I open a password protected zip with these commands? Not yet supported I'll look at now and get back to you. cya! Tommy. |
| ||
Thanks! It would be nice to have some kind of "fail-check" if the zip is protected and if the password used failed or not. |
| ||
This is super, tanx Tommy! exactly what i was looking for :D |
| ||
ok, so i'm prob being silly but - when eva i try to use this mod i get a load of errors C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x1d9): undefined reference to `zipOpen' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x21a): undefined reference to `zipOpen' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x410): undefined reference to `zipOpenNewFileInZip' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x43e): undefined reference to `zipWriteInFileInZip' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x4ce): undefined reference to `zipClose' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x60d): undefined reference to `unzOpen' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x758): undefined reference to `unzLocateFile' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x79c): undefined reference to `unzLocateFile' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x7e8): undefined reference to `unzOpenCurrentFile' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x82e): undefined reference to `unzGetCurrentFileSize' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0x878): undefined reference to `unzReadCurrentFile' C:/blitzmax/mod/pub.mod/zipengine.mod/zipengine.a(zipengine.bmx.d.o.b)(code+0xa9f): undefined reference to `unzClose' i get this everytime any file is built, none ofr them will run, i did the 'bmk makemods' without error |
| ||
Gavin, have you done makemods in both debug and release mode? Quick Build turned off? |
| ||
worked like a treat, although i had to change blitzgl.win32.c to remove bglDeleteContext(0); else it gave me a conflict error, but thats nothing to do with this module. thanks for the great work |
| ||
While the .zip functionality looks *extremely* nice, personally I am very hesitant to mess with the 'original' modules... After all, any future 'official' updates through synmod can royally mess up your compatibility again. Any chance of seeing this released as an entirely *seperate* module rather than a zlib module replacement? |
| ||
i know its not this module causing the ogl prob. as i get it if i compile without it, all the demos seem to run if i comment out the line in the blitzgl file tho :S |
| ||
i know its not this module causing the ogl prob. as i get it if i compile without it, all the demos seem to run if i comment out the line in the blitzgl file tho :S |
| ||
Module works fine here. Only thing I miss is password protection for files... :) Any news? |
| ||
LoadStream isnt supported anymore, you should use LoadByteArray instead. |
| ||
Just recognised that as well... :( Can someone please update the module? |
| ||
With 1.10 I get a new error when rebuilding modules: Compiling:zipengine.bmx Compile Error: Identifier 'zipOpen' not found [C:/Programme/BlitzMax/mod/pub.mod/zipengine.mod/zipengine.bmx;26;4] Any fix for that possible??? :( |
| ||
What use the password Zip? |
| ||
hmm? |
| ||
Hope this works for everyone: http://legion.gibbering.net/grisu/zengine.zip |
| ||
Could you please put this zip archive online? I have made some synchmods, now the module doesn t work any more :( |
| ||
Always keep a fresh backup of the bmx dir in an archive. Make this right after your first install and syncmods. Then you have it for later.... :) |
| ||
Anyway thanks for reply and mail :). |
| ||
Whenever I try to build the new zlib module in the download I get: D:\BlitzMax\bin\bmk makemods -r pub.zlib Compiling:adler32.c Build Error: failed to compile D:/BlitzMax/mod/pub.mod/zlib.mod/adler32.c Any ideas? (I have MinGW properly installed) |
| ||
Dont worry got it working now :-) |
| ||
You smell of dead hookers... |
| ||
Tommybear are u still working on this or is it dead? |
| ||
Please edit, Noel - seems a bit out of place. Anyone got passworded zips going? those are very useful for protecting zips for whatever reason (just curious). Might want to make an app that is able to read them - you never know! |
| ||
@TommyBear - a big thank you :) if i might make a suggestion. i noticed that the Pub.Zlib mod is essentially unmodified with the exception of the imports into zlib.bmx. i moved the 8 new files: bmxsupport.c unzip.h/.c zip.h/.c ioapi.h/.c crypt.h to the root of the zipengine.mod folder and then moved the imports, consts, and extern statements from zlib.bmx to zipengine.bmx: Import Pub.ZLib Import brl.basic Import BRL.Stream Import "zip.c" Import "unzip.c" Import "ioapi.c" Import "bmxsupport.c" Rem bbdoc: Append Modes End Rem Const APPEND_STATUS_CREATE = 0 Const APPEND_STATUS_CREATEAFTER = 1 Const APPEND_STATUS_ADDINZIP = 2 Rem bbdoc: Compression methods End Rem Const Z_DEFLATED = 8 Rem bbdoc: Compression levels End Rem Const Z_NO_COMPRESSION = 0 Const Z_BEST_SPEED = 1 Const Z_BEST_COMPRESSION = 9 Const Z_DEFAULT_COMPRESSION = -1 Rem bbdoc: Compare modes End Rem Const UNZ_CASE_CHECK = 1 Const UNZ_NO_CASE_CHECK = 2 Rem bbdoc: Result Codes End Rem Const UNZ_OK = 0 Const UNZ_END_OF_LIST_OF_FILE = -100 Const UNZ_EOF = 0 Const UNZ_PARAMERROR = -102 Const UNZ_BADZIPFILE = -103 Const UNZ_INTERNALERROR = -104 Const UNZ_CRCERROR = -105 Extern Rem bbdoc: Open new zip file (returns zipFile pointer) End Rem Function zipOpen:Byte Ptr( fileName:Byte Ptr, append:Int ) Rem bbdoc: Closes an open zip file End Rem Function zipClose( zipFilePtr:Byte Ptr, archiveName:Byte Ptr ) Rem bbdoc: Open a file inside the zip file End Rem Function zipOpenNewFileInZip( zipFilePtr:Byte Ptr, fileName:Byte Ptr, zipfi, .. extrafield_local, size_extrafield_local, .. extrafield_global, size_extrafield_global, .. comment:Byte Ptr, compressionMethod:Int, .. level:Int ) Rem bbdoc: Write into a zip file End Rem Function zipWriteInFileInZip( zipFilePtr:Byte Ptr, buffer:Byte Ptr, bufferLength:Int ) Rem bbdoc: Open a zip file for unzip End Rem Function unzOpen:Byte Ptr( zipFileName:Byte Ptr ) Rem bbdoc: Return status of desired file and sets the unzipped focus to it End Rem Function unzLocateFile:Int( zipFilePtr:Byte Ptr, fileName:Byte Ptr, caseCheck:Int ) Rem bbdoc: Opens the currently focused file End Rem Function unzOpenCurrentFile:Int( zipFilePtr:Byte Ptr ) Rem bbdoc: Gets info about the current file End Rem Function unzGetCurrentFileSize:Int( zipFilePtr:Byte Ptr ) Rem bbdoc: Read current file, returns number of bytes End Rem Function unzReadCurrentFile:Int( zipFilePtr:Byte Ptr, buffer:Byte Ptr, size:Int ) Rem bbdoc: Close current file End Rem Function unzCloseCurrentFile( zipFilePtr:Byte Ptr ) Rem bbdoc: Close unzip zip file End Rem Function unzClose( zipFilePtr:Byte Ptr ) EndExtern a quick change from LoadStream() to LoadByteArray() and it compiles just fine. this essentially allows you to keep Pub.ZLib unmodified while gaining the benefits of this fantastic mod. |
| ||
great work |
| ||
anyway to unzip the er zip without knowing file names? |
| ||
*bump* |
| ||
i can provide some code for that. i recently ported the Irrlicht zip engine and it reads zip header to get a list of file entries. it doesnt do a * type thing but it will give you the ability to loop through all of them and unzip. unless someone else beats me to the punch i should be able to provide you the essentials sometime this evening (-6 GMT is my TZ). |
| ||
ya, that would be cool |
| ||
this is kinda clunky cause its pulled from the Irrlicht port and i had to strip out a bunch of stuff. this works as is but it should also allow you to roll your own based on how it works.Strict Framework BRL.Blitz Import BRL.FileSystem Import BRL.LinkedList Import BRL.StandardIO Public ' ---------------------------------------------------------------- ' example on using the classes ' ---------------------------------------------------------------- ' open the zip to read Local test:TStream=ReadFile("D:/develop/ggtray.zip") If test Then ' read in the file headers Local reader:TZipReader=TZipReader.create(test,False,False) ' we are done with the file CloseStream(test) Print("-------------------------------") Print(reader.getFileCount()+" files in the zip") ' show the files in the zip Local temp:SZipFileEntry For temp=EachIn reader.FileList Print(temp.zipFileName) Next EndIf ' ---------------------------------------------------------------- ' the supporting cast ' ---------------------------------------------------------------- ' the fields crc-32, compressed size ' And uncompressed size are set To zero in the Local ' header Const ZIP_INFO_IN_DATA_DESCRIPTOR:Short = $0008 Type SZIPFileDataDescriptor Extends PACK_STRUCT Const size:Int=12 Field CRC32:Int Field CompressedSize:Int Field UncompressedSize:Int Method fillFromBank(databank:TBank,offset:Int=0) CRC32=PeekInt(databank,offset) CompressedSize=PeekInt(databank,offset+4) UncompressedSize=PeekInt(databank,offset+8) EndMethod EndType Type SZIPFileHeader Extends PACK_STRUCT Const size:Int=30 Field Sig:Int Field VersionToExtract:Short Field GeneralBitFlag:Short Field CompressionMethod:Short Field LastModFileTime:Short Field LastModFileDate:Short Field DataDescriptor:SZIPFileDataDescriptor Field FilenameLength:Short Field ExtraFieldLength:Short Method New() DataDescriptor=New SZIPFileDataDescriptor EndMethod Method fillFromBank(databank:TBank,offset:Int=0) Sig=PeekInt(databank,offset) VersionToExtract=PeekShort(databank,offset+4) GeneralBitFlag=PeekShort(databank,offset+6) CompressionMethod=PeekShort(databank,offset+8) LastModFileTime=PeekShort(databank,offset+10) LastModFileDate=PeekShort(databank,offset+12) DataDescriptor.CRC32=PeekInt(databank,offset+14) DataDescriptor.CompressedSize=PeekInt(databank,offset+18) DataDescriptor.UncompressedSize=PeekInt(databank,offset+22) FilenameLength=PeekShort(databank,offset+26) ExtraFieldLength=PeekShort(databank,offset+28) EndMethod EndType Type SZipFileEntry Field zipFileName:String Field simpleFileName:String Field path:String Field fileDataPosition:Int ' position of compressed data in file Field header:SZIPFileHeader Function create:SZipFileEntry() Return New SZipFileEntry EndFunction Method New() header=New SZipFileHeader EndMethod Method Less:Int(other:SZipFileEntry) Return simpleFileName.Compare(other.simpleFileName)<0 EndMethod Method EqEq:Int(other:SZipFileEntry) Return simpleFileName.Compare(other.simpleFileName)=0 EndMethod Method Compare(other:Object) If SZipFileEntry(other) Return simpleFileName.Compare(SZipFileEntry(other).simpleFileName) Else Return -1 EndMethod EndType Type TZipReader Field zipFile:TStream Field FileList:TList ' core::array<SZipFileEntry> Field IgnoreCase:Int Field IgnorePaths:Int Method New() FileList=New TList EndMethod Function create:TZipReader(file:TStream,bIgnoreCase:Int,bIgnorePaths:Int) If Not file Then Return Null Local retval:TZipReader=New TZipReader retval.zipFile=file retval.IgnoreCase=bIgnoreCase retval.IgnorePaths=bIgnorePaths ' scan Local headers While retval.scanLocalHeader() Wend ' prepare file index For binary search retval.FileList.sort() Return retval EndFunction ' returns count of files in archive Method getFileCount:Int() Return FileList.Count() EndMethod ' returns data of file Method getFileInfo:SZipFileEntry(index:Int) If index<0 Or index>=FileList.Count() Then Throw "TZipReader.getFileInfo(): Invalid index "+index Return SZipFileEntry(FileList.ValueAtIndex(index)) EndMethod ' returns SZipFileEntry Method findFile:SZipFileEntry(simpleFilename:String) Local retval:SZipFileEntry=Null Local entry:SZipFileEntry=SZipFileEntry.create() entry.simpleFileName = simpleFilename If (IgnoreCase) Then entry.simpleFileName=entry.simpleFileName.ToLower() If (IgnorePaths) Then deletePathFromFilename(Varptr(entry.simpleFileName)) Local link:TLink=FileList.FindLink(entry) If link retval=SZipFileEntry(link.Value()) Else ?Debug For Local i:Int=0 To FileList.Count()-1 If (getFileInfo(i).simpleFileName = entry.simpleFileName) DebugLog("File "+entry.simpleFileName+" in archive but Not found.") Exit EndIf Next ? EndIf Return retval EndMethod ' scans For a Local header, returns False If there is no more Local file header. Method scanLocalHeader:Int() Local tempbank:TBank Local entry:SZipFileEntry=SZipFileEntry.create() entry.fileDataPosition = 0 ' populate the header with what was read If Not entry.header.fillFromReader(zipFile,SZIPFileHeader.size,0,0) Then Throw "TZipReader.scanLocalHeader(): unable to read from file" DebugLog(entry.header.CompressionMethod) If entry.header.Sig <> $04034b50 Then Return False ' Local file headers End here. ' read filename entry.zipFileName=ReadString(zipFile,entry.header.FilenameLength) extractFilename(entry) ' move forward length of extra field. If (entry.header.ExtraFieldLength) SeekStream(zipFile,StreamPos(zipFile)+entry.header.ExtraFieldLength) EndIf ' If bit 3 was set, read DataDescriptor, following after the compressed data If (entry.header.GeneralBitFlag & ZIP_INFO_IN_DATA_DESCRIPTOR) ' read data descriptor entry.header.DataDescriptor.fillFromReader(zipFile,SZIPFileDataDescriptor.size,0,0) EndIf ' store position in file entry.fileDataPosition = StreamPos(zipFile) ' move forward length of data SeekStream(zipFile,StreamPos(zipFile)+entry.header.DataDescriptor.CompressedSize) ?Debug DebugLog("added file "+entry.simpleFileName+" from archive") ? FileList.AddLast(entry) Return True EndMethod ' splits filename from zip file into useful filenames And paths Method extractFilename(entry:SZipFileEntry) ' check length of filename If Not entry.header.FilenameLength Then Return ' change the case if ignore is on If (IgnoreCase) Then entry.zipFileName=entry.zipFileName.ToLower() ' make sure there is a path Local thereIsAPath:Int=(ExtractDir(entry.zipFileName)<>"<bad_dir>") ' store just the filename entry.simpleFileName = StripDir(entry.zipFileName) ' store an empty path If (thereIsAPath) Then entry.path = ExtractDir(entry.zipFileName) Else entry.path="" ' simpleFileName must be zipFileName if not to ignore paths If Not IgnorePaths entry.simpleFileName = entry.zipFileName ' thanks To Pr3t3nd3r For this fix EndIf EndMethod ' deletes the path from a filename Method deletePathFromFilename(filename:String Ptr) filename[0]=StripDir(filename[0]) EndMethod EndType ' generic type for reading in PACK_STRUCT structures from files Type PACK_STRUCT Global size:Int=0 Method fillFromBank(bank:TBank,start:Int) EndMethod ' returns true if successful Method fillFromReader:Int(fileToRead:TStream,tbsize:Int,readeroffset:Int=0,bankoffset:Int=0) If Not fileToRead Then Return False ' create the bank Local structbank:TBank=CreateBank(tbsize) ' read from the file structbank.Read(fileToRead,readeroffset,tbsize) ' populate the STRUCT with what was read fillFromBank(structbank,bankoffset) ' clear out the bank structbank=Null Return True EndMethod Method getBank:TBank() Return Null EndMethod EndType |
| ||
@gman: Could u create a zip with all source code files? And add unzipping of password-protected zips? |
| ||
@Grisu - upon your request i went ahead and updated my version of the ZipEngine mod to allow passwords... but i want to get with TommyBear and see if he wishes to implement those changes into his. if he does not, then i will go ahead and write it as an extension to the ZipEngine mod. if he does then i will send him what i have. @TommyBear - would you mind if i sent you my modified version of ZipEngine? it includes the following features: a) ZipEngine mod contains all special files so Pub.ZLib is unmodified. b) password support for zip/unzip c) enumeration of files in the zip |
| ||
Thanks for your work you put in this! I would suggest you e-mail Tommy directly as he doesn't seem reading here any more. :( |
| ||
As far as I can see, Tommy put the code in the Public Domain. You can go right ahead and post your new module. I hope you do It's really great stuff, and a very good thing to get it working with an unmodified Pub.Zlib. |
| ||
P.S.: Please start a new fresh topic then as well. "ZipEngine 2.0" :) What you could also add is some crc32-checks for zip files if you are out of new ideas. I have made some, but they are rather slow... :) |
| ||
i have sent an email to TB to see what his preferences are so based on his reply i will get something out. as far as features go, unless there are any bugs in it i will probably let it sit for a while. im currently pretty hot n' heavy on the next Irrlicht mod :) |
| ||
ok, lets hope tb checks his mails better than the forums here! :P |
| ||
Any news? :( I really need this module... |
| ||
none yet. i will compile the files into a distrib based on the format of the ZE1.0 distrib and call it ZE2.0. if TB disapproves i will then shift it out. should be sometime pretty late this evening (im -6 GMT). |