Blitz Zip Userlib (Create an manipulate zip files)

BlitzPlus Forums/BlitzPlus Programming/Blitz Zip Userlib (Create an manipulate zip files)

Phil Newton

(Posted 2006) [#1]

Latest Version: 1.2 (27th August, 2007)

Here's a little something I've been working on for the past few days. It's a set of userlib declarations and wrapper functions that allow you to extract files from zip archives, and also to create your own zip files.

A quick example of creating a zip and adding files to it:

; Open our new archive
Local zipOut	= ZipApi_CreateZip("my-zip.zip")
 
; Add some test files
ZipApi_AddFile(zipOut, "example1.bb")
ZipApi_AddFile(zipOut, "example2.bb")
ZipApi_AddFile(zipOut, "example3.bb") 

; Close the zip
ZipApi_CloseZip(zipOut)

You can also add data to a zip file directly from a Blitz bank.

Download: Download Now (119.4KB)

More Information: Blitz.ZipApi - Project Homepage

Online Documentation: http://docs.sodaware.net/blitz.zipapi/

Any feedback/bug reports are greatly appreciated.

Dock	(Posted 2006) [#2]

Wow, great. I haven't tried this yet but it's a really useful feature to be able to have.

Andres

(Posted 2006) [#3]

Very good stuff, is there anyway to use passwords too?

Phil Newton

(Posted 2006) [#4]

@Dock: Thanks!
@Andres: Not yet, but I plan to add it in the next version.

Pineapple

(Posted 2006) [#5]

This caught my eye... great stuff! :D

Thanks for sharing

Dabz

Phil Newton

(Posted 2007) [#6]

It took its sweet time, but version 1.1 is now available (see first post). This version supports passwords, both for extracting files and adding them to a new archive.

b32	(Posted 2007) [#7]

Wow .. this is an amazing library. I've managed to extract a .zip file today, and it works very nice.

I tried using one of my own archives. At first, the first example stopped after showing one file.
I changed to loop to:
Until ZipApi_GotoNextFile(zipIn) ;;ZIPAPI_UNZ_END_OF_LIST_OF_FILE
and it showed all files in the archive. Maybe that happened because my archive had only one file in the first folder, all other files are placed in subfolders.
Then I converted this example to extract the archive.
Because the .zip I used contained (sub)folders, I needed Mr. Picklesworth FixPath routine from the archive to create the folders. (For that routine, I needed to replace the "/" with "\" in the searchpath.)

Finally, this is the code I used:

Graphics 800, 600, 0, 2
; --------------------------------------------------
; --
; -- examples/example1.bb
; --
; -- A short example of how to iterate through a 
; -- ZIP file's contents using the ZipApi library.
; --
; --------------------------------------------------

; Include required libraries
Include "../Blitz_Basic_Bank.bb"
Include "../Blitz_File_FileName.bb"
Include "../Blitz_File_ZipApi.bb"

; Open the zip file
Local zipIn = ZipApi_Open("test-file.zip")

; Check the zip file was valid
If zipIn = 0 Then
	; Not a valid ZIP file - display error and exit program
	Print "There was an error whilst trying to open the zip file."
	WaitKey()
	End
EndIf

; Set file pointer to first file
ZipApi_GotoFirstFile(zipIn)

; Print a fancy header
Print LSet("FileName", 30) + RSet("Packed Size", 14) + RSet("Normal Size", 14) + RSet("Ratio", 10)

; Begin iterating through files
Repeat
	
	; Get the current file's information
	Local fileInfo.ZIPAPI_UnzFileInfo	= ZipApi_GetCurrentFileInfo(zipIn)
	
	; Generate fancy file data, containing file name, size and compression ratio
	Local fileData$	= ""
	Local compressionRatio# = Float((fileInfo\CompressedSize) / Float(fileInfo\UnCompressedSize)) * 100.0

	fn$ = Replace$(fileinfo\Filename, "/", "\")
	FixPath(fn$)
	
	If FileType(fn$) <> 2 Then
		outname$ = ZipApi_ExtractFile(zipIn, fileinfo\Filename, fileinfo\Filename$)
	End If
		
	fileData$	= LSet(fileInfo\FileName, 30)
	fileData 	= fileData + RSet(fileInfo\CompressedSize, 14)
	fileData 	= fileData + RSet(fileInfo\UnCompressedSize, 14)
	fileData 	= fileData + RSet(compressionRatio + "%", 10)
	
	; Cleanup & output
	ZIPAPI_UnzFileInfo_Dispose(fileInfo)
	Print fileData
	
	
Until ZipApi_GotoNextFile(zipIn) ;;ZIPAPI_UNZ_END_OF_LIST_OF_FILE

; Close & Cleanup
ZipApi_Close(zipIn)

Print "Press any key.."
WaitKey()

End


;-----------------------------------------------------------------------------------------------------
;-----------------------------------------------------------------------------------------------------

; ID: 1567
; Author: Mr. Picklesworth
; Date: 2005-12-17 15:36:23
; Title: FixPath
; Description: Creates every missing folder in a file path

Function FixPath(path$)
;Creates every missing folder in a path. (Fills in gaps in a filepath)
;Handy for file saving.
;Written by Dylan McCall (Mr. Picklesworth)
	path$=extractfilepath(path$)
	Local c=1,pathTo$
	Repeat
		slash=Instr(path$,"\",c)
		If slash=0
			If c>=Len(path$)+1
				Exit
			Else
				slash=Len(path$)+1
			EndIf
		EndIf
		
		folder$=Mid(path$,c,slash-c)
		If FileType(pathTo$+folder)=0 Then CreateDir(pathTo$+folder)
		c=slash+1
		pathTo$=pathTo$+folder+"\"
	Forever
	Return 1
End Function

;FUNCTION ExtractFilePath
;Accepts a filepath with filename and returns only the path
;i.e. pass c:\temp\test.txt the return value will be c:\temp\
Function ExtractFilePath$(sFilePath$) ;Written by Perturbatio
    ;LOCAL VARS
    Local iStartPos% = 0
    Local iSearchPos% = 0
    Local iFilePathLength = 0
    Local sFileExt$ = ""
    
    ;BEGIN FUNCTION CODE
    iFilePathLength = Len(sFilePath$)
    iSearchPos% = iFilePathLength
    
    While (iStartPos% < 1) And (iSearchPos% > 1)
        
        iStartPos% = Instr(sFilePath$, "\", iSearchPos%)
        iSearchPos% = iSearchPos% - 1
        
    Wend
    
    If iStartPos = 0 Then ;if the filepath contains no backslashes
        sFileExt$ = sFilePath$
    Else
        sFileExt$ = Left$(sFilePath$, iStartPos%)
    EndIf
    
    
    Return sFileExt$    
End Function

Ow .. not really worth mentioning .. I needed to add WaitKey() commands to the examples to see their output.

The extract from/to banks functions seems very exciting, too. I can imagine this is very useful.
It works very well and thanks for showing this. I didn't knew this was possible at all.

Phil Newton

(Posted 2007) [#8]

Thanks for the feedback b32, it seems a bug crept into that example. The constant should actually be called "ZIPAPI_END_OF_LIST_OF_FILE" not "ZIPAPI_UNZ_END_OF_LIST_OF_FILE". Once that's changed it reads through all of the files. I've fixed the online example to reflect this change.

I think the "WaitKey" is required because it runs in a terminal window, and once it's completed the window closes. Still, great to see you're finding it useful, and thanks again for spotting that bug!

b32	(Posted 2007) [#9]

I was trying to compress/decompress banks, when I noticed that sometimes the uncompressed bank is not created.
I first got an error saying 'bbBank does not exist' when I tried it on an empty bank. Later on, I got the same error with a bank that contains a bitmap that is filled with a solid color. I filled the image with purple (255..0..255) and drew a face on it in Paint.
On this example, I can't read a banksize because of an empty bank:

	
	; Include required libraries
	Include "../Blitz_Basic_Bank.bb"
	Include "../Blitz_File_FileName.bb"
	Include "../Blitz_File_ZipApi.bb"
		
	bank = CreateBank(255)
	bank1 = ZipApi_Compress(bank)
	bank2 = ZipApi_UnCompress(bank)
	Print BankSize(bank2)

	FreeBank bank
	FreeBank bank1
	FreeBank bank2
	
	End

Phil Newton

(Posted 2007) [#10]

Shouldn't it read:

bank2 = ZipApi_UnCompress(bank1)

The uncompress command will return an invalid handle because it's trying to uncompress invalid data (It returns ZIPAPI_DATA_ERROR instead).

Still, it might me more useful if it returned the original data in a new bank to stop runtime errors. The same goes for trying to compress an empty bank - it will return 0 instead of a new bank. Would it be better to return new banks in both of these cases?

b32	(Posted 2007) [#11]

Ow .. that's stupid, the example was wrong .. I understand the error checking now. It seems my experiment returned a ZIPAPI_BUF_ERROR.
However, I don't understand why. When I poke random data in a bank, size 5000, the compressed version of the bank can be uncompressed. However, when I leave the original bank empty, or fill it with predicable numbers, it can't. Is that right ?

	; Include required libraries
	Include "../Blitz_Basic_Bank.bb"
	Include "../Blitz_File_FileName.bb"
	Include "../Blitz_File_ZipApi.bb"		

	;create bank	
	bank = CreateBank(5000)

;	;without these lines, it returns a data error
;   ;but with them, the bank can be extracted 
;
;	For i = 0 To BankSize(bank) / 2
;		PokeByte bank, i, Rand(255)
;	Next
	
	;compress
	bank1 = ZipApi_Compress(bank)
	bank2 = ZipApi_UnCompress(bank1)
	Print "org:" + BankSize(bank)
	Print "cmp:" + BankSize(bank1)
	
	;cmp. ratio
	Print "scale = " + BankSize(bank) / Float(BankSize(bank1))

	Print "error? " + bank2
	
	If bank2 > 0 Then
	
		;test uncompressed bank
		Print "unc:" + BankSize(bank2)
	
	End If
	
	FreeBank bank
	FreeBank bank1
	FreeBank bank2

	WaitKey()
	End

I think the structure you chose is a good solution. I don't think is it necessary to be able to compress banks with a length of zero. If any error occurs, it might be better to abort, it will make debugging much easier I think.

Phil Newton

(Posted 2007) [#12]

That's an interesting problem. To uncompress packed data, zlib needs to know how big of an output buffer to use. You can pass this in the second parameter of ZipApi_UnCompress, but if it's not passed then the app will "guess" and create a bank 100 times bigger than the input bank's size. This works fine in most cases, but if you're compressing an empty bank it will be much smaller, so the output buffer would need to be more than 100 times bigger than the input.

The best solution I can think of would be to do the following:

bank1 = CreateBank(5000)
bank2 = ZipApi_Compress(bank1)

; Resize the output, and store the original size at the last 4 bytes
ResizeBank(bank2, BankSize(bank2) + 4)
PokeInt(bank2, BankSize(bank2) - 4, BankSize(bank1))

; To unpack...

; Get the unpacked size
Local unpackedSize	= PeekInt(bank2, BankSize(bank2) - 4)

; Trim the size off the end
ResizeBank(bank2, BankSize(bank2) + 4)

; Unpack it
bank3	= ZipApi_UnCompress(bank2, unpackedSize)

Print BankSize(bank3)

This adds the original size to the packed data, so it can be extracted later. It works fine as long as you remember to trim the last 4 bytes before unpacking it. I could integrate this functionality into the library, but it would mean the packed data wouldn't be compatible with other libraries without some modification of the bank data (and it might break existing apps).

It wouldn't be too hard to add as two extra functions though (CompressBank and UncompressBank). This would preserve compatibility and also solve the problem.

Let me know if you'd find that useful, and if so I'll add it to the next release. Thanks for all your feedback too, it's extremely helpful!

b32	(Posted 2007) [#13]

Ow .. now it makes sense. Again, I feel a bit stupid. Thanks for clearing this up. It works now, I was going by the example and didn't realise there was another optional parameter.

As for the extra functions, that is a good question.
When I downloaded this, I went by the examples to figure out what I should do.
From this perspective, I would say: adjust the example to use the "unpackedSize" parameter. Since this is the 'safe' method, I would consider it the 'official' method. Leaving the parameter out is optional. It is then up to the user to find a way to store the original size of the data.
Maybe, instead of estimating, you could define the temp. bank size with the maximum size that is available.

The CompressBank and UncompressBank functions are a good solution. Since they can co-exist next to the original functions, they would harm nobody. For me, I see no use in altering the compressed data, so it wouldn't matter if the size is stored there as well. For some reason I would say to store the 'size' at the start of the bank, but I don't know why. Maybe it is because I've seen that so often with other structures.

It works too good, this lib. I wanted to see if it is possible to make a single-file installer using the bank compression functions.
So I've converted all the files into banks, compressed them, converted them into Data statements and then with another program I uncompressed them and wrote them back to disk.
(To be able to use such a long source, I used "Include" to include it and didn't try to open it with the IDE.)
I've stored and restored 7.56 MB of files this way. It took BB about 5 minutes to compile, but it works! Amazing, don't you think ? Well, I've never done anything like it ..

So, actually, I didn't encounter any problems at all. It just works, and it is very good.

Phil Newton

(Posted 2007) [#14]

Version 1.2 is now available (see first post for links).

This version adds several new commands, but the most important are : ZipApi_CompressBank and ZipApi_UnCompressBank. CompressBank adds extra data to the end of the bank which is used by UnCompressBank to get the size of the unpacked data.

Another new command is ZipApi_GetUncompressedSize, which gets the total uncompressed size of a zip file. These are also a few bug-fixes here and there, but nothing major.

Abrexxes

(Posted 2007) [#15]

The first link is only 1.1. For 1.2 we must go to the homepage. ;)

Very cool stuff !

cu

Phil Newton

(Posted 2007) [#16]

Thanks for the heads-up Abrexxes - all fixed now :)